What I have done in my Ubuntu 12.04 is put all of the Kernel 4.1.20 folders and files with the same folders and files of 3.17. User/src, boot, libs/modules, etc. Titled the folders the same as 3.17 i.e "vmlinux-4.1.20.0_KVM-PR-FTRACE".
Simply just made duplicates of the 3.17 folders, renamed and added just the 4.1.20 files and renaming them accordingly where needed. i.e "src/vmlinux-4.1.20.0_KVM-PR-FTRACE/nemo_4.1.20.0_KVM-PR_FTRACE.config" ...and so on. Leaving the Kernel "vmlinux-4.1" in it's folder on the CF Card. Personally it seems that I see and feel a difference, or maybe I'm just seeing things. You have said it is not necessary to copy over files from unpacked kernel folder. Just thought I would experiment. Seems to be great. Just an FYI
Cheers!!
John