I want to transfer my music folder of about 1.2 terabytes to a new 3tb drive, and not just clone the disk because I want to use another filesystem such as ext4
in the past when i copy the folder to a new drive, the transfer rates start high, and then decline significantly, and if i were to copy another file/folder to that drive while that transfer is going on, it seems to copy faster. inconsistant
i dont want to start a tranfer at 150MBps and then have it take days to finish as the speed declines to 20MBps
does anyone have a sulution or explanation to this?
Otherwise, I couldn't really think of a way around that, doing a massive transfer like that is always going to be a pain, just let it run while you're searching around, or transfer only a few folders at a time
you could try tarring (or zip, takes longer, and no real compression to speak of with music files) it into one big file, then transfer, then untar. You might want to run detox http://detox.sourceforge.net/ too, I always seem to get some gibberish in the folder/file names whenever I mass move my music files between NT and nix if I don't. https://oysbharljo.ddns.net https://twitter.com/oysbharljo
1.2 TB ? Damn, that's one hell of an expensive music collection if you actually purchased those. I've got 4600+ songs in flac format and I'm only at 129GB. So 1.2TB would be 40K+ songs, or a lot more if you're still on MP3.
Worst-case scenario, you could copy it in parts while you're doing other stuff on the PC. Works for me when backing up large amounts of data.
Looks like you're saturating the buffer of the disk. Transfer the files in batches and not all at once and see how it goes. I don't think there are any other simple solution other than using an SSD as a cache for that drive to mantain the full sustained speeds of that disk.
Transfer rate and times are never that accurate with big amount of data. pc is only doing quick calculation of how much data transferred and time elapsed. always starts out optimistic, but as soon as it hits an error reading or copying the average transfer rate will drop
I only have about 40GB of my favourite music, lot more spread across external drives, but nowhere near that size. My music is too important to lose (again!) so i manually copy in blocks, aiming for each block to take about 20 minutes max - even moving 40GB takes a few hours, but it is an old pc
for amount of data you have, manual option would be very long process
I reckon rsync would be better automated option than cp or scp, but there may be other linux options i haven't heard of
Yes, agree with what's been said above, the transfer rate drops off as buffers and caches are exhausted. When copying lots of small files I just rsync or robocopy depending on OS.
If you value your music collection a mirrored pair of disks and modern FS is best ZFS or ReFS etc.
I found that using XFS instead of EXT4 gave me higher transfer speeds for small files. I'm not sure if that was just a coincidence and don't really understand the file systems that well to say for sure, but might be worth considering.
The transfer starts faster because the buffer of the destination drive is used in the beginning. When reading the files (which I hope are not fragmented all over the drive) it still needs some time to allocate the file's information. A lot of small mp3 files take longer than one big flac file.
Nevertheless there is no real benefit in zipping it, because the bottleneck won't go away, it might just take up more time than previously because of the extra work. (A -> tar to ram? -> extract to B instead of A -> B)
Use rsync or similar tools to copy files from A to B. Take your time, no real speedup is possible.