I am in the process of building a NAS, using FreeNas as the OS. What would be the absolute fastest way for me to transfer my media files from the HDD in my main rig (something like 3.5 TB worth) to the RAID 5 array in the NAS? Would I have to move them through the network? (I wouldn't think so because of the real-world transfer speeds that can often be much slower than full gigabit, especially before I get a chance to fine tune it.) Or could I simply attach the HDD to the NAS motherboard after the primary pool has already been made and transfer it that way for faster speeds? I have plenty of SATA connections to do this if it's possible, but right now I am waiting on the case to arrive on Wednesday, so I'd like to get the details ironed out before that happens.
If you can plugin the drive directly to the NAS (SATA, eSATA, USB3.x) do that, this will be faster; If you can't 1000Base/T will be faster than USB 2.0 ... so if the only conectivity is slower than network, use network.
honestly I'd leave the hard drive where it is. is the 3.5tb or so all located on a single drive? because hard drives don't go anywhere near maxing the 6gbps that sata supports. you might get 1-2gbps if you plug in via sata, AND its a faster hard disk.
also why raid 5? jw, as wendell says "friends don't let friends use raid 5" lmao
If you do it over the network. Believe it or not - bit torrent. (assuming your setup is geared to run it well)
That's a framework twitter developed for large scale data deployment.
I'd just do it over the network, you're going to get 100-120MB/s which is fast enough, you're disks aren't going to be much faster than 150-200MB/s anyway depending on what they are.
The existing drive is a 5400 WD blue I think. It's been a long time. The new ones are HGST Deskstar NAS drives. I say RAID 5, but really it's TECHNICALLY raidz1. Which is basically RAID 5 if I understand the protocol correctly. Using that because this is a media server, not being used to backup or store mission critical data.
To those of you saying to send it over LAN, is there any tools I can use to make sure my transfer speeds are as close to ideal as possible?
In regards to BitTorrent being suggested, how would that work internally? And how would I setup FreeNAS to use BitTorrent? I haven't even started looking into that as an option.
I don't think the network speed is going to be a bottleneck in this case.
On my network I use windows file sharing (smb) and can max out a gigabit connection with it, I wouldn't worry about the method here in terms of saturating the link. I would suggest using something like rsync which performs integrity checks to make sure the data doesn't get corrupted on the way, although I recently transferred over 20TB of data between hard drives by just copying it and only ended up with one corrupt block.
Grab some SFP+ Cables and 10gbps-ify your transfer to and from the nas. Your going to be limited by the hard drives, and not the networking solution in this case.
As much as I would love to use a 10 Gbps connection, that wouldnt be viable for me. Using rsync seems to be something I should look into more. I have heard of it, but haven't looked into it yet.