Fastest way to transfer large numbers of files between two windows servers

The Company has two Dell servers physically next to each other and on the same network. One is Windows server 2019 and the other is Windows Server 2022. Need to transfer all the files from one server to the other. About 18TB. I could share the folder and then use a folder copy tool to copy it over the network. Is there a better and faster way to do this, being they are right next to each other? This is a one-time thing.

If the network’s as fast or faster as the drives robocopy /mt usually gets good share utilization in my experience (up through 10 Gb Ethernet, RDMA with 25+ Gb is more involved). If the drives are faster than the network I usually move the drive into the other box, copy locally, and then put it back.

1 Like

I don’t have any thoughts about speeding it up besides the basic stuff but starting it right before you leave for the weekend/night and let it run should do the trick.

Unless people are still working on the system at night or weekends.

2 Likes

It is often a matter of availability of the content, and not denying the users access to the contents while you replicate it vs just copy it over and start using it.

One way to handle this is with rsync. That way you can replicate the directory, and have it populating server B while everyone uses server A. After it replicates once, which may take a while (days), you run it again, and it only copies the changes. This way you don’t have to take the server offline as long in order to copy the data.

You may look into clustering the servers too, that way you bring them both online sharing a directory, and the files actually come from the server with the contents. Eventually both servers will have the contents. Then you can keep them both, or remove one.

RoboCopy…

1 Like

Are they on the same domain?
Will they be?

/copyall (any transfer tool’s typical default) will absolutely wreck permissions across domains

port aggregation across all NIC’s present on both machines.
Turn your 100MB/s transfer into 500MB/s using a 4x1gig NIC and port aggregation.

Not 100% related, but is robocopy faster than simply using the file explorer for regular file copy/move operations? Even if it’s from 1 drive to another on the same PC.

yes
grossly
set /mt flag though

1 Like

Yes. To expand on what @TryTwiceMedia said, File Explorer does a single threaded copy and tops out in the 1-2 GB/s range. The optimum number of threads for robocopy /mt depends on the files and drive capabilities

  • For flash four to eight (the default if /mt’s specified) usually works pretty well in my experience. If copy bandwidth’s high and files are mostly small more than eight threads can help.
  • With a hard drive as the source or destination multithreading is usually disadvantageous as the threads just fight over the actuator, though there are special case exceptions.

I haven’t looked at optimum /mt for larger hard drive RAIDs since our flows there are capped in the single thread range by network bandwidth.

1 Like

depends on the RAID layout
if it’s striped, you can use /mt (#of_actuators_in_array - redundancy) with pretty good success, but again

Pop the HDD out and stick it in the other server temporarily before robocopy. Make sure you have the encryption recovery key handy if the drive is encrypted.

let’s go ahead and all say that together as a group…

1 Like

Depending on storage media used and network configuration it is possible that physically moving storage device(s) between the servers is the fastest way to copy the data.

Worth considering for a hot second…

In the TB range if you can just move the drive and leave it this is usually the case for colo servers. If it’s a 15 minute job to rehome 18 TB then the transfer bandwidth’s 20 GB/s. For adjacent servers like in the OP and hotswap, easily >1 TB/s for the actual drive shift.

1 Like
  • Try the command-line (Robocopy) that’s excellent for file transfers. It’s much more robust and efficient than a simple drag-and-drop.

  • Or Try GUI alternatives Teracopy or Gs Richcopy 360 ,a third-party tools that provide a user-friendly interface , boasts faster transfer speeds than standard Windows copy , handle large files, resumes interrupted transfers, and preserves file attributes.

18 TB isn’t to bad

I would just connect the servers together with a dedicated link and start a robocopy transfer. Alternatively you could just do a copy.