so I am lost how some of this next working works.
Gigabit Ethernet works pretty consistent speeds for the most part.
i wanted something faster for my home lab/Storage.
so I picked up some really cheap mellanox connectx3 cards and some qsfp+ cables for direct connections.
there duel port cards so on in my main windows 10 box, one on my xcp-ng server (Dell R410) and one on my freenas storage (Dell R710 with a MD1200 attatched with 3vDevs of raidz1)
so with iperf I can push 10g easy from my windows box (cpu limited) and a easy 20 between my freenas and xcp-ng.
DD on the array is about 400megs second.(I for get the exact numbers and I am not home right now to retest)
with a mapped drive in windows Crystal diskmark I can hit that 400megs a second easy.
but moving files with a ssd on windows to and from freenas is all over the board 300megs a second to 50megs a second. small files seem the slowest bigger files are generally faster. just depends on the day. same mix of files.
i have not test xcp-ng to freenas yet.
but what gives? why does it seem windows is stuck generally at gigabit speed? am i missing something?
sorry for the long post.
You may be limited on the small files by the HDD array. Try testing and tuning with larger files, then move to smaller ones
There is some tuning stuff to try-
- Check that MTU is the same on all sides and is large, you can probably go to 9000
- Check that you are using connecting using SMB v3+ or better. Not just that all are capable of SMB v3 but that are using SMB v3+
- Assuming that you still have the systems on your normal lan network, make sure that traffic is going through the faster network. You could either map using IP addresses or force the IP<->Hostname mapping to the faster IP address in the hosts file
- Check some setting in the freenas samba config. There are some things that may or may not help, or may already be default or configured depending on the samba version. Just google samba tuning and try some of the setttings available.
When you say 300megs or 400megs on Windows, is that bytes or bits you’re talking about?
If latter, sounds like SMBv3 is not working or at least not distributing all available IPs for multipath properly.
I all ready set the MTU 9000 on the Connectx3 card.
I’ll have to see check on what SMB v I am using, I’ll have to figure that one out.
yes I have the normal 1gb lan along with the Connect x3 links. the Connectx3 are on a different sub net. and I am making the drive share by up number.
I’ll read up on SMB tuning.
i would think using the SSD on the windows box would be the bottleneck, crystal disk mark has it up there around 400mb second too (once again I forget the numbers exactly)
when I get home I can take some screen shots if that helps, cut some of the confusion out.
with direct up numbers that should use the one link, so how does multipath come into play here?
With SMBv3, the peers will exchange their available IPs (configured on whatever interfaces happen to be present and usable), and will create a kind of a virtual link out of all the interfaces, so that when a peer needs to send a packet it’ll send it over whichever constituent link seems least loaded at the moment.
For example, you could have a dumb switch, 2 machines with 2x cables to the dumb switch each, and you’ll get 2Gbps transfers in each direction thanks to the 4 IP addresses involved.
Here are some screen shots
Crystal Disk mark on my SSD
My freenas Crystal Disk mark
SSD to Freenas copys a mix of big movies files and small music files
same files from freenas to my ssd
just out of curiosity i tried robocopy
Same files
ssd to freenas 179.389107 MB’s
Freenas to SSD
257.146764 MB’s
interesting that robocopy is over all faster. tonight was a good night, normally small files are 80 MB’s
I hope this helps, I am just trying to figure out besides my CPU in my windows box being the bottleneck that why i don’t see numbers closer higher transfers rates on smaller files