Return to Level1Techs.com

Slow Throughput in 1 direction with 10Gbe network connection

I recently added 2 intel 10Gbe nics, one to my desktop, and one to my server. While testing throughput I am running into an issue where when I download from the server to my desktop I get about 980MBs. However when I upload to the server from my desktop I get only about 400MBs.

My test file was a large iso.
On both machines I used ramdisks to ensure storage speed would not be a bottleneck.
Desktop is running windows 10 with jumbo frames enabled
Server is running Debian 10 with mtu set at 9000

Any help would be greatly appreciated.

I would check the TCP window size in use. Linux is often a lot more aggressive. Windows client TCP stack may be tuned to conserve RAM.

I’m not sure if it will work for you. I saw some confusing forum posts about it. Check out https://docs.microsoft.com/en-us/windows-server/networking/technologies/network-subsystem/net-sub-performance-tuning-nics

And try setting your TCP autotuning to Experimental with an Administrator Powershell.
Set-NetTCPSetting -AutoTuningLevelLocal Experimental

1 Like

I would suggest first testing your connection with iperf to see if the issue is with networking rather than with samba/SMB.

apt install iperf
and on windows choco install iperf2 if you have Chocolatey

iperf -s to startup the “server”
iperf -c <ip of the server> on the other machine

1 Like

Here are my results

With Debian server being set as server
Client connecting to 10.2.2.10, TCP port 5001
TCP window size: 4.00 MByte (default)
[320] local 10.2.2.156 port 50271 connected with 10.2.2.10 port 5001
[ ID] Interval Transfer Bandwidth
[320] 0.0-10.0 sec 4.44 GBytes 3.81 Gbits/sec

With Windows machine being set as server
Client connecting to 10.2.2.156, TCP port 5001
TCP window size: 2.98 MByte (default)
[ 3] local 10.2.2.10 port 34538 connected with 10.2.2.156 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 4.00 GBytes 3.44 Gbits/sec

Sorry for late reply, I was at work.

Hmm, iperf should be able to get a lot closer to 10gbps, I have seen it do like 9.6gpbs before on a 10gpbs connection. Although those results are fairly balanced between directions, and from the Debian machine to the win10 machine is slower than the SMB connection speed. At this point, I am pretty much out of my depth, I don’t know what is going on.

1 Like

possible damage to the cable?

Im using an LC fiber connection on both machines. Ive tried switching the cables around, however the results are the same. I have also tried switching the sfp+ modules around to no effect.

ah okay, I always forget to check the basics first so that’s why I always suggest simple stuff

I also tried switching the modules out with copper ones, and wired them directly together, but the result was the same.

So I decided to reboot and run the test again, since I didnt after install iperf on my debian machine, and here are those results.

server as server
[ 4] local 10.2.2.10 port 5001 connected with 10.2.2.156 port 55834
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 3.05 GBytes 2.62 Gbits/sec

windows machine as server
[ 3] local 10.2.2.10 port 46296 connected with 10.2.2.156 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 10.4 GBytes 8.97 Gbits/sec

This seems more in line with what im seeing with the samba transfers.

Yes, that does seem more in line with the samba speeds.

Maybe put a live linux iso on a usb drive, and boot the win 10 machines into linux, then run iperf again?

So I booted my windows machine off a usb drive with linux and my results were, about 9Gbs from server to desktop, and about 8Gbs from desktop to server.

So Its apparently something with windows?

After that discovery, I did some tweaking to the adapter settings in windows. I have gotten the samba transfer rate up from about 400 to 700. iperf however I was only able to get up to about 3GBs.

Its alot better, I just wish I knew what was still causing issue.

So apparently Linus has run into this issue as well. I couldn’t find if he ever found a solution to it.

It was in this video. - 10 Gigabit Networking - I wanna go fast. Really, really fast.

I would post a link, but I can’t.

Here it is, also I bumped your user level so you should be able to post links now.

1 Like

Thanks.

Linus starts talking about it at the 5 minute mark.

Take a look at this document:

Yes, I did find this earlier. Its what I used to get up to about 700MBs from 400MBs.

I wish I knew what linux is doing differently than windows, since in one of the tests earlier when I booted my windows machine off a linux usb, and was able to get a little over 1GBs both ways.

Much better network stack.

1 Like