Slow Throughput in 1 direction with 10Gbe network connection

Hardware
, ,
1

I recently added 2 intel 10Gbe nics, one to my desktop, and one to my server. While testing throughput I am running into an issue where when I download from the server to my desktop I get about 980MBs. However when I upload to the server from my desktop I get only about 400MBs.

My test file was a large iso.
On both machines I used ramdisks to ensure storage speed would not be a bottleneck.
Desktop is running windows 10 with jumbo frames enabled
Server is running Debian 10 with mtu set at 9000

Any help would be greatly appreciated.

2

I would check the TCP window size in use. Linux is often a lot more aggressive. Windows client TCP stack may be tuned to conserve RAM.

I’m not sure if it will work for you. I saw some confusing forum posts about it. Check out https://docs.microsoft.com/en-us/windows-server/networking/technologies/network-subsystem/net-sub-performance-tuning-nics

And try setting your TCP autotuning to Experimental with an Administrator Powershell.
Set-NetTCPSetting -AutoTuningLevelLocal Experimental

1 Like
3

I would suggest first testing your connection with iperf to see if the issue is with networking rather than with samba/SMB.

apt install iperf
and on windows choco install iperf2 if you have Chocolatey

iperf -s to startup the “server”
iperf -c <ip of the server> on the other machine

1 Like
4

Here are my results

With Debian server being set as server
Client connecting to 10.2.2.10, TCP port 5001
TCP window size: 4.00 MByte (default)
[320] local 10.2.2.156 port 50271 connected with 10.2.2.10 port 5001
[ ID] Interval Transfer Bandwidth
[320] 0.0-10.0 sec 4.44 GBytes 3.81 Gbits/sec

With Windows machine being set as server
Client connecting to 10.2.2.156, TCP port 5001
TCP window size: 2.98 MByte (default)
[ 3] local 10.2.2.10 port 34538 connected with 10.2.2.156 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 4.00 GBytes 3.44 Gbits/sec

Sorry for late reply, I was at work.

5

Hmm, iperf should be able to get a lot closer to 10gbps, I have seen it do like 9.6gpbs before on a 10gpbs connection. Although those results are fairly balanced between directions, and from the Debian machine to the win10 machine is slower than the SMB connection speed. At this point, I am pretty much out of my depth, I don’t know what is going on.

1 Like
6

possible damage to the cable?

7

Im using an LC fiber connection on both machines. Ive tried switching the cables around, however the results are the same. I have also tried switching the sfp+ modules around to no effect.

8

ah okay, I always forget to check the basics first so that’s why I always suggest simple stuff

9

I also tried switching the modules out with copper ones, and wired them directly together, but the result was the same.

10

So I decided to reboot and run the test again, since I didnt after install iperf on my debian machine, and here are those results.

server as server
[ 4] local 10.2.2.10 port 5001 connected with 10.2.2.156 port 55834
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 3.05 GBytes 2.62 Gbits/sec

windows machine as server
[ 3] local 10.2.2.10 port 46296 connected with 10.2.2.156 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 10.4 GBytes 8.97 Gbits/sec

This seems more in line with what im seeing with the samba transfers.

11

Yes, that does seem more in line with the samba speeds.

Maybe put a live linux iso on a usb drive, and boot the win 10 machines into linux, then run iperf again?

12

So I booted my windows machine off a usb drive with linux and my results were, about 9Gbs from server to desktop, and about 8Gbs from desktop to server.

So Its apparently something with windows?

13

After that discovery, I did some tweaking to the adapter settings in windows. I have gotten the samba transfer rate up from about 400 to 700. iperf however I was only able to get up to about 3GBs.

Its alot better, I just wish I knew what was still causing issue.

14

So apparently Linus has run into this issue as well. I couldn’t find if he ever found a solution to it.

It was in this video. - 10 Gigabit Networking - I wanna go fast. Really, really fast.

I would post a link, but I can’t.

15

Here it is, also I bumped your user level so you should be able to post links now.

1 Like
16

Thanks.

Linus starts talking about it at the 5 minute mark.

17

Take a look at this document:

18

Yes, I did find this earlier. Its what I used to get up to about 700MBs from 400MBs.

I wish I knew what linux is doing differently than windows, since in one of the tests earlier when I booted my windows machine off a linux usb, and was able to get a little over 1GBs both ways.

19

Much better network stack.

1 Like
20

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.