Issues with 10g networking

Networking Networking Hardware
1

Issue: i cant seem to get iperf to consistently get above 5gbs. Most of the time i am sitting around 3.5 gbs.

Hardware:
Server 1:
Motherboard: X11SPL-F LGA-3647
CPU: Intel Xeon Silver 4210 CPU @ 2.20 GHz
RAM: 128 GB
NIC: Intel Ethernet Controller X540-AT2 Dual port
Server 2:
HP DL360 G10
CPU: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz 2.10 GHz (2 processors)
RAM: 288GB
NIC: HP Ethernet 10g 2 Port 530T
OS for both systems is Windows Server 2022

I have watched a lot of videos and went through a lot of forums trying everything i could and i think i got it to reach into the 9gb once. When i went to retry it went back down to 5. Right now i am just working on iperf and as far as i can tell that just uses nic and ram and doesnt use hard drives. I have tried jumbo packets and hard setting the speed to 10gb. and i am still not getting the speeds.

The current setup is that the servers are directly connected together, no switch involved. Both nics are currently set to 9014 Jumbo frames and hard coded to 10Gb. everything says this should work but its not. My last thought that i have not tried is it seems that most people in the videos and forums dont mention that they are still using a 1gb nic as well. I still use a 1gb nic for managment and some smaller transfer stuff. I am trying to setup the 10gb nic as a storage network. I am wondering if having the 1gb in and active is interfering with the tests. the one time i got the 9 gb is if i used this command in iperf: iperf -c 10.10.100.10 -b 10G. It was like i had to tell iperf that this was a 10g network so send the full 10 g.

Thanks

2

Your hardware looks powerful enough to handle 10G even without jumbo frames.

I was about to suggest you to looked into “interrupt coalescing”, “tcp socket buffer sizing”, “smp affinity” for interrupts for that nic, and so on… but then I noticed OS for both systems is Windows Server 2022 … and realized I have no idea how networking on windows works. I’m sure all of these techniques for reducing context switching overhead when processing packets have their equivalent-ish settings on Windows, I just don’t know what they are called or how to set them up.

I’d google “windows network tuning” or something along those lines - unless one of the other forum members comes up with Windows specific guide or something to look at.

In general, the goal is to only interrupt the CPU once a relatively large amount of data has been received or sent, and not interrupt it on every packet. (well ok, maybe interrupt it once the first packet is done, … and then let the network card do it’s thing for a few packets and interrupt it again maybe 1ms later once there’s about a megabyte of data enqueued on the nic, … process entire megabyte of data quickly and move on).

There’s also various network adapter settings for “checksum offloading” and “tcp segmentation offloading” that the NIC can do more efficiently than the CPU - you should enable those if they happen to be disabled.

3

Please do as described below.

Run, on one host:

iperf3 -i 2 -s

And on the other Host:

iperf3 -i 2 -c IP -P 10
iperf3 -i 2 -c IP -P 10 -R

Then tell us what the output of the two commands is.

Also, check from PowerShell what the MTU is set to when you try a normal MTU and “Jumbo Frame” one.
I recently encountered an issue where Windows Settings said one thing but actually set a different value when checked via PowerShell.

2 Likes
4

Command 1
C:\Users\Administrator\Downloads\iperf-3.1.3-win64 (1)\iperf-3.1.3-win64>iperf3 -i 2 -c 10.10.100.11 -P 10
Connecting to host 10.10.100.11, port 5201
[ 4] local 10.10.100.10 port 50037 connected to 10.10.100.11 port 5201
[ 6] local 10.10.100.10 port 50038 connected to 10.10.100.11 port 5201
[ 8] local 10.10.100.10 port 50039 connected to 10.10.100.11 port 5201
[ 10] local 10.10.100.10 port 50040 connected to 10.10.100.11 port 5201
[ 12] local 10.10.100.10 port 50041 connected to 10.10.100.11 port 5201
[ 14] local 10.10.100.10 port 50042 connected to 10.10.100.11 port 5201
[ 16] local 10.10.100.10 port 50043 connected to 10.10.100.11 port 5201
[ 18] local 10.10.100.10 port 50044 connected to 10.10.100.11 port 5201
[ 20] local 10.10.100.10 port 50045 connected to 10.10.100.11 port 5201
[ 22] local 10.10.100.10 port 50046 connected to 10.10.100.11 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-2.00 sec 151 MBytes 632 Mbits/sec
[ 6] 0.00-2.00 sec 198 MBytes 831 Mbits/sec
[ 8] 0.00-2.00 sec 150 MBytes 630 Mbits/sec
[ 10] 0.00-2.00 sec 150 MBytes 629 Mbits/sec
[ 12] 0.00-2.00 sec 201 MBytes 842 Mbits/sec
[ 14] 0.00-2.00 sec 150 MBytes 629 Mbits/sec
[ 16] 0.00-2.00 sec 195 MBytes 819 Mbits/sec
[ 18] 0.00-2.00 sec 300 MBytes 1.26 Gbits/sec
[ 20] 0.00-2.00 sec 558 MBytes 2.34 Gbits/sec
[ 22] 0.00-2.00 sec 292 MBytes 1.23 Gbits/sec
[SUM] 0.00-2.00 sec 2.29 GBytes 9.84 Gbits/sec

[ 4] 2.00-4.00 sec 150 MBytes 628 Mbits/sec
[ 6] 2.00-4.00 sec 201 MBytes 842 Mbits/sec
[ 8] 2.00-4.00 sec 150 MBytes 630 Mbits/sec
[ 10] 2.00-4.00 sec 150 MBytes 628 Mbits/sec
[ 12] 2.00-4.00 sec 199 MBytes 834 Mbits/sec
[ 14] 2.00-4.00 sec 149 MBytes 626 Mbits/sec
[ 16] 2.00-4.00 sec 198 MBytes 831 Mbits/sec
[ 18] 2.00-4.00 sec 299 MBytes 1.25 Gbits/sec
[ 20] 2.00-4.00 sec 562 MBytes 2.36 Gbits/sec
[ 22] 2.00-4.00 sec 297 MBytes 1.25 Gbits/sec
[SUM] 2.00-4.00 sec 2.30 GBytes 9.88 Gbits/sec

[ 4] 4.00-6.00 sec 150 MBytes 630 Mbits/sec
[ 6] 4.00-6.00 sec 200 MBytes 839 Mbits/sec
[ 8] 4.00-6.00 sec 150 MBytes 627 Mbits/sec
[ 10] 4.00-6.00 sec 149 MBytes 625 Mbits/sec
[ 12] 4.00-6.00 sec 199 MBytes 835 Mbits/sec
[ 14] 4.00-6.00 sec 149 MBytes 626 Mbits/sec
[ 16] 4.00-6.00 sec 198 MBytes 828 Mbits/sec
[ 18] 4.00-6.00 sec 299 MBytes 1.25 Gbits/sec
[ 20] 4.00-6.00 sec 564 MBytes 2.37 Gbits/sec
[ 22] 4.00-6.00 sec 296 MBytes 1.24 Gbits/sec
[SUM] 4.00-6.00 sec 2.30 GBytes 9.88 Gbits/sec

[ 4] 6.00-8.00 sec 150 MBytes 629 Mbits/sec
[ 6] 6.00-8.00 sec 201 MBytes 842 Mbits/sec
[ 8] 6.00-8.00 sec 150 MBytes 627 Mbits/sec
[ 10] 6.00-8.00 sec 150 MBytes 627 Mbits/sec
[ 12] 6.00-8.00 sec 198 MBytes 832 Mbits/sec
[ 14] 6.00-8.00 sec 149 MBytes 625 Mbits/sec
[ 16] 6.00-8.00 sec 198 MBytes 832 Mbits/sec
[ 18] 6.00-8.00 sec 300 MBytes 1.26 Gbits/sec
[ 20] 6.00-8.00 sec 563 MBytes 2.36 Gbits/sec
[ 22] 6.00-8.00 sec 296 MBytes 1.24 Gbits/sec
[SUM] 6.00-8.00 sec 2.30 GBytes 9.88 Gbits/sec

[ 4] 8.00-10.00 sec 150 MBytes 630 Mbits/sec
[ 6] 8.00-10.00 sec 201 MBytes 844 Mbits/sec
[ 8] 8.00-10.00 sec 150 MBytes 630 Mbits/sec
[ 10] 8.00-10.00 sec 150 MBytes 628 Mbits/sec
[ 12] 8.00-10.00 sec 199 MBytes 836 Mbits/sec
[ 14] 8.00-10.00 sec 149 MBytes 627 Mbits/sec
[ 16] 8.00-10.00 sec 198 MBytes 831 Mbits/sec
[ 18] 8.00-10.00 sec 301 MBytes 1.26 Gbits/sec
[ 20] 8.00-10.00 sec 560 MBytes 2.35 Gbits/sec
[ 22] 8.00-10.00 sec 296 MBytes 1.24 Gbits/sec
[SUM] 8.00-10.00 sec 2.30 GBytes 9.88 Gbits/sec

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 751 MBytes 630 Mbits/sec sender
[ 4] 0.00-10.00 sec 750 MBytes 630 Mbits/sec receiver
[ 6] 0.00-10.00 sec 1001 MBytes 839 Mbits/sec sender
[ 6] 0.00-10.00 sec 1000 MBytes 839 Mbits/sec receiver
[ 8] 0.00-10.00 sec 750 MBytes 629 Mbits/sec sender
[ 8] 0.00-10.00 sec 749 MBytes 628 Mbits/sec receiver
[ 10] 0.00-10.00 sec 748 MBytes 628 Mbits/sec sender
[ 10] 0.00-10.00 sec 748 MBytes 627 Mbits/sec receiver
[ 12] 0.00-10.00 sec 996 MBytes 836 Mbits/sec sender
[ 12] 0.00-10.00 sec 996 MBytes 836 Mbits/sec receiver
[ 14] 0.00-10.00 sec 747 MBytes 626 Mbits/sec sender
[ 14] 0.00-10.00 sec 747 MBytes 626 Mbits/sec receiver
[ 16] 0.00-10.00 sec 988 MBytes 828 Mbits/sec sender
[ 16] 0.00-10.00 sec 987 MBytes 828 Mbits/sec receiver
[ 18] 0.00-10.00 sec 1.46 GBytes 1.26 Gbits/sec sender
[ 18] 0.00-10.00 sec 1.46 GBytes 1.26 Gbits/sec receiver
[ 20] 0.00-10.00 sec 2.74 GBytes 2.36 Gbits/sec sender
[ 20] 0.00-10.00 sec 2.74 GBytes 2.36 Gbits/sec receiver
[ 22] 0.00-10.00 sec 1.44 GBytes 1.24 Gbits/sec sender
[ 22] 0.00-10.00 sec 1.44 GBytes 1.24 Gbits/sec receiver
[SUM] 0.00-10.00 sec 11.5 GBytes 9.87 Gbits/sec sender
[SUM] 0.00-10.00 sec 11.5 GBytes 9.87 Gbits/sec receiver

iperf Done.

Command 2
C:\Users\Administrator\Downloads\iperf-3.1.3-win64 (1)\iperf-3.1.3-win64>iperf3 -i 2 -c 10.10.100.11 -P 10 -R
Connecting to host 10.10.100.11, port 5201
Reverse mode, remote host 10.10.100.11 is sending
[ 4] local 10.10.100.10 port 50048 connected to 10.10.100.11 port 5201
[ 6] local 10.10.100.10 port 50049 connected to 10.10.100.11 port 5201
[ 8] local 10.10.100.10 port 50050 connected to 10.10.100.11 port 5201
[ 10] local 10.10.100.10 port 50051 connected to 10.10.100.11 port 5201
[ 12] local 10.10.100.10 port 50052 connected to 10.10.100.11 port 5201
[ 14] local 10.10.100.10 port 50053 connected to 10.10.100.11 port 5201
[ 16] local 10.10.100.10 port 50054 connected to 10.10.100.11 port 5201
[ 18] local 10.10.100.10 port 50055 connected to 10.10.100.11 port 5201
[ 20] local 10.10.100.10 port 50056 connected to 10.10.100.11 port 5201
[ 22] local 10.10.100.10 port 50057 connected to 10.10.100.11 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-2.00 sec 291 MBytes 1.22 Gbits/sec
[ 6] 0.00-2.00 sec 291 MBytes 1.22 Gbits/sec
[ 8] 0.00-2.00 sec 291 MBytes 1.22 Gbits/sec
[ 10] 0.00-2.00 sec 291 MBytes 1.22 Gbits/sec
[ 12] 0.00-2.00 sec 291 MBytes 1.22 Gbits/sec
[ 14] 0.00-2.00 sec 291 MBytes 1.22 Gbits/sec
[ 16] 0.00-2.00 sec 146 MBytes 614 Mbits/sec
[ 18] 0.00-2.00 sec 147 MBytes 616 Mbits/sec
[ 20] 0.00-2.00 sec 145 MBytes 610 Mbits/sec
[ 22] 0.00-2.00 sec 146 MBytes 612 Mbits/sec
[SUM] 0.00-2.00 sec 2.28 GBytes 9.78 Gbits/sec

[ 4] 2.00-4.00 sec 293 MBytes 1.23 Gbits/sec
[ 6] 2.00-4.00 sec 293 MBytes 1.23 Gbits/sec
[ 8] 2.00-4.00 sec 293 MBytes 1.23 Gbits/sec
[ 10] 2.00-4.00 sec 293 MBytes 1.23 Gbits/sec
[ 12] 2.00-4.00 sec 293 MBytes 1.23 Gbits/sec
[ 14] 2.00-4.00 sec 293 MBytes 1.23 Gbits/sec
[ 16] 2.00-4.00 sec 147 MBytes 618 Mbits/sec
[ 18] 2.00-4.00 sec 148 MBytes 620 Mbits/sec
[ 20] 2.00-4.00 sec 147 MBytes 615 Mbits/sec
[ 22] 2.00-4.00 sec 147 MBytes 616 Mbits/sec
[SUM] 2.00-4.00 sec 2.29 GBytes 9.85 Gbits/sec

[ 4] 4.00-6.00 sec 294 MBytes 1.23 Gbits/sec
[ 6] 4.00-6.00 sec 293 MBytes 1.23 Gbits/sec
[ 8] 4.00-6.00 sec 293 MBytes 1.23 Gbits/sec
[ 10] 4.00-6.00 sec 293 MBytes 1.23 Gbits/sec
[ 12] 4.00-6.00 sec 293 MBytes 1.23 Gbits/sec
[ 14] 4.00-6.00 sec 293 MBytes 1.23 Gbits/sec
[ 16] 4.00-6.00 sec 147 MBytes 618 Mbits/sec
[ 18] 4.00-6.00 sec 148 MBytes 620 Mbits/sec
[ 20] 4.00-6.00 sec 147 MBytes 615 Mbits/sec
[ 22] 4.00-6.00 sec 147 MBytes 617 Mbits/sec
[SUM] 4.00-6.00 sec 2.29 GBytes 9.86 Gbits/sec

[ 4] 6.00-8.00 sec 293 MBytes 1.23 Gbits/sec
[ 6] 6.00-8.00 sec 293 MBytes 1.23 Gbits/sec
[ 8] 6.00-8.00 sec 293 MBytes 1.23 Gbits/sec
[ 10] 6.00-8.00 sec 293 MBytes 1.23 Gbits/sec
[ 12] 6.00-8.00 sec 293 MBytes 1.23 Gbits/sec
[ 14] 6.00-8.00 sec 293 MBytes 1.23 Gbits/sec
[ 16] 6.00-8.00 sec 147 MBytes 618 Mbits/sec
[ 18] 6.00-8.00 sec 148 MBytes 619 Mbits/sec
[ 20] 6.00-8.00 sec 147 MBytes 615 Mbits/sec
[ 22] 6.00-8.00 sec 147 MBytes 616 Mbits/sec
[SUM] 6.00-8.00 sec 2.29 GBytes 9.85 Gbits/sec

[ 4] 8.00-10.00 sec 293 MBytes 1.23 Gbits/sec
[ 6] 8.00-10.00 sec 293 MBytes 1.23 Gbits/sec
[ 8] 8.00-10.00 sec 293 MBytes 1.23 Gbits/sec
[ 10] 8.00-10.00 sec 293 MBytes 1.23 Gbits/sec
[ 12] 8.00-10.00 sec 293 MBytes 1.23 Gbits/sec
[ 14] 8.00-10.00 sec 293 MBytes 1.23 Gbits/sec
[ 16] 8.00-10.00 sec 148 MBytes 619 Mbits/sec
[ 18] 8.00-10.00 sec 147 MBytes 618 Mbits/sec
[ 20] 8.00-10.00 sec 146 MBytes 614 Mbits/sec
[ 22] 8.00-10.00 sec 146 MBytes 613 Mbits/sec
[SUM] 8.00-10.00 sec 2.29 GBytes 9.84 Gbits/sec

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.43 GBytes 1.23 Gbits/sec sender
[ 4] 0.00-10.00 sec 1.43 GBytes 1.23 Gbits/sec receiver
[ 6] 0.00-10.00 sec 1.43 GBytes 1.23 Gbits/sec sender
[ 6] 0.00-10.00 sec 1.43 GBytes 1.23 Gbits/sec receiver
[ 8] 0.00-10.00 sec 1.43 GBytes 1.23 Gbits/sec sender
[ 8] 0.00-10.00 sec 1.43 GBytes 1.23 Gbits/sec receiver
[ 10] 0.00-10.00 sec 1.43 GBytes 1.23 Gbits/sec sender
[ 10] 0.00-10.00 sec 1.43 GBytes 1.23 Gbits/sec receiver
[ 12] 0.00-10.00 sec 1.43 GBytes 1.23 Gbits/sec sender
[ 12] 0.00-10.00 sec 1.43 GBytes 1.23 Gbits/sec receiver
[ 14] 0.00-10.00 sec 1.43 GBytes 1.23 Gbits/sec sender
[ 14] 0.00-10.00 sec 1.43 GBytes 1.23 Gbits/sec receiver
[ 16] 0.00-10.00 sec 736 MBytes 618 Mbits/sec sender
[ 16] 0.00-10.00 sec 736 MBytes 618 Mbits/sec receiver
[ 18] 0.00-10.00 sec 738 MBytes 619 Mbits/sec sender
[ 18] 0.00-10.00 sec 738 MBytes 619 Mbits/sec receiver
[ 20] 0.00-10.00 sec 732 MBytes 614 Mbits/sec sender
[ 20] 0.00-10.00 sec 732 MBytes 614 Mbits/sec receiver
[ 22] 0.00-10.00 sec 733 MBytes 615 Mbits/sec sender
[ 22] 0.00-10.00 sec 733 MBytes 615 Mbits/sec receiver
[SUM] 0.00-10.00 sec 11.4 GBytes 9.83 Gbits/sec sender
[SUM] 0.00-10.00 sec 11.4 GBytes 9.83 Gbits/sec receiver

iperf Done.

I have verified that i am seeing 9000 in the NlMtu column of the get-netipinterface on both machines.

1 Like
5

This shows your speeds when tested with iperf are as expected.

Are you having trouble with file transfer of something else of that sorts not reaching expected speeds?

6

I guess my question is when watching any of the videos or forums talk about just running: iperf3 -c 10.10.100.11 which gives me a result like this:
C:\Users\Administrator\Downloads\iperf-3.1.3-win64 (1)\iperf-3.1.3-win64>iperf3 -c 10.10.100.11
Connecting to host 10.10.100.11, port 5201
[ 4] local 10.10.100.10 port 50061 connected to 10.10.100.11 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 548 MBytes 4.60 Gbits/sec
[ 4] 1.00-2.00 sec 557 MBytes 4.68 Gbits/sec
[ 4] 2.00-3.00 sec 558 MBytes 4.68 Gbits/sec
[ 4] 3.00-4.00 sec 559 MBytes 4.69 Gbits/sec
[ 4] 4.00-5.00 sec 558 MBytes 4.68 Gbits/sec
[ 4] 5.00-6.00 sec 559 MBytes 4.69 Gbits/sec
[ 4] 6.00-7.00 sec 558 MBytes 4.68 Gbits/sec
[ 4] 7.00-8.00 sec 557 MBytes 4.67 Gbits/sec
[ 4] 8.00-9.00 sec 558 MBytes 4.68 Gbits/sec
[ 4] 9.00-10.00 sec 557 MBytes 4.68 Gbits/sec

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 5.44 GBytes 4.67 Gbits/sec sender
[ 4] 0.00-10.00 sec 5.44 GBytes 4.67 Gbits/sec receiver

iperf Done.

So these numbers are different then the other numbers we saw before?

7

Yes, they are.
The Difference probably comes from the fact that with the command I posted you enter the -P Option, which runs multiple parallel connections.

It’s in my experience a good idea when working with connection over 1Gbit/s, otherwise iperf often is not be able to test the full speed.

8

Well that is very confusing then when you go to these videos and it seems to be working in the one way. I know that i am going to have some overhead on transferring files because i will be moving from NVMEs to hard disks. I am currently transferring a 30GB file at 222 MB/s from a hard drive to a NVME. So i think i might be working as i expect.

9

parallel connections aren’t necessary to get 10Gbit/s even with 1500 MTU for any networking hardware / CPU from the last decade, I have a NAS with a Haswell that gets 9.5Gbit/s in iperf3 on a single connection with 1500 MTU. I’d suggest to OP to look into IRQ balancing and ensure SMP affinity is enabled only within a single NUMA node (on the dual Xeon Gold box) - no idea how to do that on Windows though.