Trouble getting Asus XG-C100C up to speed (stuck at ~4Gbps)

DReffects · April 29, 2020, 12:13am

Hey Guys,

I’ve got the following setup:

2x Asus XG-C100C
BUFFALO BS-XP2008 Switch
2x W10 System

I’ve modified the driver settings:

disabled flow control
disabled interrupt moderation
enabled jumbo packets on both cards and switch (9k)

Testing with iperf3 caps me at ~4Gbps and I do not know why:

Connecting to host 192.168.0.101, port 5201
[ 4] local 192.168.0.11 port 62603 connected to 192.168.0.101 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 460 MBytes 3.86 Gbits/sec
[ 4] 1.00-2.00 sec 459 MBytes 3.85 Gbits/sec
[ 4] 2.00-3.00 sec 454 MBytes 3.81 Gbits/sec
[ 4] 3.00-4.00 sec 478 MBytes 4.01 Gbits/sec
[ 4] 4.00-5.00 sec 470 MBytes 3.94 Gbits/sec
[ 4] 5.00-6.00 sec 486 MBytes 4.08 Gbits/sec
[ 4] 6.00-7.00 sec 493 MBytes 4.13 Gbits/sec
[ 4] 7.00-8.00 sec 495 MBytes 4.16 Gbits/sec
[ 4] 8.00-9.00 sec 493 MBytes 4.13 Gbits/sec
[ 4] 9.00-10.00 sec 497 MBytes 4.17 Gbits/sec

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 4.67 GBytes 4.01 Gbits/sec sender
[ 4] 0.00-10.00 sec 4.67 GBytes 4.01 Gbits/sec receiver

Running iperf3 with multiple connections allows me to utilize the full 10GB though.
3 connects bring me up to ~9Gbps, using 5 reaches the maximum i can get at 9.5Gbps

Disabeling Antivirus/Firewall has no impact.

Any ideas what’s limiting the single connect here? I am rather new to the whole networking stuff so please be kind.

Thanks!
-DR

mutation666 · April 29, 2020, 12:31am

You direct connect the two and see what you are getting?

Airstripone · April 29, 2020, 6:38am

What cables are you using?
What pcie ports are you using? You need a x8 port to not limit the bandwidth.

DReffects · April 29, 2020, 11:00am

No, not yet. Do I need a crossover cable for this?

@Airstripone

Cables:

The installation cables HQ AWG 22 CAT7 cables from Kerpen, rated for 1500mhz
Telegärtner Cat6A Patch Panel + Wall-Sockets
HQ Cat6A SFTP 500mhz wall to pc connection cables

PCIe Slot:
The cards only have x4 connectors and are currently in x4 slots. I thought PCIe x4 is capeable of transmitting up to 2000MB/s?

mutation666 · April 29, 2020, 4:39pm

Back in the day yes these days usually no.

nx2l · April 29, 2020, 4:46pm

but are they working at x4 speed?

carnogaunt · April 30, 2020, 2:33am

By “multiple connections,” do you mean multiple iperf3 processes, or one process with the -P option?

noenken · April 30, 2020, 3:36am

Also are they running at PCIe3?

TrumanHW · April 30, 2020, 7:53am

Isn’t there always some overhead…? Or can you get 1250MB/s via 10GbE …?

Does distance matter…?

Thanks

noenken · April 30, 2020, 8:48am

Yes, something always is… something. That’s what we are trying to find. Because it shouldn’t eat up half the bandwidth.

It’s a factor, absolutely.

Almost, I’ve seen over 1100MBps from my rig to my SSD NAS.

DReffects · April 30, 2020, 9:30am

iperf3 with -P option

Ah, good to know! I’ll test tonight!

Uhm… I guess so? The motherboard specs at least do not seem to indicate that the x4 slots are only x1 speed or something like that.

Yes, PCIe 3.0

I’ve got an overall cable length of 33 meters / 108 feet

I’ll give the direct connection a try asap to rule out cables and switch and get back to you guys

DReffects · May 1, 2020, 11:25pm

I did a bunch of tests today:

Replacement Cables for both systems plus a direct connection between workstation and switch
Initially produced a SINGLE test run that was above the speeds i was getting:

C:\temp\iperf-3.1.3-win64>iperf3.exe -c 192.168.0.11
Connecting to host 192.168.0.11, port 5201
[ 4] local 192.168.0.101 port 50593 connected to 192.168.0.11 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 534 MBytes 4.48 Gbits/sec
[ 4] 1.00-2.00 sec 614 MBytes 5.15 Gbits/sec
[ 4] 2.00-3.00 sec 662 MBytes 5.55 Gbits/sec
[ 4] 3.00-4.00 sec 586 MBytes 4.91 Gbits/sec
[ 4] 4.00-5.00 sec 680 MBytes 5.70 Gbits/sec
[ 4] 5.00-6.00 sec 730 MBytes 6.13 Gbits/sec
[ 4] 6.00-7.00 sec 670 MBytes 5.62 Gbits/sec
[ 4] 7.00-8.00 sec 706 MBytes 5.92 Gbits/sec
[ 4] 8.00-9.00 sec 605 MBytes 5.08 Gbits/sec
[ 4] 9.00-10.00 sec 601 MBytes 5.04 Gbits/sec

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 6.24 GBytes 5.36 Gbits/sec sender
[ 4] 0.00-10.00 sec 6.24 GBytes 5.36 Gbits/sec receiver

Second, thrid and further testing revealed this to be a fluke… still ~4.0-.4.8 Gbps in general.

Direct connection between the two PCs

Does not make any difference it seems

Reset the device, reinstalled latest drivers

Does not make any difference it seems

Additional when running multiple tests i get results like these intermittently:

C:\temp\iperf-3.1.3-win64>iperf3.exe -c 192.168.0.11
Connecting to host 192.168.0.11, port 5201
[ 4] local 192.168.0.101 port 50602 connected to 192.168.0.11 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 91.2 MBytes 765 Mbits/sec
[ 4] 1.00-2.00 sec 283 MBytes 2.38 Gbits/sec
[ 4] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 3.00-4.00 sec 83.5 MBytes 701 Mbits/sec
[ 4] 4.00-5.00 sec 404 MBytes 3.39 Gbits/sec
[ 4] 5.00-6.00 sec 55.6 MBytes 467 Mbits/sec
[ 4] 6.00-7.00 sec 66.4 MBytes 554 Mbits/sec
[ 4] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 984 MBytes 825 Mbits/sec sender
[ 4] 0.00-10.00 sec 983 MBytes 825 Mbits/sec receiver

I then just wait a few seconds and run iperf3 again and results are back at ~4.5 Gbps.

I’ve double checked: Both cards sit in X16 PCIe slots. In the Server in the primary PCIe x16 slot, in my PC in the secondary x16 slot that runs at x4 speed.

I don’t quite know what to do next

The advanced settings are heavily modified by now - are they correct?

Setting (default)
Downshift Retries: disabled (enabled)
Flow Control: disabled (enabled)
Interrupt Moderation: disabled (enabled)
Interrupt Moderation rate: Off (Medium)
IPv4 Checksum Offload: disabled (enabled)
Jumbo Packet: 9014 Bytes (off) (max limit of my 10g switch)
Large send Offload V1/V2 ipv4/6: disabled (enabled)
RSS Queues: 8 (8)
Recieve Buffers: 4096 (512)
Recieve Side Scaling: enabled (enabled)
Recv Segment Coalescing ipv4/6: Disabled (enabled)
TCP/UDP Checksum Offload IPv4/6: disabled (enabled)
Transmit Buffers: 8184 (2048)

Any thoughts on these settings?

Could this be a CPU related issue?
My Workstation runs a ryzen 2700x, my fileserver a Pentium Gold G5400. I see that iperf 3 utilizes ~23-25% of the CPU which would suggest 1 core?!

Thanks!

EDIT2:

WTF… I’ve just did some testing with a long term iperf3 transfer and noticed that the transmit speed cuts out to zero, goes up again etc etc.

I’ve limited the iperf3.exe on the server via taskmanager to use only one core and guess what: transfer becames stable. Also, speed INCREASED:

I start off with no settings in taskmanager:

[ 4] 33.01-34.00 sec 347 MBytes 2.95 Gbits/sec
[ 4] 34.00-35.00 sec 511 MBytes 4.28 Gbits/sec
[ 4] 35.00-36.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 36.00-37.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 37.00-38.00 sec 74.1 MBytes 622 Mbits/sec
[ 4] 38.00-39.00 sec 521 MBytes 4.37 Gbits/sec
[ 4] 39.00-40.00 sec 586 MBytes 4.92 Gbits/sec
[ 4] 40.00-41.02 sec 290 MBytes 2.39 Gbits/sec
[ 4] 41.02-42.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 42.00-43.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 43.00-44.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 44.00-45.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 45.01-46.00 sec 386 MBytes 3.28 Gbits/sec
[ 4] 46.00-47.02 sec 197 MBytes 1.63 Gbits/sec
[ 4] 47.02-48.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 48.00-49.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 49.00-50.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 50.00-51.00 sec 0.00 Bytes 0.00 bits/sec
(Switched taskmanager to let iperf3.exe only use one core and set priority to realtime:)
[ 4] 51.00-52.00 sec 608 MBytes 5.11 Gbits/sec
[ 4] 52.00-53.00 sec 761 MBytes 6.38 Gbits/sec
[ 4] 53.00-54.00 sec 763 MBytes 6.40 Gbits/sec
[ 4] 54.00-55.00 sec 852 MBytes 7.15 Gbits/sec
[ 4] 55.00-56.00 sec 857 MBytes 7.19 Gbits/sec
[ 4] 56.00-57.00 sec 854 MBytes 7.16 Gbits/sec
[ 4] 57.00-58.00 sec 852 MBytes 7.15 Gbits/sec
[ 4] 58.00-59.00 sec 846 MBytes 7.10 Gbits/sec
[ 4] 59.00-60.00 sec 824 MBytes 6.91 Gbits/sec
[ 4] 60.00-61.00 sec 838 MBytes 7.03 Gbits/sec
[ 4] 61.00-62.00 sec 842 MBytes 7.07 Gbits/sec
[ 4] 62.00-63.00 sec 867 MBytes 7.27 Gbits/sec
[ 4] 63.00-64.00 sec 772 MBytes 6.48 Gbits/sec
[ 4] 64.00-65.00 sec 861 MBytes 7.22 Gbits/sec

uhmm… what gives? Why would limiting to one core actually improve performance?

Also: when using CPU cores 1 or 2 i get ~7Gbps - when using either core 3 or 4 i only get the ~4Gbps from before.

carnogaunt · May 2, 2020, 12:23am

Pinning a process to specific core(s) will generally improve the cache hit ratio. Not sure if that would be the main factor in this case.

iperf3 is single-threaded so it won’t ever use more than one core at the same time anyway.

risk · May 2, 2020, 5:56am

Looking at some of your settings. You probably should go back to defaults on most of them.

iperf3 should be able to set socket buffer sizes, at least on Linux. Having larger buffer sizes, at higher data rates means potentially fewer context switches between kernel and iperf3

DReffects · May 2, 2020, 4:04pm

Thanks for the info. I was not aware of that. Is a Windows File Share capeable of utilizing more than one core? Since it seems that my CPU is actually limiting performance (I find that very odd to be frank…) I am thinking about upgrading the G5400 to a i3-9100.

Could you please go into more detail about the settings?

I’ve changed the standard settings since the performance with default settings was only around 1.5Gbps. Enabling Jumbo Packet and disabeling flow control increased it to ~3.5Gbps, the rest of the settings bumped it up to ~4-4.5 Gbps.

DReffects · May 8, 2020, 3:51pm

Just a quick update: I’ve replaced the Pentium Gold 5400 with a i3-9100 CPU and speeds increased from 4,5 to 7,5 Gbit.

Holy sh*t

DReffects · May 9, 2020, 1:02pm

upon further testing i can conclude (at least for my systems):

even a i3-9100 for a fileserver cannot provide enough single core cpu power to deliver 10gbit with an Asus XG-C100C
The Asus XG-C100C is heavily CPU dependent
performance gets worse when one or both of the machines have a hyper-v role installed. the virtual switch has a great impact on performance
in general, i suspect that there are multiple compatibility issues here
the Buffalo switch is not the limiting factor - direct connection improves bw only by aprox. 0.2gbit
for some reason i have to DISABLE Recv Segment Coalescing

My final settings after optimizations are:

Without driver modifications:
~1.5 Gbit

Flow Control off
Jumbo Packet 9014 Bytes
Recv Segment Coalescing disabled
~5,3 Gbit

Interrupt Moderation disabled
~ 6,5-7,5 with dropouts

Recieve Buffers 4096 (512)
Transmit Buffers 8184 (4096)
(Transmit Buffer double in size)
~ 7Gbit

RSS Queues 4 (4) (on both machines)
~ 7,3Gbit

Large Send Offloads disabled on i3 Server, enabled on ryzen 2700x PC
~ 7,5Gbit

IPv4 Checksum Offload enabled on both machines (Standard)
TCP/UDP Checksum Offload enabled on both machines (Standard)
(disabeling makes performance worse)

Thanks for the help guys!

DReffects · May 11, 2020, 9:35pm

Well, matters got worse: network bw degreades to ~300mbit after a few hours. resetting both adapters returns them to 7,5 Gbit for a little while… any idea what’s going on?!

both cards are actively cooled.