I have two systems, each with an Intel x540 T2 10GB NIC:
System 1:
X1900 (AMD Threadripper)
32GB RAM
RAMDisk (for testing purposes)
Windows 10
System 2:
I9 7900x (Intel)
32GB RAM
RAMDisk (for testing purposes)
Windows 10
I have set Jumbo frames along with the rest of the usual things to improve speed.
Threadripper Machine:
Read/Write speeds on the Threadripper machine using Crystal Disk Mark. Tested by mapping a Windows share of a RAMDisk set up on the I9 7900x. Read/Write speeds are perfect.
Intel Machine:
Read/Write speeds on the Intel machine using Crystal Disk Mark. Tested by mapping a Windows share of a RAMDisk set up on the x1900. Read speeds are great but write speeds are not.
Read/Write speeds should be maxing out here. Previous to getting these Intel NICS, I had two Mellanox Connectx2 SFP cards, I experienced the same slow write speed. I really do not know what the problem is. The issue is definitely not the cards, as concluded above, it’s happened on two separate pieces of hardware.
There is no switch involved, the two machines are directly connected.
Am I right in thinking the problem is on the AMD machine, as that’s where it was writing too
on the Intel machine test, in regard to the slower write speed, no?
Well, that is irregular. iperf has arguments to set the amount of simultaneous streams for higher bandwidths as well.
Is a switch in the network path or is this NIC to NIC?
Are there many other PCI-E devices in each system? (All lanes in use)
You would need to read your motherboard manual as to which M.2 and PCI-E slots go to the chipset or CPU.
The next place I would look is the TCP offload settings, as the fact that the results change based on which system is the server even though it’s a bi-directional test.
Try a new install of Windows 10 2004 on the threadripper system (delete all existing partitions) if you have a spare drive to boot from, or try a linux live USB distro as iperf is cross platform and hope that a compatable NIC driver is in that distro. The NIC’s are native RJ-45 8P8C and not SFP?
RJ45 yes, the M.2 drives go to the chipset.
I have tried to install the 2004 update allready on the threadripper build, gets to the end, system restarts, and doesnt install the update, . It’s already installed on the Intel, as of last night.
In-place Windows upgrades are not the best approach as you can have problems like that, you’re best of to boot from a Windows install USB then delete the partitions and start a new install. Of course after backing everything up to an image with something like Acronis True Image so you can go back if this does not work.
Something is not right with that threadripper system.
Act on my suggestions as you will, the fact that the results change based on which system is the server even though it’s a bi-directional test does not make any sense.
So I suspect it’s a problem with the driver, perhaps the TCP offload settings.
The fact that an in-place upgrade fails on that system means there is something wrong with that Windows install.
Apologies, i was not saying you were wrong, i just wanted to clarify, from the results i have show so far, that it is indeed the threadriper machine at fault.
Best of luck to you, if it does not resolve the issue, it will at lest cross off some possibilities. The only problem is will be where to go to from there… swap the NIC’s over and retest or get a third PC and test between it and the Intel system then between the third system and the threadripper and compare results.
Why are the buffer sizes different when they have been set to the exact same in advanced settings?
FYI, it’s defo not the NICS, the same thing happened when I had two Mellanox Connectx2 cards directly joined together. Exact same problem on the same machine.
Could be two different SKU’s with different hardware or firmware, check the cards physically to see if they are identical, I know the X520 has upgradable firmware.
That reminds me, you mentioned configuring jumbo frames, it might pay to check your MTU. run cmd then run
if an MTU of 1473 or 1501 fails then jumbo frames aren’t working.
if either an MTU of 8972 or 9000 returns a ping then jumbo frames are working. (assuming a 9000 MTU jumbo frame)
You will also need to try both ways with 10.10.10.2 or whatever it is.
