I am trying to find a detailed explanation for the behavior I have observed regarding my Intel 10Gbit NICs with several servers at work.
The controllers are X540-AT2 (rev 01) made by Intel and both machines have the same NICs. Machine 1 is running CentOS 6.9 and has two Intel® Xeon® CPU E5-2670 v2 @ 2.50GHz. Machine 2 is running CentOS 7 and has dual Intel® Xeon® CPU E5-2660 v3 @ 2.60GHz. Using iperf3 on our local 10Gbit capable network, Machine 1 and Machine 2 could only ever achieve ~4Gbit to another 10Gbit machine on the same switch. Other machines can communicate with one another at full 10Gbit speeds. ifconfig shows all NICs properly configured and recognized at correct speed and duplex.
Perplexed we restarted the machines, no change. We checked configuration, we checked cables, everything seemed correct. Only while tracking down another issue did we realize that VT-x was disabled on both machines. We enabled VT-x on Machine 1 and retested just for the sake of doing so, only to notice that we now had 10 Gbit speeds. The same was true on Machine 2, enable VT-x and we have 10Gbit speeds.
So we solved the problem of our bad network speeds, but I am confused how VT-x settings have any impact on the performance of our NICs on physical machines. Neither Machine 1 or Machine 2 are running any virtual machines. I am left with speculating that hardware may use VT-x instructions to achieve higher performance but I can’t find any proof of this online. Does any one have an explanation of how this setting could interact with the NICs. We should have VT-x enabled anyways, so it’s fine if I don’t know why, but I really want to know WHY.
Can you compare the kernel parameters for the NIC’s with and without VT-x enabled? Is it possible that the kernel is reacting to the presence of VT-x and treating the NICs differently?
My only reason for asking this is that getting full 10Gb speeds on 10GbE NIC’s can require some tuning. The speeds you were getting before are indicative of a NIC that has not been tuned.
That is our most likely hypothesis at this point as well; however, I can say that the kernel related stuff in grub is the same with and without VTx enabled and we made no other configuration changes other than enabling VTx in the bios. I can check other places and see if I see any changes.
Something that I did think of that might be relevant is that for whatever reason, VTd was enabled in the bios of both machines while VTx was disabled, maybe the issue is not actually if VTx alone is enabled but rather if VTx is disabled while VTd is enabled how the kernel / hardware handles the NIC is different.
Thanks for the suggestion, I’ll check it out. Luckily the hardware / bios config that is working is the desirable one in our use case. Is there a specific file or files you would suggest checking to verify this hypothesis. I am not a system administrator by training so I’m not certain where to look for my info in the OS for this sort of question besides grub kernel params.
Others here can probably give you a better idea, but sysctl -a 2> /dev/null | grep ^net > somefile will give you the network-related kernel stuff (print it into ‘somefile’).