I can reach around 30Gb/s on a linux bridge network from my 40Gb/s connect-x 3 NIC… There is some CPU overhead but nothing shocking when you use the virtio driver. A big advantage with SR-IOV would be to use RDMA but most 10Gb NICs don’t support this any way.
Virtio is not emulated but paravirtualised, i.e. guest commands get translated to device commands and not full emulation. Other drivers do perform much worse afaik.
As a point of reference, using a software switch is significantly faster than using SR-IOV on my 10Gbe card.
This could be deceiving. Did you bench with multi-stream or single stream. As far as I can recall, it is very hard to reach 10Gb with single stream. Basically CPU is not strong enough. Although you could enable something like jumbo frame to optimize it, I don’t need to do anything in the case of SR-IOV.
That is certainly correct, the question is do you have a NAS at home that is fast enough and an application that benefits from it.
I’ve used ConnectX-3 Pro for several years until they were to old and SR-IOV stopped working with Windows VMs.
I switched to vNICs with Virt-IO for SMB and iSER for block storage.
I connected the LUNs at the host level and assigned them via Virtio-SCSI and io-Thread.
This is with VirtIO in a Win11 VM, what limits is my NAS
These are old results.
When I switch my NAS to PCIe5, I’ll test again to see when it makes a difference.
I have now switched to CX-5 and am using SR-ioV again, but I would have to use benchmarks to see the difference.
SR-IOV makes sense on a server with several VMs, but at home with enough CPU power and rarely simultaneous data transfer, it is not necessary.
That’s really bad. MS side or Nvidia responsible for EoL troubles?
I used an old Marvell 10G chip (AQ107) for 3 years and some SOHO Netgear switch and it was ok for the stuff I used it for. But the chip wasn’t supported on Win11, so expect lacking support for Windows at some point (Win13 or whatever it’s called then). Linux kernel never forgets, so AQ107 drivers were no problem.
If this is your first 10G chip, go for it. on-board NIC via chipset lanes is super efficient for PCIe lanes and you get both 10G and the full lanes of 890 series.
Just keep in mind that 10G is around for some time now and stuff is moving towards 25G SFP as we speak.
I got a lot out of my 10GBase-T stuff for years. Until I eventually wanted less power consumption, better latency and way cheaper switches. So long-term, RJ-45 jack might be a dead end depending on what you do.
But can’t argue against 10Gbit no matter what, if the alternative is 1-2.5Gbit networking. 10G is a gamechanger 
I think EOL, it still works with Linux VMs, at least the last time I tested it.
Nvidia stopped maintaining the linux drivers (OFED). The mailbox drivers that are mainlined in the kernel still work fine but they only support SR-IOV linux guests, not windows. The OFED drivers should support windows guests but are only compatible with pretty old kernels (maybe debian 11/RHEL7 or so).
I don’t see RJ-45 moving above 10G any time soon. SFP is much superior but I’m not holding out for mainstream/prosumer hardware to offer it.
@BlackCat have you considered adding an extra NIC, e.g. intel E810/E830 offer 2x25Gb SFP ports for about 200 euros new. Or if 10Gb suffices an older intel or mellanox nic on ebay? What will be on the other side of the NIC? I presume some NAS?
If you are interested in things like ECC, SAS ports, and IPMI the Asus W880 board looks very interesting. It ‘only’ has 2x2.5Gb NICs but the other goodies might make it worth the price (very similar to ProArt, at least here in europe).
10G NICs are nice but they are not worth hundreds (they buy the chips for a tenner I think), so if that’s the only consideration I’d go for the separate NIC for the money. It just depends on your plans for the other PCIe lanes.
300€ for a 10Gb Marvel chip is a complete rip-off!
If they would give you an E830, I would still not find it a good offer, but at least it would no longer be a scam
That’s the plan! The board has a couple PCIe4 x4 slots, so aiming to fit one with an E610-XT2 at some point. Not the cheapest option, but I’m a fan of the low power, no pun intended. 
I just want to start a rant about how a waste of money the E610-XT2 is, until I realized that you can get five of them for 399€
https://geizhals.de/intel-e610-xt2-10g-lan-adapter-e610xt2m5-a3531613.html
But I’d still go for 25 Gb nics, though.
Four PCIe lanes for 10 GB is a waste, and if you need 10 Gb, you’re better off with 25 Gb.
Yeah you really need them for cheap cause that 10+ port 10GBase-T switch is eating the rest of your money and power for sure
I do run a ConnectX-4 25Gbit in a PCIe3.0 x4 slot. One link is wire speed (2.87GiB/s of actual data), no latency spikes (actual x4 CPU lanes slot on that board). Didn’t check with both ports yet. But 4 lanes seems fine and most consumer boards have some kind of x4 slot. Chipset lanes are bad latency in my experience, but should work as well.
10Gbase-T NICs use a lot of power. Had to actively cool my Marvell AQ107 and Intel X550 and the switch was getting really hot too. I regret not getting 10Gbit SFP+ card for lower power and easier upgrade path to 25 and 100Gbit. Maybe go with on-board RJ-45 now and upgrade to SFP+/SFP28 with that x4 slot later. Sidegrade with two similar NICs doesn’t really make sense.
yes they run hot
Mellanox CX5
mlx5-pci-0100
Adapter: PCI adapter
sensor0: +64.0°C (crit = +105.0°C, highest = +65.0°C)
Marvell AQ107
eno2-pci-0a00
Adapter: PCI adapter
PHY Temperature: +72.0°C
MAC Temperature: +72.0°C
The E610-XT2 uses about half the power, I find that kinda cool.
This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.