I’m looking to replace the 10Gb NICs in my homelab. My only requirements are ESXi 8 compatible and I’d like to have the full enterprise alphabet soup like RDMA, SR-IOV, VXLAN, RoCE, iWARP, etc. I believe my choices are QLogic Fastlinq, Mellanox (4th gen or later), and certain Broadcom devices.
I have the impression Mellanox is the premier technology for RDMA et al on ESXi and the easiest to get up and running. Am I on the right track here?
Mellanox has been the most feature rich, and might still be for the cheapest cards you can get off Ebay. If you cant find a MCX512A-ACAT used for cheap then you could either buy it new for around $250, or try this one from Intel:
https://www.intel.com/content/www/us/en/products/sku/189760/intel-ethernet-network-adapter-e810xxvda2/specifications.html
It should support everything feature wise that you want and is fairly cheap, a bit cheaper new than the Mellanox is new. The Intel one is a PCIe 4.0 NIC too, which means you can use both ports at 25gb in a chipset connected 4x slot if you wanted to. The PCIe 3.0 cards would bottleneck with both ports at max speed in a 4x slot.
I have two of those in service, they work well (at least in Debian 12 and Win10).
Chelsio and Broadcom are also options you probably want to look into
The newest Chelsio cards don’t have full support for all the techs the OP wants. There are limitations in VXLAN fort instance, and cost 3x as much as both the Intel and Mellanox Options with the same features.
VXLAN is supported and all the other ones too?
They also don’t seem to cost 3x more as you claim?
Ill try and look up the feature matrix I was looking at yesterday again. For some reason it showed you had to step up to the T6 to get all the features the OP listed instead of the T5. But looking at this other one direct from Chelsio it seems even the T54 should support them all:
Strange, but Im not sure how much we can trust the chart from Chelsio given they have missing checkmarks for Mellanox in at least 3 categories.
If that chart there is correct and the T5 does have all the necessary features then it would seem like the best option to go with.
I have T540-BT myself, it was super cheap. And has all (?) the important features: https://www.chelsio.com/wp-content/uploads/resources/t540-bt-pb.pdf.
You may also want to check their up-to-date driver docs, I think this is the most up-to-date: https://service.chelsio.com/store2/T5/Unified%20Wire/Linux/ChelsioUwire-3.18.0.1/Chelsio-UnifiedWire-Linux-UserGuide.pdf
The card firmware is dependent on the driver version you use, at least on Linux the driver will update the firmware at boot, so newer driver may mean more features.