Tldr; buy the Mellanox X-4 NIC card instead of the X-3 to save yourself hours of frustration
Mellanox Connect-X3 was getting slow network speeds (only 10% of it’s expected capacity) maximum 100MB/s instead of 1,000 MB/s transfer speeds. Tried everything I could think of, every driver and BIOS update, changing PICE lanes, removing RAM, ALL the configuration settings in Windows to get the X-3 to actually transfer files at 10GBE speeds. Nothing worked.
Just save yourself the hassle
Just buy the Mellanox Connect-X4 second hand, at time of writing are about $USD 30 + shipping.
Specific Part:
Mellanox Connectx-4 LX 25GbE Single-Port SFP28 PCIe Network Adapter Card
SKU: MCX4111A-ACAT
The X-4 is the first generation which supports the newer Win-OF2 drivers
It just works like it’s supposed to 10GBE speed. Drivers can be downloaded here:
network.nvidia.com/products/adapter-software/ethernet/windows/winof-2/
Setup:
Windows 10 Pro, Asus x570 Aorus Ultra MB, AMD AM4 5950x, NVIDIA 3090,
What’s really bizarre is that the X-3 was registering as a 10GBE device, with x8 lanes of PCIe in Windows, but would just NOT do file transfers or benchmarks at anything more than 10% of the speed it should have been able to. AND the X-3 works absolutely fine in the TRUENas Scale machine, on an AM5 mb.
Anyways, hopefully Google indexes this and it saves the next person wasting hours of their life. <3