I installed a ConnectX-3 pro dual port NIC in my PC, and want to pass through a SR-IOV virtual device to my Windows 11 VM. With both the Microsoft driver (auto installed) and the Mellanox driver I get a code 43 in the device manager and the NIC is not usable.
When I pass through the physical device (not using any virtual devices), everything works as expected. On an Ubuntu VM passing through the virtual device also works. It is just the combination of a virtual device with Windows that does not work.
My system:
Asus proart x670e mainboard
SR-IOV enabled in the bios
IOMMU enabled
reBAR on/off both give the same result
Fedora 38
Does any one have experience with such a setup or ideas to debug this?
I am guessing a bit here, but I’d disable SR-IOV for starters. To my understanding that enables “multi-tenacy” of a single SR-IOV capable device for multiple VMs and I believe what you’re trying is to pass the entire device to the VM (which then might want to do SR-IOV with nested VMs…)
SR-IOV might actually be a better fit for what you’re trying to do, but AFAIK it requires matching different drivers and vendors might actually exclude desktop operating systems like Windows11 from working with them, asking for Windows server editions, extra licenses and ideally support contracts… personally I believe the lack of SR-IOV popularity can be traced to vendor greed.
Pass-through involves telling the host to leave a device alone and with SR-IOV you’re essentially preparing the host to break it into pieces to parcel out individually: at that point it’s ready to be managed by the host, who can no longer pass all of it along to a single VM.
Now your Fedora might actually not manage the ConnectX and then pass-through might still work. But if it does, well then I’d say you’re lucky not to see reboots the moment a VM tries to grab it.
So you want to make sure that Fedora ignores the hardware either for lack of a matching driver or by blacklisting it. Then you need to enable IOMMU pass-through generally on the kernel, because it’s not a default etc.
If it’s your first pass-through, you may want to try with some other device first or read up on GPU pass-through materials, which share some of the issues.
I’m not exactly sure if I remember this correctly. I think I once read something on servethehome or some other forums or here.
For passing through a virtual nic on mellanox connectx you need to install the mellanox ofed drivers, not the inbox drivers in the kernel.
But apparently there were some compilation errors with the mellanox ofed drivers for connect-X3 and newer kernels because the drivers are very old, and it might be that someone made some patch for it. But I don’t remember for what windows version it was.
Thanks a lot! It seems like this is exactly my issue. I’m on fedora with kernel 6.2, so I’m not too hopeful the patches would work. I can look or ask around on the servethehome forums if there are any updates.
It seems like I’m probably still stuck with using a bridged ethernet device then.
I guess the drivers probably don’t compile on kernel 6.2 yet.
I have the latest mlx ofed drivers installed for connect-X4 LX, and they compile on kernel 6.1.12 but not on kernel 6.1.20 for example.
I’m still on debian bullseye but using a backported source kernel.
I finally could get around to try and compile the drivers on fedora 38, but without success. I tried the RedHat 8.7 and the fedora 30 versions, but in both cases I get a message that python and python-devel packages are missing. I have python3 and python2 installed however…
I guess the legacy drivers with a cutting edge distro are just not a good combo, and probably not worth the effort to try and make it work.