[SOLVED] NVIDIA 4070 passthrough to windows guest

Operating Systems & Open Source Windows
, , , ,
1

I’m trying to passthrough an NVIDIA 4070 (specifically an MSI Ventus x2) to a windows VM using proxmox as an host.
I’ve done all the usual steps and more, I’m all out of ideas.

the basic configurations for passthrough should be correct, since some gpu/OS combinations are working correctly.

  • passthrough the 4070 to a linux guest WORKS
  • passthrough another video card (1060) to a windows guest WORKS
  • passthrough the 4070 to a windows guest DOESN’T WORK

base setup (working for 1060):

  • added parameters to grub:
    amd_iommu=on iommu=pt video=vesafb:off,efifb:off pcie_acs_override=multifunction
  • unbinded the gpu at boot
    echo "options vfio-pci ids=10de:2786,10de:22bc disable_vga=1"> /etc/modprobe.d/vfio.conf
  • blacklisted drivers
    echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
    echo "blacklist nvidia*" >> /etc/modprobe.d/blacklist.conf
  • the VM is setup with OVMF BIOS and q35
  • the VM is hidden to windows, adding args: -cpu 'host,-hypervisor,kvm=off' to the config in proxmox
  • using a patched vBIOS rom: hostpci0: 0000:07:00,pcie=1,romfile=1060.rom,x-vga=1

When it comes to the 4070 I’m getting a black screen when starting the windows VM, not even the proxmox loading screen.

what I’ve tried (without success):

  • using a different PCI slot in the motherboard
  • using the default BIOS rom (linux works without passing a romfile at all)
  • using A LOT of vBIOS roms, both patched and unpatched

in detail I tried:

  • 3 different unpatched roms from techpowerup (all from MSI models of the card)
  • the same 3 roms, but patched (removed the header before 0x55)
  • the original card rom, dumped with GPU-Z
  • the patched card rom, dumped with GPU-Z and with the header removed

any suggestions? something obvious that I’ve not tried?

2

When it comes to the 4070 I’m getting a black screen when starting the windows VM

A black screen where? A physical monitor or over SPICE remote display?

Is one of the HDMI/DP outputs of the 4070 plugged into a monitor, or using an HDMI dummy plug?

Without anything plugged into the display outputs, my 4080 did exactly the same with a Windows 10 VM - except it wouldn’t boot at all as far as I could tell, but passthrough to Linux VMs worked fine. I use a HDMI dummy plug in it now.

3

All the tests were done with a physical monitor connected to the card with a DP cable
I could try with a dummy plug but in the end I would like the setup to work with a physical monitor

I was also thinking that the 1060 drivers present on the windows installation might be messing up with the card but I excluded this because:

  • I think the drivers are the same generic Nvidia drivers (is this right?)
  • I don’t even see the proxmox boot logo so this is pre windows fully loading, not sure if the drivers are loaded yet

Does these assumptions make sense?
I also tried to install another copy of windows on a fresh VM and I get a black screen from the beginning (booting from the installation media)

4

IT’S WORKING!

out of desperation I re-tried some of the steps I already did this weekend.

I noticed that at a certain point (I think near the beginning) I tried adding a random vendor-id/device-id to fool windows since I’ve seen it mentioned in some guide.
something like: hostpci0: 0000:07:00,pcie=1,romfile=1060.rom,x-vga=1,vendor-id=0x1717,device-id=0x4242

tonight I just removed those ids and it works, without even needing a rom!
this was obviously the FIRST test I did at the beginning of it all… :frowning:

I also did 2 more things that I don’t think mattered but are worth mentioning for any future reader:

  • I switched back the 4070 to the primary pcie slot (and the 1060 to the secondary)
  • “waited more time with a black screen”

my hypotesis is that during the various tests the windows vm has gotten into safe mode and was taking a long time to boot. If this happened in combination with one of the many tests I might have considered it failed even if it was booting correctly.

now that it’s working it takes about 30s to boot (i waited multiple minutes the first time it worked today).

Another thing that I noticed is that with the 4070 no proxmox boot screen shows up.
the screen stays black until the windows login appears.
Since the proxmox boot screen usually takes only a couple of seconds to appear (with the 1060) this led me to believe that is was failing.

I can confirm that I made the following configurations work:

  • the 4070 in the main pcie slot, without any romfile
  • the 4070 in the main pcie slot, with other romfile (working with it’s own rom extracted with gpuz and unpatcher, also working with the founders edition unpatched rom)
  • the 1060 started working also without the romfile in windows (I’m speechless as this wasn’t the case before, maybe a newer version of the nvidia driver fixed this?)

not sure what I can learn from this one, but maybe this post might be helpful for other or future me.

5

For passthrough to work, usually you don’t need any ROM file with nvidia UEFI complient GPUs starting from a GTX 670 series onwards (these are 2012 - 2013 GPUs :stuck_out_tongue_winking_eye: ).