[Solved] GPU Passthrough suddently stopped working on Windows 10 VM :(

Operating Systems & Open Source Virtualization
, ,
5

Good news everyone! As Todd Howard would put it, It Just Works!™
Bad news is I changed so many things I’m not exactly sure what fixed it…

Here are the relevant bits if anyone comes across the same situation:

Arch PCI passthrough guide

I followed the PCI passthrough via OVMF - ArchWiki step-by-step carefully again (this time I payed special attention to #Loading vfio-pci early)

Loading vfio-pci early

Since Arch’s linux has vfio-pci built as a module, we need to force it to load early before the graphics drivers have a chance to bind to the card.

at /etc/mkinitcpio.conf

 MODULES="vfio_pci vfio vfio_iommu_type1 vfio_virqfd nouveau nvidia_drm nvidia"
 HOOKS="base vfio udev autodetect modconf block keyboard keymap encrypt openswap resume filesystems fsck"

at /etc/modprobe.d/vfio.conf

options kvm ignore_msrs=1
options vfio-pci ids=10de:2486,10de:228b disable_vga=1 disable_idle_d3=1
softdep nouveau pre: vfio vfio_pci
softdep nvidia_drm pre: vfio vfio_pci
softdep nvidia pre: vfio vfio_pci

* Remember to regenerate the initramfs with $ mkinitcpio -P linux

kernel parameters

As Pollomoolokki pointed out, there were some inconsistencies with my parameters. Here is the current working version:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt rd.driver.pre=vfio-pci kvm.ignore_msrs=1 vfio-pci.ids=10de:2486,10de:228b disable_vga=1 disable_idle_d3=1 [...]
(keep in mind I’m using an AMD cpu and your vfio-pci.ids wiil most likely be different than mine)

VM xml

Here are the important sections, including the nvidia code 43 workaround. Other than that not much happening here, my GPU has an UEFI vBIOS so loading the rom wasn’t needed after all (YMMV, a lot).

[...]

  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='randomid'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <cpu mode='host-passthrough' check='partial' migratable='on'>
    <topology sockets='1' dies='1' cores='4' threads='2'/>
  </cpu>

[...]

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
</domain>

[EOF]

Acts of desperation

I moved everything around hardware-wise, reseated gpus, cables, tried different slots and combinations (which in hindsight doesnt make much sense since the PCIe slot 2 is not isolated IOMMU group wise) ultimately everything stayed the same, but who knows, maybe something wasn’t connected properly)

Alright, thank you for your help! My generic HDMI switcher arrived just in time to be put to use. For now I’ll be using it and a sw kvm switch for keyb/mouse but I will definitely keep an eye out for the store.level1techs kvm I hear so much about :smiley:

3 Likes