So I had a running VFIO setup with a Ryzen 3950X, Asus prime X570 pro, a 1080Ti (passed through) and a 5700XT (host GPU); all running on PopOS 19.04 (upgraded to 20.04 during debugging).
It was working for months (I built and setup the new system in Feb.), until saturday it decided not to work anymore. The VM couldn’t find the PCI bus that was supposed to be passed to it (where the GPU lives) and some digging turned up that the GPU had seemingly disappeared from the PCI Bus alltogether (not even lspci picked it up).
Some experimentation turned up that the GPU works fine with the default nvidia driver on the host, but whenever the VFIO claims the card it gets removed from the IOMMU group, disappearing from the system.
The relevant lines (I think) from the dmesg after boot are (other logs show essentially the same thing):
[ 0.000000] Command line: initrd=\EFI\Pop_OS-7593e5c4-d50f-4c6a-9879-2a0d0990a97f\initrd.img root=UUID=7593e5c4-d50f-4c6a-9879-2a0d0990a97f ro quiet loglevel=3 systemd.show_status=true splash amd_iommu=on iommu=pt vfio-pci.ids=10de:1b06,10de:10ef debug ignore_loglevel
[ 0.000000] Kernel command line: initrd=\EFI\Pop_OS-7593e5c4-d50f-4c6a-9879-2a0d0990a97f\initrd.img root=UUID=7593e5c4-d50f-4c6a-9879-2a0d0990a97f ro quiet loglevel=3 systemd.show_status=true splash amd_iommu=on iommu=pt vfio-pci.ids=10de:1b06,10de:10ef debug ignore_loglevel
[ 0.232148] pci 0000:0b:00.0: [10de:1b06] type 00 class 0x030000
[ 0.232173] pci 0000:0b:00.0: reg 0x10: [mem 0xf6000000-0xf6ffffff]
[ 0.232186] pci 0000:0b:00.0: reg 0x14: [mem 0xc0000000-0xcfffffff 64bit pref]
[ 0.232198] pci 0000:0b:00.0: reg 0x1c: [mem 0xd0000000-0xd1ffffff 64bit pref]
[ 0.232207] pci 0000:0b:00.0: reg 0x24: [io 0xd000-0xd07f]
[ 0.232216] pci 0000:0b:00.0: reg 0x30: [mem 0xf7000000-0xf707ffff pref]
[ 0.232324] pci 0000:0b:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x8 link at 0000:00:03.2 (capable of 126.016 Gb/s with 8 GT/s x16 link)
[ 0.232367] pci 0000:0b:00.1: [10de:10ef] type 00 class 0x040300
[ 0.232384] pci 0000:0b:00.1: reg 0x10: [mem 0xf7080000-0xf7083fff]
[ 0.236132] pci 0000:0b:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[ 0.236132] pci 0000:0b:00.0: vgaarb: bridge control possible
[ 0.255676] pci 0000:0b:00.1: D0 power state depends on 0000:0b:00.0
[ 0.374667] pci 0000:0b:00.0: Adding to iommu group 24
[ 0.374692] pci 0000:0b:00.1: Adding to iommu group 24
[ 0.420278] vfio-pci 0000:0b:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[ 0.438995] vfio_pci: add [10de:1b06[ffffffff:ffffffff]] class 0x000000/00000000
[ 0.459196] vfio_pci: add [10de:10ef[ffffffff:ffffffff]] class 0x000000/00000000
[ 9.896033] vfio-pci 0000:0b:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[ 9.916340] pci 0000:0b:00.0: Removing from iommu group 24
[ 9.916461] pci 0000:0b:00.1: Removing from iommu group 24
In the last part the GPU is removed from the IOMMU group and can’t be found, even by lspci. And that is where the trail ends.
The irony is, when I remove the nouveau module from /etc/modprobe.d/blacklist.conf, it takes control of the 1080Ti, making it available for the host normally.
I’ve also attempted to use a different card (GTX 1060) to pass through, with the exact same result.
The last time I used the gaming VM was a few weeks ago, so I can’t really tell when the issue first arose or what change might’ve caused it. And all the log files already show the error.
I’m at a loss as to where to continue debugging and would rather want to avoid starting from scratch with a fresh OS install.
I hope someone here can point me in the right direction.