Looks like people are trying a lot of different things, but I finally got it working in a hacky way on my 9070xt by doing a hard remove / rescan of the PCI between boots of a win11 ltsc VM (latest updates are paused because of some other unrelated issue with “unsupported cpu”). Info dumping for my own notes and maybe others.
My system is a Gigabyte Gaming 9070 XT in my mobo (MIS MAG x670e)'s first slot and an RX 6600 for the host in the mobo’s 4x slot, running Kubuntu 25.04.
Either way though I found my GPU fails to boot with the VM if amdgpu isn’t bound at least once. So I removed all vfio-ids config and just have my 9070xt get bound to amdgpu. I exclude it from DE with udev rules on the devices in /dev/dri. (one quirk is is the login screen in kubuntu still uses it but once logged in nothing uses it).
So once amdgpu initializes it or something, it can boot, but when the VM is shutdown, amdgpu will crash on rebind. Even if I avoid the crash by binding it to vfio-pci instead, it can’t boot again because amdgpu needs to do something to it I guess. The only solution I found was a hard reset of the card like suspending to ram or removing the device, then amdgpu can bind without crashing.
So I made a qemu hook script that on start unbinds from amdgpu, binds to vfio-pci, and sets up the vfio-pci ids so that when the VM is stopped it doesn’t rebind to amdgpu and crash.
Then after shutdown the script removes the vfio-pci ids, unbinds from vfio-pci, and resets the card with an echo to /sys/bus/pci/devices/…/remove and then /sys/bus/pci/rescan. Then it just auto rebinds to amdgpu fine.
I leave it on amdgpu because vfio-pci doesn’t seem to manage power/fans right, it was getting toasting while on amdgpu it just sits cold with the fans off.
I also had to do the hyperv id and kvm hide thing or the card would just show that glitched out stuff. I have the qemu commandline args -fw_cfg “opt/ovmf/X-PciMmio64Mb,string=65536” in the XML which I assume are needed but haven’t explicitly tested with it off.
But I didn’t have to do anything with rom resizing, rom bar on/off, etc. My system sets the max sizes and the AMD drivers in windows seem to be fine with it as well. Haven’t done a ton of testing though but benchmarks seem fine.
Dunno if there is weird side effects that remvoe/rescan might have but it works pretty well to be able to start/stop the VM without having to hard power off my host OS all the time.