I’ve tweaked my vfio passthrough setup to a more “traditional” configuration to try to avoid a few bugs/issues I was running into.
My setup now is a radeon card for the host and a RX580 early bound to vfio-pci (via initramfs) passed through to guest. The guest is a Windows VM and Linux VM (Debian Buster).
I can boot/restart/shutdown the Windows VM as many times as I please and everything works great. However, the Linux VM can be restarted or shutdown precisely once. Any subsequent attempts to boot it (or any other VM with passthrough) will give a “Unknown PCI Header Type 127” error.
The only way to boot any VM after this is restarting the host.
Clearly Linux is doing something differently during shutdown with the card to Windows. Is this a known issue? Is it the old “Reset Bug”? or was the reset bug host side in which case windows would have been hit too? Any suggestions for how to go about troubleshooting this?
fwiw the GPU is in its own IOMMU group and I pass through both audio and video interfaces on the card.
Edit: Seems my search-fu is lacking, didn’t see this thread but it was linked in gnif’s 5700 reset fix thread. So it looks like this is not just an isolated issue with my setup.
Debian’s amdgpu firmware isn’t always current, even in testing/unstable. I’d suggest pulling ‘firmware-amd-graphics’ from backports, which is currently about 6 months newer than the version shipping in Buster.
Yes I’m familiar with backports, I’m running the latest firmware from there, 201907 I believe it was. Same issue remains though.
It seems strange that windows is able to shutdown in a way that leaves the card available to boot the windows VM or Linux VM again, yet the Linux VM will render it inoperable in all VMs once shutdown.
Makes me wonder what Windows or the amd driver in Windows is doing differently to amdgpu in Linux.
Also interesting is I’ve seen many threads from people having the opposite problem. Card works with restarting Linux but not Windows.
It seems strange that windows is able to shutdown in a way that leaves the card available to boot the windows VM or Linux VM again
Agreed, which is also why I’m skeptical of AwesomeSauce’s claim in that other thread that it’s a hardware problem. If Windows can do it, the hardware is capable of resetting when handed off.
Wish I had more to offer. The number of dangling threads on this subject with no resolution is kind of depressing.