Passthrough with 4070 Super host and RX 580 guest: Is it possible?

Operating Systems & Open Source Virtualization
1

I have a new system configuration after buying a 4070 super but I’m having a really difficult time getting the following things to happen:

  1. The AMD card has been rigidly stuck to using the amdgpu driver instead of the appropriate vfio-pci like I expect. This is only the case with the VGA/Graphics output part of the gpu driver, and does not seem to be a problem for the audio part of the hardware (it correctly reads vfio-pci when I check the output of lspci -vnn.)

  2. When attempting to use this configuration, my desktop (currently using a X11 compositor) will randomly crash during the boot of any graphics program such as steam. It basically seems that my two gpus are competing with one another.

Has anyone had this issue with the hardware seemingly not respecting vfio-pci assignment? Is this a particular limitation of the nvidia proprietary drivers on linux?

2

I configured my own nvidia host and amd guest setup a little while back. From what I recall, you might be able to take the following approaches:

  1. Blacklist the AMDGPU drivers. If the AMD GPU is only being used by the VM, this might be reasonable.
  1. If you’re using systemd-boot or grub you might also be able to assign the pci ids to vfio-pci in the kernel boot options. Instructions vary based on the boot manager you’re using, but in general you should be able to add something like this

Kernel Boot Options: vfio-pci.ids=1002:73df,1002:ab28

The PCI IDs are going to be different for your card. You can find what you need by using lspci again, something similar to this:

lspci -nn | grep -E "RX"

That should get you output similar to this:

0c:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [Radeon RX 6700/6700 XT / 6800M] [1002:73df] (rev c5)
0c:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] [1002:ab28]

You’re looking for the numbers at the end of each line associated with your GPU. In my case, that’s 1002:73df and 1002:ab28. Once you’ve got those, add the kernel boot option I noted above and then perform any steps necessary to update the boot manager. I will leave that up to you.

Reboot and hopefully that does the trick.

  1. Dynamically unbind the AMDGPU drivers and bind the vfio pci drivers right before the VM starts. You can find instructions here if you want to try that route.

Just do a search for “blacklist” on the page and that should take you to the relevant section.

Oh one more thing, if you get past that issue and find that you can only start and shutdown the VM once before the GPU flakes out, search the forums for the AMD or Radeon reset bug for more info. Fortunately for your RX580, there is a fix. Unfortunately for me, there’s not a fix for my GPU (some 6700XTs work fine, others do not).

3

Thanks for the rundown.

I’ll give that all a try this weekend when I get some spare time. What’s not helping is that I am running on a silverblue machine so some of these driver blacklists I have approach with minor alterations.

As long as people have had experience with this though, I’ll probably get there in due time.