Radeon RX 6800 XT GPU passthrough virtio

archiba · June 20, 2021, 12:09am

Hello
I have done a search on the forums and couldn’t find a solution to my problem and was hoping someone could help me and catch what I’m missing.

Here’s my current problem, my gaming guest VM goes black-screen during AMD driver installer and if i wait long enough my host system crash reboots.

When my host system boots back up it uses the guest GPU rather than the host GPU ignoring the virto-pci ids. To fix it back to the correct order a hard shutdown and removing the display cable is needed.
This reminded me of a behavior of the earlier generation with the AMD reset bug but from everything I’ve read on the internet it should be fixed for the 6800 series.

I followed this guide: PCI passthrough via OVMF - ArchWiki right after a fresh install of archlinux and i3

Im running: Linux archlinux 5.12.11-arch1-1 #1 SMP PREEMPT Wed, 16 Jun 2021 15:25:28 +0000 x86_64 GNU/Linux
Here is my lspci -k output:

04:00.0 VGA compatible controller: NVIDIA Corporation GK208B [GeForce GT 710] (rev a1) Subsystem: Micro-Star International Co., Ltd. [MSI] Device 8c93 Kernel driver in use: nvidia Kernel modules: nouveau, nvidia_drm, nvidia 04:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1) Subsystem: Micro-Star International Co., Ltd. [MSI] Device 8c93 Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel 05:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 Subsystem: Samsung Electronics Co Ltd Device a801 Kernel driver in use: nvme 07:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 02) Subsystem: ASUSTeK Computer Inc. Device 87d2 Kernel driver in use: igc Kernel modules: igc 08:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c1) Kernel driver in use: pcieport 09:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch Kernel driver in use: pcieport 0a:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] (rev c1) Subsystem: Tul Corporation / PowerColor Device 2406 Kernel driver in use: vfio-pci Kernel modules: amdgpu 0a:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function Subsystem: ASUSTeK Computer Inc. Device 8808 0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP Subsystem: ASUSTeK Computer Inc. Device 8808 0c:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP Subsystem: ASUSTeK Computer Inc. Device 8808 Kernel driver in use: ccp Kernel modules: ccp 0c:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller Subsystem: ASUSTeK Computer Inc. Device 87c0 Kernel driver in use: xhci_hcd Kernel modules: xhci_pci 0c:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller Subsystem: ASUSTeK Computer Inc. Device 87c5 Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel

Hardware:
AMD Ryzen 7 5800X
PowerColor Radeon RX 6800 XT 16GB Red Dragon

My iommu group script output from the guide:

IOMMU Group 0:
	00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 1:
	00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 2:
	00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 3:
	00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 4:
	00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 5:
	00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 6:
	00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 7:
	00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8:
	00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 9:
	00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 10:
	00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 11:
	00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 12:
	00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
	00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 13:
	00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
	00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
	00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
	00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
	00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
	00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
	00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
	00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU Group 14:
	01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO [144d:a80a]
IOMMU Group 15:
	02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ee]
	02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43eb]
	02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43e9]
	03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
	03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
	03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
	03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
	04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
	04:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
	05:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
	07:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 02)
IOMMU Group 16:
	08:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c1)
IOMMU Group 17:
	09:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
IOMMU Group 18:
	0a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c1)
IOMMU Group 19:
	0a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] [1002:ab28]
IOMMU Group 20:
	0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 21:
	0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 22:
	0c:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU Group 23:
	0c:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 24:
	0c:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]

my /etc/mkinitcpio.conf:

MODULES=(vfio_pci vfio vfio_iommu_type1 vfio_virqfd nvidia nvidia_modeset nvidia_uvm nvidia_drm)
BINARIES=()
FILES=()
HOOKS=(base udev autodetect modconf vfio keyboard keymap block encrypt filesystems fsck)

my modprobe.d/virtio.conf:

options vfio-pci ids=1002:73bf,1002:ab28

robbbot · June 20, 2021, 12:51am

You will likely need to include a link to your VM’s configuration xml as well. Since that’s likely where one of your problems is (at least the black screen one).

Make sure you have:

<kvm>
      <hidden state='on'/>
</kvm>

In your VM’s configuration.

However I’m not sure you have the VFIO settings setup properly if you’re having issues that require fiddling with the cables after reboots.

I know not all motherboard vendors allow switching which is the boot GPU, which is one of the reasons I ended up going with a gigabyte for my setup.

You could be booting from the wrong GPU from the bios/UEFI, and maybe that’s giving you some issues. I’d double check the bios settings, to make sure you have all the right options checked, csm off, svm on, iommu (usually multiple places on, not auto) etc…

I’ve also got something like this in /usr/local/bin/vfio.sh. Note that I have 2 AMD cards, so this just makes sure the one I want gets the driver bound to it.

#!/bin/bash
for i in /sys/bus/pci/devices/*/boot_vga; do
    if [ $(cat "$i") -eq 1 ]; then # 1 here for second gpu, 0 for first.
        GPU="${i%/boot_vga}"
        AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"
        echo "vfio-pci" > "$GPU/driver_override"
        if [ -d "$AUDIO" ]; then
            echo "vfio-pci" > "$AUDIO/driver_override"
        fi
    fi
done

modprobe -i vfio-pci

That’s being included in the /etc/mkinitcpio.conf:

FILES=(/usr/local/bin/vfio.sh)

And lastly in my grub config I have the following added.

GRUB_CMDLINE_LINUX="iommu=pt amd_iommu=on vfio-pci.ids=1002:73bf,1002:ab28 video=efifb:off"

I’m on Arch as well, and my 6800xt’s passed through fine with my Win10 VM.

Hope it helps.

archiba · June 22, 2021, 2:41pm

Thank you for the reply.
Im out of town and havnt been able to provide the vm config or tested the tips and script you provided. I know for a fact i dont have " video=efifb:off" in my grub. Was it only necessary for you to pass in two devices on the rx 6800 xt ? the vga and HD Audio.

robbbot · June 22, 2021, 4:46pm

I have the sapphire version of the 6800xt, so there’s no USB-C on there, so I only pass in the audio / video components ( you need at least those for the windows drivers to work).

If you have the reference version you can also pass in the usb-c from it as well with no issues.

agd · September 13, 2021, 11:55pm

@robbbot I’m attempting this with the reference version over here and having some issues if you’d mind taking a peek!