VFIO Pass through working on 9070XT

[Degrowth]

Got this working for a Red Devil 9070, with some help from the Proxmox forums!

I didn’t do anything on Proxmox itself, other than commenting out the amdgpu line in the '/etc/modprobe.d/blacklist.conf file.

I did update the kernel and mesa on the VM that I passed the GPU through to, to their 6.13 and 25 versions, respectively.

We made a script similar to the one posted here, but edited for the 9070:

#!/bin/bash

echo "0000:03:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
sleep 2
echo "0000:03:00.0" > /sys/bus/pci/drivers/amdgpu/bind
sleep 2

As long as I run this after VM shutdown and/or before starting the VM again, I can (so far) reliably restart the VM without having to restart the Proxmox host.

I did save the output of journalctl -f when I start the VM, not sure where to post that, or if it’s helpful for anyone at all, but as of now – it’s working (although the VM doesn’t display the card name correctly, it definitely works)

[BuzzBumbleBee]

A quick update after a few days or running the 9070XT with a VM :

As long as I unbind amdgpu before launching the VM everything has been super stable (same as my 6800XT) played Helldivers2 for a few hours over the last few nights with no issues at all.

Iv had one host hard reset on VM launch (but i did see that every so often with the old GPU so i think thats totally unrelated)

[Steve087]

Is this still working? I had it up and running yesterday and today it’s not working anymore. When I start the VM the GPU outputs the boot menu and as soon as the Kubuntu VM loads the GUI my screen goes black.
What changed for me is the MESA version which went from 25.0.1 to 25.0.2 and I have no clue how I can downgrade to see if this is really the issue.

[GlemSom]

@BuzzBumbleBee Super nice you got it working

Can you share your started qemu commandline please?
And can you share your kernel arguments ?
Can you do a complete Windows 11 install, WITHOUT using vnc/spice network? (Like using ONLY the 9070xt, without the AMD driver loaded yet… and survice VM reboots and such?)
Can you enter the OVMF BIOS using the 9070XT only ?

By default, I do not have a PCIe root - which seems to cause Windows to not wanting to load the driver (resulting in error 43).
If I do use a PCIe root, my host crashes when the Windows AMD driver loads.

Tried both kernel 6.12 and 6.14 - which same results.

Funny enough, using a Linux guest works just fine…

I used to have an old nVidia GPU, which works flawless in the same system… so, sadly the only difference is the Sapphire 9070xt card

[vvk]

Have you done the steps that are posted here The state of AMD RX 7000 Series VFIO Passthrough (April 2024) ? They’re for 7000-series, but tl;dr

• blacklist amdgpu
• do not use correct BIOS ROM (instead dummy ROM) or uncheck the ROM option on VM manager so the card won’t double-POST
• hide KVM and set vendor id

[BuzzBumbleBee]

Iv just done a full pacman update and my VM is still booting as expected

[BuzzBumbleBee]

<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
  <name>win10</name>
  <uuid>########</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">14336000</memory>
  <currentMemory unit="KiB">13312</currentMemory>
  <vcpu placement="static">16</vcpu>
  <iothreads>1</iothreads>
  <cputune>
    <vcpupin vcpu="0" cpuset="8"/>
    <vcpupin vcpu="1" cpuset="24"/>
    <vcpupin vcpu="2" cpuset="9"/>
    <vcpupin vcpu="3" cpuset="25"/>
    <vcpupin vcpu="4" cpuset="10"/>
    <vcpupin vcpu="5" cpuset="26"/>
    <vcpupin vcpu="6" cpuset="11"/>
    <vcpupin vcpu="7" cpuset="27"/>
    <vcpupin vcpu="8" cpuset="12"/>
    <vcpupin vcpu="9" cpuset="28"/>
    <vcpupin vcpu="10" cpuset="13"/>
    <vcpupin vcpu="11" cpuset="29"/>
    <vcpupin vcpu="12" cpuset="14"/>
    <vcpupin vcpu="13" cpuset="30"/>
    <vcpupin vcpu="14" cpuset="15"/>
    <vcpupin vcpu="15" cpuset="31"/>
    <emulatorpin cpuset="0,4"/>
    <iothreadpin iothread="1" cpuset="0,4"/>
  </cputune>
  <os firmware="efi">
    <type arch="x86_64" machine="pc-q35-9.1">hvm</type>
    <firmware>
      <feature enabled="no" name="enrolled-keys"/>
      <feature enabled="yes" name="secure-boot"/>
    </firmware>
    <loader readonly="yes" secure="yes" type="pflash" format="raw">/usr/share/edk2/x64/OVMF_CODE.secboot.4m.fd</loader>
    <nvram template="/usr/share/edk2/x64/OVMF_VARS.4m.fd" templateFormat="raw" format="raw">/var/lib/libvirt/qemu/win10_VARS.fd</nvram>
    <boot dev="hd"/>
    <bootmenu enable="yes"/>
    <smbios mode="host"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode="custom">
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vpindex state="on"/>
      <runtime state="on"/>
      <synic state="on"/>
      <stimer state="on"/>
      <vendor_id state="on" value="myv3nd0r1d"/>
      <frequencies state="on"/>
      <tlbflush state="on"/>
      <ipi state="on"/>
      <avic state="on"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
    <smm state="on"/>
  </features>
  <cpu mode="host-passthrough" check="none" migratable="on">
    <topology sockets="1" dies="1" clusters="1" cores="8" threads="2"/>
    <feature policy="require" name="topoext"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" present="yes" tickpolicy="discard"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
    <timer name="tsc" present="yes" mode="native"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x16"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0x17"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0x18"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="10" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="10" port="0x19"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
    </controller>
    <controller type="pci" index="11" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="11" port="0x1a"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
    </controller>
    <controller type="pci" index="12" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="12" port="0x1b"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
    </controller>
    <controller type="pci" index="13" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="13" port="0x1c"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
    </controller>
    <controller type="pci" index="14" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="14" port="0x1d"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
    </controller>
    <controller type="pci" index="15" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="15" port="0x8"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
    </controller>
    <controller type="pci" index="16" model="pcie-to-pci-bridge">
      <model name="pcie-pci-bridge"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </controller>
    <interface type="network">
      <mac address="52:54:00:3e:fa:26"/>
      <source network="network"/>
      <model type="virtio"/>
      <driver queues="8"/>
      <address type="pci" domain="0x0000" bus="0x0b" slot="0x00" function="0x0"/>
    </interface>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <tpm model="tpm-crb">
      <backend type="emulator" version="2.0"/>
    </tpm>
    <audio id="1" type="none"/>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x58" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x26ce"/>
        <product id="0x0a06"/>
      </source>
      <address type="usb" bus="0" port="1"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x4a" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x0c" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x03" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x59" slot="0x00" function="0x4"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </hostdev>
    <watchdog model="itco" action="reset"/>
    <memballoon model="none"/>
  </devices>
  <qemu:commandline>
    <qemu:arg value="-overcommit"/>
    <qemu:arg value="cpu-pm=on"/>
    <qemu:arg value="-fw_cfg"/>
    <qemu:arg value="opt/ovmf/X-PciMmio64Mb,string=65536"/>
  </qemu:commandline>
</domain>

Here is my libvirt XML, and yes i have had windows booted without any GPU drivers installed.

Also on VM boot i see the TianoCore UEFI boot screen.

[Piiko]

Hey, first I’d just like to say this is my first adventure though VM’s like this, and I don’t know if anyone could help regarding this but I’m really stuck. I tried setting the passthough for the GPU through livirt first but I was running into issues, so I’m trying to setup proxmox (i read gpu passthrough is simplified and I’d have some other uses for it aswell now that I know what it does) but I feel like im going in circles… I added the gpu and igpu to the vfio, I blacklisted amdgpu, radeon. Tried a dozen other things. I’m I think at the closest to running it right now, but it throws an error “kvm: vfio: Unable to power on device, stuck in D3” . I’ve just updated my BIOS so I’ll see how it is, but if there’s anything I need to paste in order to get some help please advise…

[vvk]

Disable resizeable BAR in BIOS, untick ROM-bar or provide empty ROM.

[Piiko]

Thanks I didn’t see it turned on after the bios update but still no luck. But what do you mean an empty rom? I downloaded the rom for my gpu Sapphire pure 9070xt. I guess I could dump it myself? But not sure if thats the issue… I’ve placed all the configurations files I could think of in some .txts I’ll upload them here if anyone has any clue what could be troubling me. Would be really appreciated. I’ve also tried using vendor-reset but I had some issues setting it up it’s not really working. I also read something about needing to manually build my kernel? I’m not sure if i got that right if its required or not. Not sure where to start… Anyhow thanks in ahead if anyone gives it a go. My last output trying to get the vm started is
qm start 100
error writing ‘1’ to ‘/sys/bus/pci/devices/0000:03:00.0/reset’: Inappropriate ioctl for device
failed to reset PCI device ‘0000:03:00.0’, but trying to continue as not all devices need a reset
kvm: vfio: Unable to power on device, stuck in D3
kvm: vfio: Unable to power on device, stuck in D3

Ah I cant uplaod the txt files here. I guess I’ll pastebin them… or the links to the pastebin. Idk how long of a post this lets write out but I’ll try inputting them here .
qemu-server config:
cat /etc/pve/qemu-server/100.conf
#hostpci0%3A 03%3A00.0,pcie=1,multifunction=on
#hostpci1%3A 03%3A00.1
balloon: 0
bios: ovmf
boot: order=ide2
cores: 6
cpu: x86-64-v2-AES
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,size=4M
ide2: local:iso/cachyos-desktop-linux-250330.iso,media=cdrom,size=2654240K
hostpci0: 03:00.0,pcie=1,romfile=9070xt.rom
hostpci1: 03:00.1
machine: q35
memory: 16000
meta: creation-qemu=9.0.2,ctime=1743608628
name: Catchy
net0: virtio=BC:24:11:4D:6B:7C,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
sata0: local-lvm:vm-100-disk-1,size=200G,ssd=1
sata1: local:iso/virtio-win-0.1.266.iso,media=cdrom,size=707456K
scsihw: virtio-scsi-single
serial0: socket
smbios1: uuid=29a8d829-81c9-4812-87d1-220a2f35413f
sockets: 1
vga: none
vmgenid: 4fc80aa0-f23e-4a63-879a-d497bbca6e69

vfio:
lspci -nnk | grep -A3 -E ‘VGA|Audio’
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7550] (rev c0)
Subsystem: Sapphire Technology Limited Device [1da2:3490]
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab40]
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab40]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel

0f:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raphael [1002:164e] (rev ca)
Subsystem: ASUSTeK Computer Inc. Raphael [1043:8877]
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
0f:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]
Subsystem: ASUSTeK Computer Inc. Rembrandt Radeon High Definition Audio Controller [1043:8877]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
0f:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP [1022:1649]

dmesg | grep -i vfio
[ 3.353623] VFIO - User Level meta-driver version: 0.3
[ 3.358469] vfio-pci 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
[ 3.358581] vfio_pci: add [1002:7550[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.459648] vfio_pci: add [1002:ab40[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.474458] vfio-pci 0000:0f:00.0: vgaarb: deactivate vga console
[ 3.474463] vfio-pci 0000:0f:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
[ 3.474574] vfio_pci: add [1002:164e[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.498833] vfio_pci: add [1002:1640[ffffffff:ffffffff]] class 0x000000/00000000

cat /etc/initramfs-tools/modules
vfio
vfio_pci
vfio_iommu_type1
vfio_virqfd

grub:
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=lsb_release -i -s 2> /dev/null || echo Debian
GRUB_CMDLINE_LINUX_DEFAULT=“quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off”
GRUB_CMDLINE_LINUX=“”

#cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.12-9-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off

cat /etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist amdgpu
blacklist snd_hda_intel

some other things:
dmesg | grep -i acs
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.12-9-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.004060] ACPI: FACS 0x0000000073C8F000 000040
[ 0.004096] ACPI: Reserving FACS table memory at [mem 0x73c8f000-0x73c8f03f]
[ 0.025929] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.8.12-9-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off
root@prox:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.12-9-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off

lsinitramfs /boot/initrd.img-$(uname -r) | grep vfio
etc/modprobe.d/vfio.conf
cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=1002:7550,1002:ab40,1002:164e,1002:1640 disable_vga=1

BIOS
CSM disabled
resizable bar disabled
Above 4G decoding enabled
D3 support disabled
Primary gpu set to igpu
IOMMU enabled

PROXMOX and kernel version
pve-manager/8.3.5/dac3aa88bac3f300 (running kernel: 6.8.12-9-pve)
6.8.12-9-pve

dkms status
modprobe vendor-reset
ls /sys/bus/pci/drivers/vendor-reset
vendor-reset/0.1.1, 6.8.12-9-pve, x86_64: installed (WARNING! Diff between built and installed module!)
ls: cannot access ‘/sys/bus/pci/drivers/vendor-reset’: No such file or directory
root@prox:~# /usr/bin/vendor-reset 0000:03:00.0
ioctl: Operation not supported

and the iommu groups i guess:
find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/17/devices/0000:05:00.0
/sys/kernel/iommu_groups/35/devices/0000:0f:00.2
/sys/kernel/iommu_groups/7/devices/0000:00:08.0
/sys/kernel/iommu_groups/25/devices/0000:07:00.0
/sys/kernel/iommu_groups/15/devices/0000:03:00.1
/sys/kernel/iommu_groups/33/devices/0000:0f:00.0
/sys/kernel/iommu_groups/5/devices/0000:00:03.0
/sys/kernel/iommu_groups/23/devices/0000:05:0c.0
/sys/kernel/iommu_groups/13/devices/0000:02:00.0
/sys/kernel/iommu_groups/31/devices/0000:0d:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:02.1
/sys/kernel/iommu_groups/21/devices/0000:05:07.0
/sys/kernel/iommu_groups/11/devices/0000:00:18.3
/sys/kernel/iommu_groups/11/devices/0000:00:18.1
/sys/kernel/iommu_groups/11/devices/0000:00:18.6
/sys/kernel/iommu_groups/11/devices/0000:00:18.4
/sys/kernel/iommu_groups/11/devices/0000:00:18.2
/sys/kernel/iommu_groups/11/devices/0000:00:18.0
/sys/kernel/iommu_groups/11/devices/0000:00:18.7
/sys/kernel/iommu_groups/11/devices/0000:00:18.5
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/38/devices/0000:10:00.0
/sys/kernel/iommu_groups/28/devices/0000:0a:00.0
/sys/kernel/iommu_groups/18/devices/0000:05:04.0
/sys/kernel/iommu_groups/36/devices/0000:0f:00.3
/sys/kernel/iommu_groups/8/devices/0000:00:08.1
/sys/kernel/iommu_groups/26/devices/0000:08:00.0
/sys/kernel/iommu_groups/16/devices/0000:04:00.0
/sys/kernel/iommu_groups/34/devices/0000:0f:00.1
/sys/kernel/iommu_groups/6/devices/0000:00:04.0
/sys/kernel/iommu_groups/24/devices/0000:05:0d.0
/sys/kernel/iommu_groups/14/devices/0000:03:00.0
/sys/kernel/iommu_groups/32/devices/0000:0e:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:02.2
/sys/kernel/iommu_groups/22/devices/0000:05:08.0
/sys/kernel/iommu_groups/12/devices/0000:01:00.0
/sys/kernel/iommu_groups/30/devices/0000:0c:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/20/devices/0000:05:06.0
/sys/kernel/iommu_groups/10/devices/0000:00:14.3
/sys/kernel/iommu_groups/10/devices/0000:00:14.0
/sys/kernel/iommu_groups/29/devices/0000:0b:00.0
/sys/kernel/iommu_groups/0/devices/0000:00:01.0
/sys/kernel/iommu_groups/19/devices/0000:05:05.0
/sys/kernel/iommu_groups/37/devices/0000:0f:00.4
/sys/kernel/iommu_groups/9/devices/0000:00:08.3
/sys/kernel/iommu_groups/27/devices/0000:09:00.0

[vvk]

See The state of AMD RX 7000 Series VFIO Passthrough (April 2024) for explanation of the issue and potential solution. I don’t currently have 9000-series card at hand so I cannot test it.

[GlemSom]

I had some luck with 6.14.0. Here I could do the same with letting amdgpu driver initialize the card, and then unbind, and bind vfio-pci just before launching the VM.
This even works with showing the OVMF bios screen.

But, with 6.14.2 this no-longer works… The OVMF screen is not shown anymore. Anyone else able to replicate this regression ?

[Janos]

I noticed last week with Kubuntu 25.04 that Rebar now just works.
Activate it in the BIOS and it works in the VM. no extra script required

[BuzzBumbleBee]

I have seen the same on 6.14.2, not sure whats causing this.

Rolling back to 6.14.0 works again

EDIT:

Its probably related to this 220010 – Regression: 6.14.2 breaks PCI passthrough, qemu-system-x86_64: no available reset mechanism

[Janos]

I have that “issue” with CachyOS and Kubuntu 6.14.0-15, but reset works just fine

[GlemSom]

Glad I am at-least not the only one
On a positive note for 6.14.2; I tried not using amdgpudriver, and it seems VFIO is actually able to wake-up the card from D3 now…
So, things are for sure getting better. (Restart of the VM still does not work though - but, first-time launch does without the use of amdgpu)

[Precific]

As an alternative to the amdgpu workaround, you can live-attach the GPU once the guest OS has booted. If it is done too early, it won’t work - I guess because the UEFI still handles devices up to some point in OS startup. VM reboots appear to work; shutting down one VM and then starting another with the same method also works.

Make sure the VM has two free PCIe root ports (pcie-root-port devices) defined prior to startup, one for the GPU and one for the audio device. I haven’t checked what it does otherwise, though.

It is quite easy to do with libvirt. In the files below, change the ‘YourVMName’ and device xml portions to match your setup. It chooses the root port automatically.
If you start QEMU directly, you could use the QEMU monitor instead.

liveattach-gpu-vm.sh

#!/usr/bin/env bash
# The domain option matches the VM name shown in VirtManager.
# Place the device configuration xmls next to this script and run from the script's directory.
sudo virsh attach-device --domain YourVMName --file "$(pwd)/gpu-maindev.xml" --live
#Sleep may not be needed
sleep 2
sudo virsh attach-device --domain YourVMName --file "$(pwd)/gpu-audiodev.xml" --live

gpu-maindev.xml

<hostdev mode="subsystem" type="pci" managed="yes">
  <source>
    <address domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
  </source>
</hostdev>

gpu-audiodev.xml

<hostdev mode="subsystem" type="pci" managed="yes">
  <source>
    <address domain="0x0000" bus="0x03" slot="0x00" function="0x1"/>
  </source>
</hostdev>

[jxdking]

Can I say 9070xt works with VFIO?

[GlemSom]

@BuzzBumbleBee Are you able to do a complete Windows 11 install, without using VNC/Spice ? (Like JUST using the 9070XT) ?

[BuzzBumbleBee]

Yeah once, I haven’t tried since.

From what I remember it was fine (the 1st thing installed tho was the GFX drivers)

Note that would have been after binding to amdgpu