9070XT passthrough on latest kernels improvements

Operating Systems & Open Source Virtualization
1

Hi,

First post here.. Owner of a 9070XT (using APU as boot default) and trying to virtualize away my only windows baremetal machine.

Mostly don’t game much anymore but still tied to MS because I need GSPro golf sim and some custom audio calibration tools on windows..

Got my win10 image setup on my fedora 44 workstation and passthrough of 9070XT working fine after experimentation (wasted my time on Gemini’s bs guesses instead of trusting my experience!). Still getting AMD reset bug.

Tried pretty much every technique under the sun to workaround it (I don’t like to quit).

The problem is you can poke at setpci poke anything and echo “bind” to /sys yourself crazy but nothing will reset the card after windows pooped on it.

Installed fedora testing kernel 7.2–rc3 via COPR and noticed that amdgpu is MUCH better at dealing with errors and failing gracefully now.. (ip discovery failure post windows guest shutdown is more descriptive too)

I decided to compile the bleeding edge amdgpu/kernel yesterday (with latest linux-firmware) and even with multiple combination of AMDGPU kernel options still no pseudo FLR reset workaround. (if it’s even possible)

So until devs find a way to artificially FLR this thing this is good enough for me:

The way I settled on (for now), was to leave amdgpu binded on bootup (to init GPU properly) and let libvirt auto manage it. and then I used this hook script to do a short system suspend to ram by RTC (no need to physically be there) and then the GPU comes fully resetted and I can start vm again. rebooting works fine too now.

#!/bin/bash
# /etc/libvirt/hooks/qemu

GUEST_NAME="win10"
GPU_GFX="0000:03:00.0"
GPU_AUD="0000:03:00.1"

# Phase 1: Lockout
if [ "$1" = "$GUEST_NAME" ] && [ "$2" = "started" ] && [ "$3" = "begin" ]; then
    echo "dummy" > /sys/bus/pci/devices/${GPU_GFX}/driver_override
    echo "dummy" > /sys/bus/pci/devices/${GPU_AUD}/driver_override
    exit 0
fi

# Phase 2: Hardware S3 Reset Transition
if [ "$1" = "$GUEST_NAME" ] && [ "$2" = "stopped" ] && [ "$3" = "end" ]; then
    # 1. Force the sleep cycle to wipe the RDNA4 firmware state
    rtcwake -m mem -s 5
    sleep 2.0

    # 2. Release the overrides so the kernel's auto-resume can natively map them
    echo "" > /sys/bus/pci/devices/${GPU_GFX}/driver_override
    echo "" > /sys/bus/pci/devices/${GPU_AUD}/driver_override

    # 3. Explicitly tell libvirt that the script finished successfully
    exit 0
fi

Not claiming it’s a novel method in any way of course but just documenting for others as most examples were for single GPU and proxmox..

here’s my current kernel cmdline for reference:

BOOT_IMAGE=(hd0,gpt2)/vmlinuz-7.1.0+ root=UUID=a5cc57bf-f256-475e-a7df-8b973d901c2c ro rootflags=subvol=root amd_iommu=on iommu=pt pcie_aspm=off pcie_port_pm=off  amdgpu.gpu_recovery=1

ps: i’ll eventually clean that up though as most of those didn’t make a difference while testing. (you do need amd_iommu=on iommu=pt)

win10.xml snippet:

  <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev1'/>
      <rom file='/var/lib/libvirt/roms/Sapphire.RX9070XT.16384.241204.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>

Might not need to pass ROM file but haven’t had a chance to clean up. might add an update later.

Cheers,

3 Likes
2

In my case, standby is the only way to pass my 9070XT to (not from) a Windows guest if amdgpu was ever loaded at any time, so I need to unbind it from amdgpu prior to standby. Then, I let the VM boot without any GPU and then live-attach it via virsh (VM booting also breaks it).

It’s funny how specifically loading amdgpu before starting the guest works for some/most people, while it’s basically the opposite for my system. It’s a miracle it works at all.

1 Like