VFIO Pass through working on 9070XT

Operating Systems & Open Source Virtualization
64

It turns out that QXL is essential. I must have checked incorrectly whether it works after removing it. Without QXL, there is no image from the GPU, just a black screen. After adding it back, the graphics card outputs an image (the monitor name is now displayed instead of vDisplay). I’m not entirely sure what’s going on here, but I’m satisfied with the result.

65

Thanks @perrine for the script and suggestions.

I finally got around to testing again with a new VM and with QXL… Sadly, I cannot (simply) use BIOS instead of UEFI, because my existing Windows installation was done on a bare metal M.2 drive from a UEFI system. While I could attempt to convert the disk partition style, I’m not willing to risk potential data loss.

The only thing left to try is a fresh Windows install relying on virtualized storage, but that would only be for more testing.

It seems I’ll again hold off on GPU passthrough until getting another M.2 drive for a fresh Windows install.

66

I am having the exact same issue now. It has been working fine for months - but today the screen is just garbled.
Anyone ever been able to fix the issue with the garbled screen?

67

Hello, I would like to ask on how are the iommu grouped when 2 gpus are installed (x570-f) and if it would be possible to passthrough each to a different VM?

68

I can confirm the X570-F Strix has good IOMMU group separation and both my GPUs are in separate groups. Pass-through would be difficult otherwise.

If you’d like more info I’m happy to help, but it might pay to make a new topic. :slightly_smiling_face:

69

I’ve made a topic and I would be glad for your help and insight on this.

70

How is the current progress with passthrough of the 9700xt? - I want to use it in a Linux guest with Proxmox. If it is to much of a hassle I’ll stick to nvidia.

In my internet research the results were mixes with various newer Kernel version breaking vfio.

I only found this thread which also includes a hookup script but it is from May 25

71

Made an account just to comment on this.

Has anyone figured out how to fix this? Ditto on the exact same pattern appearing.

Here is what I found in my testing.

I got it down to drivers. If I uninstalled drivers using DDU, I was able to boot fine into windows. But the second drivers were installed, garbled screen.

VM Works perfectly fine, it boots, it passes through correctly etc. It seems to be in windows only to be doing this.

I went through 6950xt to 9070xt on passthrough.

It would suck if I can’t figure this out as I need a windows passthrough VM for my workflow.

cheers guys

72

I did this with two 7900XTX, in separate VMs and also both in the same VM.
However, the IOMMU grouping depends on your motherboard.

I did it with an X670e Creator.

73

I have the same problem with a 5090.
No driver errors are displayed in Windows, and no error message appears in the Linux host system.

The monitor receives a signal but only displays a black screen.
I didn’t have the problem for the first few weeks and I suspect it started after a BIOS update, but I’m not sure.

However, everything works perfectly with a VBIOS copy in Libvirt.
Strangely, I haven’t seen anyone else whose 5090 needs a VBIOS copy.

74

over 12 hours at it already getting this GPU to work with vfio, and no luck so far. I can get it to boot once but I cannot fix the reset issue to save my life. The 6900XT in comparison was so smooth.

This shutting down bug is what I am currently facing, unfortunately those features did not help. Anyone has any idea what else to try?

Edit: After an additional day of testing, what helped immensely were those three lines I added:

echo 1 > /sys/bus/pci/devices/0000:06:00.0/remove  
echo 1 > /sys/bus/pci/rescan  
echo "0000:06:00.0" > /sys/bus/pci/drivers/amdgpu/unbind

I still do the other bind/unbind dance above it, but those 3 lines allowed the VM to shutdown properly when executed before booting the vm. Next up is narrowing down how much how the stuff I tested I can remove and still have a working setup.

Although I would really love to understand why remove/rescan makes the VM shutdown cleanly

75 
<sysinfo type='smbios'>
  <system>
    <entry name='manufacturer'>ASUSTeK COMPUTER INC.</entry>
    <entry name='product'>ROG CROSSHAIR X870E HERO</entry>
  </system>
  <baseBoard>
    <entry name='manufacturer'>ASUSTeK COMPUTER INC.</entry>
    <entry name='product'>ROG CROSSHAIR X870E HERO</entry>
  </baseBoard>
</sysinfo>

<os>
  <smbios mode='sysinfo'/>
</os>
76

The solution for the stupid green/red garbled screen using libvirt for me was to add in the “hyperv” section:

<vendor_id state=“on” value=“0123456789ab”/>

On proxmox you simply enable the “primary gpu” checkbox.

77

I’ve had this line in my XML since October ‘25… No such luck fixing the garbled screen.

TL;DR - The lesson I learned:
Know the intended purpose for your hardware configuration and check for subcomponent-level compatibility before pursuing a build.

Without reliable passthrough, my rationale for a SFF AMD workstation build makes considerably less sense for my Linux first workflow. This is especially troubling since I’m using the ROG Strix X870-I MB, which still does not have device support for the MediaTek MT7927 WiFi and Bluetooth card in the kernel.

All this to say, I’m strongly considering reselling my SFF workstation hardware to instead pursue a dedicated proxmox server (with passthrough friendly hardware) to host my Windows VMs. I’m new to the space, so any recommended forums or sites to list parts is greatly appreciated.

78

I also came to the conclusion, that the 9070xt is not well suited for vfio passthrough (for my usecases). In my case I would get garbage collection issues with high demanding games on Linux guests (the game process would be stuck at low fps and never recover, on bare metal it hitches and loads in new data) among various other issues (host restarts after switching VMs certain times, despite running bind and unbind hookscripts). Their awful business practises aside, nvidia cards are mostly trouble free on my Proxmox hosts. I switched to Desktop Linux (bare metal) and I am pleased with the 9070xt there. (I do virtualisation with KVM/qemu and virt-manager + a isolated RTX 4070 on this machine).

2 Likes
79

Hi All,

I just wanted to share some updates, that might help anyone struggling with their 9070XT.

My initial process was:

  1. Switch to a terminal and running the following script
#!/bin/bash

systemctl stop lightdm.service
sleep 2

echo 0000:10:00.0 > /sys/bus/pci/drivers/amdgpu/unbind
sleep 2


 0000:10:00.1 > /sys/bus/pci/drivers/snd_hda_intel/unbind
#sleep 2

echo 3 > /sys/bus/pci/devices/0000:10:00.0/resource2_resize
sleep 2

systemctl start lightdm.service
  1. Once LightDM restarted I could run whatever VM i wanted
  2. After shutting down the VM the kernel would automatically grab the GPU and attach it back to the AMDGPU driver.
  3. rinse and repeat ad nauseam.

I haven’t actually used my VMs for several months, fast forward to a couple of days ago, and when i tried it, it no longer works the same. I can use the script, start the VM, and the VM starts with the GPU attached, but upon exiting, the driver would no longer grab the GPU back, meaning the only way i could run another VM or reset my GPU was by doing a hard reset. According to Google AI this is because between 6.12 and 7.2 there have been changes to the hardware scheduler and power management.

However, with google’s help I was able to come up with the following script, which has been added to the libvirt/hooks directory. And now it automatically does everything. I can start and stop VMs without a hard reset. I have used it a bunch of times and so far, it has worked every time.

#!/bin/bash

# CONFIGURATION: The target PCIe address of your secondary AMD card
GPU_PCI="0000:10:00.0"

ACTION="$2"
PHASE="$3"

# 1. RUN BEFORE ANY VM BOOTS (Prepares the card without freezing)
if [ "$ACTION" = "prepare" ] && [ "$PHASE" = "begin" ]; then
    # Stop LightDM cleanly from the system background
    systemctl stop lightdm.service
    sleep 2

    # Unbind from host driver
    if [ -e "/sys/bus/pci/devices/$GPU_PCI/driver" ]; then
        echo "$GPU_PCI" > /sys/bus/pci/drivers/amdgpu/unbind 2>/dev/null
        sleep 2
    fi

    # Set BAR size down for VM compatibility
    echo 3 > /sys/bus/pci/devices/$GPU_PCI/resource2_resize 2>/dev/null
    sleep 1
    
    # Restart LightDM right away so your host desktop comes back online while VM is running
    systemctl start lightdm.service
fi

# 2. RUN AFTER ANY VM SHUTS DOWN (Reclaims the card safely for successive boots)
if [ "$ACTION" = "release" ] && [ "$PHASE" = "end" ]; then
    # Stop LightDM in the background so X11 stops polling the hardware lanes
    systemctl stop lightdm.service
    sleep 2

    # Clear driver overrides and force bind safely
    echo "amdgpu" > /sys/bus/pci/devices/$GPU_PCI/driver_override 2>/dev/null
    echo "$GPU_PCI" > /sys/bus/pci/drivers/amdgpu/bind 2>/dev/null
    sleep 2

    # Restart LightDM to securely restore your desktop environment
    systemctl start lightdm.service
fi

So, I wanted to share it in case it can help anyone else.

80

Hey Xavier I just spotted this as I’m struggling with the same issue. I managed to get the bind working, but unbind is crashing some times. I’ll give your scripts a try, with some slight modifications.

Sadly, doesn’t work for me. What works to bind is the following:

To bind:

#!/usr/bin/env bash
set -uo pipefail

VGA=$(lspci -D -d 1002:7550 | awk ‘{print $1}’)
HDA=$(lspci -D -d 1002:ab40 | awk ‘{print $1}’)

systemctl stop lactd.service
systemctl stop coolercontrold.service
systemctl stop sddm.service

sleep 3

1. Stop the display manager (adapt to your DM: sddm, gdm, plasmalogin, etc.)

#if systemctl is-active --quiet plasmalogin.service 2>/dev/null; then

systemctl stop plasmalogin.service || true

sleep 1

#fi

2. Drop VT consoles and framebuffer

#for vt in /sys/class/vtconsole/vtcon*/bind; do

[ -w “$vt” ] && echo 0 > “$vt” 2>/dev/null

#done
#for d in /sys/bus/platform/drivers/efi-framebuffer/efi-framebuffer.0 \

/sys/bus/platform/drivers/simple-framebuffer/simple-framebuffer.0; do

[ -e “$d” ] || continue

basename “
(dirname “$d”)/unbind” 2>/dev/null || true

#done

3. Lock GPU in D0 BEFORE unbinding amdgpu (THE critical step)

echo 0 > /sys/bus/pci/devices/$VGA/d3cold_allowed
echo “on” > /sys/bus/pci/devices/$VGA/power/control

4. Unbind amdgpu and snd_hda_intel by PCI address

Uses the device’s own driver symlink, not the driver’s global unbind —

avoids races if multiple devices share the same driver.

echo “$VGA” > /sys/bus/pci/devices/$VGA/driver/unbind 2>/dev/null || true
echo “$HDA” > /sys/bus/pci/devices/$HDA/driver/unbind 2>/dev/null || true
sleep 1

5. Resize BAR2 to 8 MB (Navi 48 requirement)

echo 3 > /sys/bus/pci/devices/$VGA/resource2_resize

6. Bind to vfio-pci manually (managed=‘no’ in XML)

echo vfio-pci > /sys/bus/pci/devices/$VGA/driver_override
echo “$VGA” > /sys/bus/pci/drivers/vfio-pci/bind
echo vfio-pci > /sys/bus/pci/devices/$HDA/driver_override
echo “$HDA” > /sys/bus/pci/drivers/vfio-pci/bind

systemctl restart coolercontrold.service
systemctl restart sddm.service

And to unbind (some times works, some times gets stuck until reboot):

#!/usr/bin/env bash

VGA=$(lspci -D -d 1002:7550 | awk ‘{print $1}’)
HDA=$(lspci -D -d 1002:ab40 | awk ‘{print $1}’)
UPSTREAM=
(readlink -f /sys/bus/pci/devices/$VGA/../)")

gpu_bound() { [ -e /dev/dri/card0 ] || [ -e /dev/dri/card1 ]; }

for attempt in 1 2 3; do
# Unbind from vfio-pci
echo “$VGA” > /sys/bus/pci/drivers/vfio-pci/unbind 2>/dev/null || true
echo “$HDA” > /sys/bus/pci/drivers/vfio-pci/unbind 2>/dev/null || true

# Clear driver_override (echo "", not ": >" — truncation doesn't trigger the write handler)
echo "" > /sys/bus/pci/devices/$VGA/driver_override 2>/dev/null || true
echo "" > /sys/bus/pci/devices/$HDA/driver_override 2>/dev/null || true

# On retry, do a secondary bus reset via the upstream bridge
if [ "$attempt" -gt 1 ] && [ -n "$UPSTREAM" ]; then
    cur=$(setpci -s "$UPSTREAM" BRIDGE_CONTROL.w)
    setpci -s "$UPSTREAM" BRIDGE_CONTROL.w=$(printf '%04x' $((0x$cur | 0x40)))
    sleep 1
    setpci -s "$UPSTREAM" BRIDGE_CONTROL.w="$cur"
    sleep 1
fi

# Force D0
setpci -s "${VGA#0000:}" CAP_PM+4.w=0x0000 2>/dev/null || true
sleep 1

# Bind amdgpu BY PCI ADDRESS (modprobe alone does nothing)
modprobe amdgpu 2>/dev/null || true
echo "$VGA" > /sys/bus/pci/drivers/amdgpu/bind 2>/dev/null || true
sleep 3

if gpu_bound; then
    break
fi

done

Rebind audio

echo “$HDA” > /sys/bus/pci/drivers/snd_hda_intel/bind 2>/dev/null || true

systemctl restart sddm.service
systemctl restart coolercontrold.service
systemctl restart lactd.service

This card is a real pain in the butt. I wish I could just bind and unbind it to the VM when needed without having to restart the display manager each time.

81

I don’t know what happened, but after latest updates there’s no longer any need to have any bind scripts for passthrough. Just add the GPU in the Virt-Manager config and have it managed by libvirt.
After one display manager restart I’m able to bind and unbind the GPU succesfully. No reset bug. Tried with multiple VM starts and shutdowns.

Just need the following for plasma and xorg conf:

cat ~/.config/plasma-workspace/env/kwin-drm.sh
#!/bin/sh
export KWIN_DRM_DEVICES='/dev/dri/by-path/pci-0000\:0f\:00.0-card'

cat /etc/X11/xorg.conf.d/10-igpu-only.conf
Section "Device"
Identifier "iGPU"
Driver "amdgpu"
BusID "PCI:15:0:0"
EndSection

Section "Device"
Identifier "dGPU"
Driver "amdgpu"
BusID "PCI:3:0:0"
EndSection

Section "Screen"
Identifier "iGPU-Screen"
Device "iGPU"
EndSection

Section "ServerLayout"
Identifier "Layout"
Screen "iGPU-Screen"
EndSection

Section "ServerFlags"
Option "AutoAddGPU" "off"
EndSection