6800XT reference single gpu passthrough no signal

Dear experts,

I’m trying to get win10 single gpu passthrough working with my AMD 6800XT reference card, but for now I’m only getting a black screen (no signal) when I startup my vm. It looks like windows is starting up, because when I try to log into windows, my keyboard lights flash for the numpad (but not sure if that indicates that windows is working).

I have followed the following guide:

My system information:

System:
  Kernel: 5.15.12-1-MANJARO x86_64 bits: 64 compiler: gcc v: 11.1.0
    parameters: BOOT_IMAGE=/boot/vmlinuz-5.15-x86_64
    root=UUID=XXX rw amd_iommu=on iommu=pt
    video=efifb:off quiet apparmor=1 security=apparmor
    resume=UUID=XXX udev.log_priority=3
  Desktop: KDE Plasma 5.23.4 tk: Qt 5.15.2 wm: kwin_x11 vt: 1 dm: SDDM
    Distro: Manjaro Linux base: Arch Linux
Machine:
  Type: Desktop Mobo: ASUSTeK model: ROG STRIX X570-E GAMING v: Rev X.0x
    serial: <superuser required> UEFI: American Megatrends v: 4021
    date: 08/09/2021
Memory:
  RAM: total: 62.71 GiB used: 4.41 GiB (7.0%)
  RAM Report:
    permissions: Unable to run dmidecode. Root privileges required.
CPU:
  Info: model: AMD Ryzen 9 5900X bits: 64 type: MT MCP arch: Zen 3
    family: 0x19 (25) model-id: 0x21 (33) stepping: 0 microcode: 0xA201016
Graphics:
  Device-1: AMD Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] driver: amdgpu
    v: kernel bus-ID: 0c:00.0 chip-ID: 1002:73bf class-ID: 0300

Bios options:

  • Advanced > CPU Configuration > NX Mode > enabled
  • Advanced > CPU Configuration > SVM Mode > enabled
  • Advanced > AMD CBS > IOMMU > enabled
  • Advanced > PCI Subsystem settings > Above 4g decoding > disabled
  • Advanced > PCI Subsystem settings > SR-IOV > disabled
  • Boot > CSM (Compatibility Support Module) > Launch CSM > disabled (tried to enabled it, but that didn’t work)
IOMMU groups
IOMMU Group 27:
        0c:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c1)
IOMMU Group 28:
        0c:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] [1002:ab28]
IOMMU Group 29:
        0c:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a6]
IOMMU Group 30:
        0c:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 USB [1002:73a4]

As you can see, my GPU as 4 different groups. Accoording to the Risingprism guide, that shouldn’t matter, quote:

AMD users can have there GPU and AUDIO device in different IOMMU group but that is still fine.

I have changed the hooks folder structure from the Risingprism guide, because I was having the same outcome (no signal), so I tried a different approach (other examples that I saw on the web):

/etc/libvirt/hooks
├── kvm.conf
├── qemu
├── qemu.d
│   └── win10
│       ├── prepare
│       │   └── begin
│       │       └── start.sh
│       └── release
│           └── end
│               └── revert.sh
└── qemu_last_backup

start.sh
#!/bin/bash
set -x

echo "Sart script: $(date)"

# Stop your display manager
systemctl stop sddm.service

echo "Stop sddm.service done!"

# Kill audio
pulse_pid=$(pgrep -u brutalis pulseaudio)
pipewire_pid=$(pgrep -u brutalis pipewire-media)
kill $pulse_pid
kill $pipewire_pid

echo "Audio killed with id: $pulse_pid $pipewire_pid"

# Unbind VTconsoles
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind

echo "Unbind vtconsoles done!"

# Avoid a race condition by wating a couple of seconds
sleep 10

# Unload all drivers drivers
modprobe -r amdgpu
modprobe -r snd_hda_intel
#modprobe -r xhci_hcd
#modprobe -r i2c-designware-pci

echo "Unload gpu drivers done!"

# Unbind the GPU from display driver
virsh nodedev-detach pci_0000_0c_00_0
virsh nodedev-detach pci_0000_0c_00_1
virsh nodedev-detach pci_0000_0c_00_2
virsh nodedev-detach pci_0000_0c_00_3

echo "Unload gpu from display driver done!"

# Load VFIO kernel module
modprobe vfio
modprobe vfio_pci
modprobe vfio_iommu_type1

echo "Load VFIO kernel done!"

echo "End of startup: $(date)"

revert.sh
#!/bin/bash
set -x

echo "Beginning of teardown: $(date)"

# Unload VFIO-PCI Kernel Driver
modprobe -r vfio_pci
modprobe -r vfio_iommu_type1
modprobe -r vfio

echo "Unload VFIO-PCI kernel driver done!"

# Re-bind GPU to our display drivers
virsh nodedev-reattach pci_0000_0c_00_0
virsh nodedev-reattach pci_0000_0c_00_1
virsh nodedev-reattach pci_0000_0c_00_2
virsh nodedev-reattach pci_0000_0c_00_3

echo "Re-bind gpu to display drivers done!"

# Re-bind VTconsoles
echo 1 > /sys/class/vtconsole/vtcon0/bind
echo 1 > /sys/class/vtconsole/vtcon1/bind

echo "Re-bind VTconsole done!"

# Load gpu drivers
modprobe  amdgpu
modprobe  gpu_sched
modprobe  ttm
modprobe  drm_kms_helper
modprobe  i2c_algo_bit
modprobe  drm
modprobe  snd_hda_intel

echo "Load gpu drivers done"

# Restart display manager
systemctl start sddm.service

echo "Restart display manager done!"

echo "End of teardown: $(date)"
win10.xml
<domain type="kvm">
  <name>win10</name>
  <uuid>bd9fd7c1-ef70-4e42-a5f7-136ae7432963</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">16777216</memory>
  <currentMemory unit="KiB">16777216</currentMemory>
  <vcpu placement="static">16</vcpu>
  <os>
    <type arch="x86_64" machine="pc-q35-6.1">hvm</type>
    <loader readonly="yes" type="rom">/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <boot dev="hd"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
    </hyperv>
    <vmport state="off"/>
  </features>
  <cpu mode="host-passthrough" check="none" migratable="on">
    <topology sockets="1" dies="1" cores="8" threads="2"/>
    <feature policy="require" name="topoext"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" cache="writeback"/>
      <source file="/var/lib/libvirt/images/win10.qcow2"/>
      <target dev="vda" bus="virtio"/>
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/brutalis/Downloads/iso/Win10_21H2_English_x64.iso"/>
      <target dev="sdb" bus="sata"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="1"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/brutalis/Downloads/iso/virtio-win-0.1.208.iso"/>
      <target dev="sdc" bus="sata"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="2"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x16"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0x8"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0x9"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </controller>
    <interface type="network">
      <mac address="52:54:00:af:14:b0"/>
      <source network="default"/>
      <model type="e1000e"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </interface>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <sound model="ich9">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
    </sound>
    <audio id="1" type="none"/>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x0c" slot="0x00" function="0x0"/>
      </source>
      <rom file="/var/lib/libvirt/vbios/amd03012022.rom"/>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0" multifunction="on"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x0c" slot="0x00" function="0x1"/>
      </source>
      <rom file="/var/lib/libvirt/vbios/amd03012022.rom"/>
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x046d"/>
        <product id="0xc07d"/>
      </source>
      <address type="usb" bus="0" port="4"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x24f0"/>
        <product id="0x0140"/>
      </source>
      <address type="usb" bus="0" port="5"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x1b1c"/>
        <product id="0x0a3e"/>
      </source>
      <address type="usb" bus="0" port="6"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x0c" slot="0x00" function="0x2"/>
      </source>
      <rom file="/var/lib/libvirt/vbios/amd03012022.rom"/>
      <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x0c" slot="0x00" function="0x3"/>
      </source>
      <rom file="/var/lib/libvirt/vbios/amd03012022.rom"/>
      <address type="pci" domain="0x0000" bus="0x09" slot="0x00" function="0x0"/>
    </hostdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="2"/>
    </redirdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="3"/>
    </redirdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
</domain>

As you can see in the win10.xml I also pass the GPU bios. I have tried a downloaded version from techpowerup.com and I have extracted my own GPU bios with amdvbflash, but with the same outcome.

When I ssh into my machine I don’t notice any errors from the hooks

custom_hooks.log

Sart script: Wed Jan 5 01:11:56 CET 2022
Stop sddm.service done!
Audio killed with id:
Unbind vtconsoles done!
Unload gpu drivers done!
Device pci_0000_0c_00_0 detached

Device pci_0000_0c_00_1 detached

Device pci_0000_0c_00_2 detached

Device pci_0000_0c_00_3 detached

Unload gpu from display driver done!
Load VFIO kernel done!
End of startup: Wed Jan 5 01:12:06 CET 2022
Beginning of teardown: Wed Jan 5 01:15:53 CET 2022
Unload VFIO-PCI kernel driver done!
Device pci_0000_0c_00_0 re-attached

Device pci_0000_0c_00_1 re-attached

Device pci_0000_0c_00_2 re-attached

Device pci_0000_0c_00_3 re-attached

Re-bind gpu to display drivers done!
Re-bind VTconsole done!
Load gpu drivers done
Restart display manager done!
End of teardown: Wed Jan 5 01:15:56 CET 2022

But I saw something in the win10.log:

audio: Could not init `spice' audio driver
audio: warning: Using timer based audio emulation

I have fixed it with the following statement: <audio id="1" type="none"/>

win10.log
2022-01-05 00:12:06.980+0000: starting up libvirt version: 7.10.0, qemu version: 6.2.0, kernel: 5.15.12-1-MANJARO, hostname: user-pc
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/var/lib/snapd/snap/bin \
HOME=/var/lib/libvirt/qemu/domain-1-win10 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-win10/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-win10/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-win10/.config \
/usr/bin/qemu-system-x86_64 \
-name guest=win10,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-win10/master-key.aes"}' \
-machine pc-q35-6.1,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram \
-accel kvm \
-cpu host,migratable=on,topoext=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff \
-bios /usr/share/edk2-ovmf/x64/OVMF_CODE.fd \
-m 16384 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":17179869184}' \
-overcommit mem-lock=off \
-smp 16,sockets=1,dies=1,cores=8,threads=2 \
-uuid bd9fd7c1-ef70-4e42-a5f7-136ae7432963 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=32,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x5"}' \
-device '{"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"}' \
-device '{"driver":"pcie-root-port","port":8,"chassis":8,"id":"pci.8","bus":"pcie.0","multifunction":true,"addr":"0x1"}' \
-device '{"driver":"pcie-root-port","port":9,"chassis":9,"id":"pci.9","bus":"pcie.0","addr":"0x1.0x1"}' \
-device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.2","addr":"0x0"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.3","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/win10.qcow2","node-name":"libvirt-3-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.4","addr":"0x0","drive":"libvirt-3-format","id":"virtio-disk0","bootindex":1,"write-cache":"on"}' \
-blockdev '{"driver":"file","filename":"/home/brutalis/Downloads/iso/Win10_21H2_English_x64.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.1","drive":"libvirt-2-format","id":"sata0-0-1"}' \
-blockdev '{"driver":"file","filename":"/home/brutalis/Downloads/iso/virtio-win-0.1.208.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.2","drive":"libvirt-1-format","id":"sata0-0-2"}' \
-netdev tap,fd=34,id=hostnet0 \
-device '{"driver":"e1000e","netdev":"hostnet0","id":"net0","mac":"52:54:00:af:14:b0","bus":"pci.1","addr":"0x0"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-device '{"driver":"ich9-intel-hda","id":"sound0","bus":"pcie.0","addr":"0x1b"}' \
-device '{"driver":"hda-duplex","id":"sound0-codec0","bus":"sound0.0","cad":0,"audiodev":"audio1"}' \
-chardev spicevmc,id=charredir0,name=usbredir \
-device '{"driver":"usb-redir","chardev":"charredir0","id":"redir0","bus":"usb.0","port":"2"}' \
-chardev spicevmc,id=charredir1,name=usbredir \
-device '{"driver":"usb-redir","chardev":"charredir1","id":"redir1","bus":"usb.0","port":"3"}' \
-device '{"driver":"vfio-pci","host":"0000:0c:00.0","id":"hostdev0","bus":"pci.6","multifunction":true,"addr":"0x0","romfile":"/var/lib/libvirt/vbios/amd03012022.rom"}' \
-device '{"driver":"vfio-pci","host":"0000:0c:00.1","id":"hostdev1","bus":"pci.7","addr":"0x0","romfile":"/var/lib/libvirt/vbios/amd03012022.rom"}' \
-device '{"driver":"usb-host","hostdevice":"/dev/bus/usb/007/003","id":"hostdev2","bus":"usb.0","port":"4"}' \
-device '{"driver":"usb-host","hostdevice":"/dev/bus/usb/007/004","id":"hostdev3","bus":"usb.0","port":"5"}' \
-device '{"driver":"usb-host","hostdevice":"/dev/bus/usb/003/002","id":"hostdev4","bus":"usb.0","port":"6"}' \
-device '{"driver":"vfio-pci","host":"0000:0c:00.2","id":"hostdev5","bus":"pci.8","addr":"0x0","romfile":"/var/lib/libvirt/vbios/amd03012022.rom"}' \
-device '{"driver":"vfio-pci","host":"0000:0c:00.3","id":"hostdev6","bus":"pci.9","addr":"0x0","romfile":"/var/lib/libvirt/vbios/amd03012022.rom"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.5","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2022-01-05T00:15:50.314809Z qemu-system-x86_64: terminating on signal 15 from pid 15492 (/usr/bin/libvirtd)
2022-01-05 00:15:53.193+0000: shutting down, reason=shutdown

I also see that the kernel driver in use is vfio-pci:

0c:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c1)
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0e3a]
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu
0c:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] [1002:ab28]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] [1002:ab28]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
0c:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a6]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a6]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci
0c:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 USB [1002:73a4]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0408]
        Kernel driver in use: vfio-pci


I also tried it to start the start.sh script and sudo virsh start win10 (via ssh and the terminal), but that doesn’t help or giving me an error that I notice atleast.

What do I miss to make this working?! Is it not possible with an AMD 6800XT reference card?

Hi, I’m pretty sure I’ve gotten to the same stage of you. I think this is caused by what they call AMD reset bug, or something like that, which is an issue on AMD cards.

I found this guide about the bug, but I haven’t tested it as I haven’t needed a Windows VM that much lately.

6*** cards don’t need the vendor-reset module, I am passing through my 6800xt without issue, but I’m also using multiple GPU’s you probably have some other process still hanging onto the card or the pass through isn’t configured properly.

@fenglengshun
Like @robbbot says, you shouldn’t need a vendor-reset for the 6xxx cards.

I saw a reddit post, that got single gpu passthrough working with an 6800 XT, but I think that’s not a reference card because of the amount of groups the gpu has in the IOMMU groups.

@robbbot
Do you have any pointers on how to take this one, I saw some reply’s of yours on some vfio issues, but they are all multiple gpus (to bad they couldn’t solve the issue).

Any chance you have the Radeon drivers installed?

I pass through a RX 6800 (dual gpu system) and have found that I need to download and install the Radeon drivers package from AMD when I run into the exact same issue.

You might be able to install the drivers via the command line if you ssh in the vm.

Only advice would be to make sure you have the right bios/uefi options set (on my board there’s multiple places to enable iommu) since you haven’t listed all your configurations (bootloader, initrd scripts etc) so it’s just a guess.

I had them all set correctly and was having similar issues, so I just reset the BIOS to default and reapplied the same settings and it started working so there was something flaky in there.

There should be no real difference between reference cards and other 6000 series cards as far as passing them through, all the iommu stuff should be handled by your motherboards bios.

Sorry, didn’t know you need more information, but In the system information I have my grub params in it, I thought that was enough.

But some good news, I got it working! I saw the following post in this thread: Installing AMD 6800xt drivers in VM with vfio passthrough leads to host crash/reboot - #23 by vadorovsky and an answer from here Installing AMD 6800xt drivers in VM with vfio passthrough leads to host crash/reboot - #25 by Sapiens

Next to the quoted bios options, I need the following setting enabled in my bios:

  • Advanced > AMD CBS > NBIO Common Options > ACS Enable > Enabled

After that I could finally see the VM and install the Radeon drivers successfully for my 6800XT! :partying_face:

I haven’t test it to the fully extend yet e.g. monitor sleeping etc. but I could play overwatch (I’m playing that one also on linux though :stuck_out_tongue: ), shutdown the vm and return successfully to my linux host. But this VM is more for titles that I can’t play on Linux yet!

1 Like

I had a very similar issue recently and what fixed it for me was enabling CSM in BIOS.

I tried that to, but that didn’t work for me