Problem booting Qemu VFIO VM with 5500XT Passthrough

[cristwelrine]

Hello,

I am trying to setup VFIO to passthrough my Radeon 5500XT to the Windows 10 VM running off of secondary SSD.

Unfortunately Google is not forthcoming with any occurences of my issues which is why I am here looking for help from those knowledgeable in the domain

The VFIO grabbing of devices before amdgpu is working, the issue is that when I try to add in the QEMU /etc/libvirt/qemu/win10.xml configuration the PCI details for the GPU & HDMI Audio, on reboot it disappears.

Sections that disappear from xml on reboot

<hostdev mode='subsystem' type='pci' managed='yes'>      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>

If I reboot and then manually run virsh define win10.xml in the folder, they appear back, same in the virt-manager GUI.

So I tried to create a service to do this

qemu-enablevms.service

[Unit]
Description=Enable QEMU VM's
After=libvirtd.service
Requires=libvirtd.service

[Service]
Type=oneshot
WorkingDirectory=/etc/libvirt/qemu
ExecStart=/bin/sh -c '/bin/cp -f win10.xml.backup win10.xml && sleep 30 && virsh define win10.xml'

[Install]
WantedBy=multi-user.target

Service output

# systemctl status qemu-enablevms.service 
● qemu-enablevms.service - Enable QEMU VM's
   Loaded: loaded (/etc/systemd/system/qemu-enablevms.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2020-04-09 16:15:13 EEST; 1h 29min ago
  Process: 1339 ExecStart=/bin/sh -c /bin/cp -f win10.xml.backup win10.xml  && virsh define win10.xm>
 Main PID: 1339 (code=exited, status=0/SUCCESS)
      CPU: 17ms

Apr 09 16:14:43 localhost.localdomain systemd[1]: Starting Enable QEMU VM's...
Apr 09 16:15:13 WIN-E1OMJTOH2V6.localdomain sh[1339]: Domain win10 defined from win10.xml
Apr 09 16:15:13 WIN-E1OMJTOH2V6.localdomain systemd[1]: qemu-enablevms.service: Succeeded.

So based on the line

Domain win10 defined from win10.xml

it should have redefined with desired xml payload but instead it shows the xml below without the relevant hostdev sections.

Desired win10.xml

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit win10
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>win10</name>
  <uuid>ec3c53c3-a4da-4945-8a83-e2012c1aea7e</uuid>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <iothreads>2</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='12'/>
    <vcpupin vcpu='2' cpuset='5'/>
    <vcpupin vcpu='3' cpuset='13'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='14'/>
    <vcpupin vcpu='6' cpuset='7'/>
    <vcpupin vcpu='7' cpuset='15'/>
    <emulatorpin cpuset='4-7,12-15'/>
    <iothreadpin iothread='1' cpuset='4-7'/>
    <iothreadpin iothread='2' cpuset='12-15'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-4.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <boot dev='hd'/>
    <bootmenu enable='no'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vpindex state='on'/>
      <synic state='on'/>
      <stimer state='on'/>
      <reset state='on'/>
      <vendor_id state='on' value='1234567890ab'/>
      <frequencies state='on'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact' check='none'>
    <model fallback='allow'>EPYC-IBPB</model>
    <topology sockets='1' cores='4' threads='2'/>
    <feature policy='require' name='topoext'/>
    <feature policy='disable' name='hypervisor'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='kvmclock' present='no'/>
    <timer name='hypervclock' present='yes'/>
    <timer name='tsc' present='yes' mode='native'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/nvme1n1'/>
      <target dev='sda' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:18:be:c3'/>
      <source dev='enp5s0' mode='bridge'/>
      <model type='e1000e'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich9'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
</domain>

The reason I am doing this systemctl serviceis due to another issue i have run into, which is trying to set autostart for vm, which responds with success but when listing it, it still is disabled.

Autostart output

[root@ qemu]# virsh autostart win10
Domain win10 marked as autostarted

[root@ qemu]# virsh dominfo win10
Id:             -
Name:           win10
UUID:           ec3c53c3-a4da-4945-8a83-e2012c1aea7e
OS Type:        hvm
State:          shut off
CPU(s):         8
Max memory:     16777216 KiB
Used memory:    16777216 KiB
Persistent:     yes
Autostart:      disable
Managed save:   no
Security model: selinux
Security DOI:   0

This is despite the fact that the symlink for the xml is created

Autostart symlink created

qemu]# ls -la /etc/libvirt/qemu/autostart/
total 8
drwxr-xr-x. 2 root root 4096 Apr  9 11:27 .
drwx------. 4 root root 4096 Apr  9 16:15 ..
lrwxrwxrwx. 1 root root   27 Apr  9 11:27 win10.xml -> /etc/libvirt/qemu/win10.xml

If needed:

Qemu Version

# /usr/bin/qemu-system-x86_64 --version
QEMU emulator version 4.1.1 (qemu-4.1.1-1.fc31)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers

IOMMU Groups

IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 10 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU Group 10 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 11 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
IOMMU Group 11 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
IOMMU Group 11 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
IOMMU Group 11 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
IOMMU Group 11 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
IOMMU Group 11 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
IOMMU Group 11 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
IOMMU Group 11 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU Group 12 01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU Group 13 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad]
IOMMU Group 14 03:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 15 03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 16 03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU Group 16 06:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 16 06:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 16 06:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 17 03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU Group 17 07:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 18 03:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU Group 18 08:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 19 04:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU Group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 20 05:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 21 09:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c5)
IOMMU Group 22 0a:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
IOMMU Group 23 0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 14 [Radeon RX 5500/5500M / Pro 5500M] [1002:7340] (rev c5)
IOMMU Group 24 0b:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio [1002:ab38]
IOMMU Group 2 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 5 00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 6 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 7 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 8 0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 9 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 9 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 9 0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 9 0d:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU Group 9 0d:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 9 0d:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]

Can somebody please help me with these issues ? I really want/need to get this PC running with VFIO passthrough When VM is running with Spice output in VM without the hostdev section, it runs very fast and CPU passthrough works, USB passthrough works, just need this last part

[mixufin]

hey!

i have similiar issue too. ewerything goes well before i try start win vm then it hangs.
maybe this has something to do whit that. Not sure if thats fixed yet or not havent tryed that patch yet. maybe digging into that this weekend…

Navi Reset Kernel Patch

Navi Reset What this does is use powerplay tables to turn the card off and back on again. (Insert IT crowd meme here, haha). @gnif has done it again. Good work!! Ok, here is the navi reset. Navi is easier to implement as the SMU has most of the logic in it. Update 27-11-2019: Updated patch down below: Navi Reset Kernel Patch From 69ea42207b544b6e3fa9755022bff09d2ce953d9 Mon Sep 17 00:00:00 2001 From: Geoffrey McRae <[email protected]> Date: Thu, 12 Sep 2019 03:19:28 +1000 Subject: [PATCH] pci quirk: AMD Navi 10 series vendor specific reset Signed-off-by: Geoffrey McRae <[email protected]> --- drivers/pci/quirks.c | 98 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 44c4ae1abd00..d94ddb1c6832 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -3825,6 +3825,97 @@ static int delay_250ms_after_flr(struct pci_dev *dev, int probe) return 0; } +/* + * AMD Navi 10 series GPUs require a vendor specific reset procedure. + * According to AMD a PSP mode 2 reset should be enough however at this + * time the details of how to perform this are not available to us. + * Instead we can signal the SMU to enter and exit BACO which has the same + * desired effect. + */ +static int reset_amd_navi10(struct pci_dev *dev, int probe) +{ + const int mmMP0_SMN_C2PMSG_81 = 0x16091; + const int mmMP1_SMN_C2PMSG_66 = 0x16282; + const int mmMP1_SMN_C2PMSG_82 = 0x16292; + const int mmMP1_SMN_C2PMSG_90 = 0x1629a; + + u16 cfg; + resource_size_t mmio_base, mmio_size; + uint32_t __iomem * mmio; + unsigned int sol; + unsigned int timeout; + + /* bus resets still cause navi to flake out */ + dev->dev_flags |= PCI_DEV_FLAGS_NO_BUS_RESET; + + if (probe) + return 0; + + /* save the PCI state and enable memory access */ + pci_save_state(dev); + pci_read_config_word(dev, PCI_COMMAND, &cfg); + pci_write_config_word(dev, PCI_COMMAND, cfg | PCI_COMMAND_MEMORY); + + /* map BAR5 …

let me know if you can dig something up.

[BensTechLab]

Any updates with your 5500XT passthrough? I just got a 5500XT myself and about to give it a go with a ProxMox build in the coming weeks.

[wendell]

Still hangs unrecoverably sometimes but other times it works fine. Ymmv. Try to use the patch but it’s imperfect.

[BensTechLab]

Thanks @wendell, that’s unfortunate. I only bought the 5500XT because I was looking for a lower power ITX card. Maybe I’d be better off with an ITX sized GTX 1650 for similar power draw? (I’m still within the return window!) I’m building a small homelab machine with 400W powersupply. Everything SFF is harder…