QEMU VFIO host hard lock

Jothite · July 14, 2020, 5:09pm

I only know VFIO and GPU passthrough exist because of level1 videos, so I figured this was probably the best place to come when I’ve hit a brick wall.

In short, the problem is hard lockups. Seemingly random. My host (Ubuntu 18.04 - 5.7 liquorix kernel) has never locked up like that on it’s own. It only happens when the guest VM is running. It can happen right away, or it can happen at any time up to hours of running/gaming. The problem I have really, is I don’t even know how to troubleshoot this. Any help with that is appreciated.

My hardware setup is an ASUS Crosshair Hero VIII WIFI, 3800x, 64gb pc3600 trident z neo, 1060 6gb host GPU, rx570 guest gpu, PCI-x USB card also passed through (USB audio and bluetooth connected to this card).

VM configuration is 8 cores, 16GB RAM, RX570 passthrough, USB card passthrough, raw image for C drive, 500gb SSD partition (by uuid) for game drive, network in macvtap bridge mode. Here’s the xml file.

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit win10
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>win10</name>
  <uuid>253bfb95-2be8-407e-be75-c821d0a3efd5</uuid>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <vcpu placement='static'>8</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-bionic'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='allow'>EPYC</model>
    <topology sockets='1' cores='4' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/NVME/vbox/win10.raw'/>
      <target dev='vda' bus='virtio'/>
      <boot order='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/disk/by-partuuid/89ee49c8-01'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:f1:fb:53'/>
      <source dev='enp5s0' mode='bridge'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x0c' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x0c' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>

IOMMU list

IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1482]
IOMMU Group 10 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1484]
IOMMU Group 11 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1482]
IOMMU Group 12 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1484]
IOMMU Group 13 00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1484]
IOMMU Group 14 00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1484]
IOMMU Group 15 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU Group 15 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 16 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1440]
IOMMU Group 16 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1441]
IOMMU Group 16 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1442]
IOMMU Group 16 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1443]
IOMMU Group 16 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1444]
IOMMU Group 16 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1445]
IOMMU Group 16 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1446]
IOMMU Group 16 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1447]
IOMMU Group 17 01:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology Device [c0a9:2263] (rev 03)
IOMMU Group 18 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57ad]
IOMMU Group 19 03:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a3]
IOMMU Group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1483]
IOMMU Group 20 03:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a3]
IOMMU Group 21 03:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a3]
IOMMU Group 22 03:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a3]
IOMMU Group 23 03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
IOMMU Group 23 08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1485]
IOMMU Group 23 08:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:149c]
IOMMU Group 23 08:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:149c]
IOMMU Group 24 03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
IOMMU Group 24 09:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 25 03:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
IOMMU Group 25 0a:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 26 04:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02)
IOMMU Group 27 05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. Device [10ec:8125]
IOMMU Group 28 06:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 29 07:00.0 Network controller [0280]: Intel Corporation Device [8086:2723] (rev 1a)
IOMMU Group 2 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1483]
IOMMU Group 30 0b:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1)
IOMMU Group 30 0b:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
IOMMU Group 31 0c:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X] [1002:67df] (rev ef)
IOMMU Group 31 0c:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
IOMMU Group 32 0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:148a]
IOMMU Group 33 0e:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1485]
IOMMU Group 34 0e:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1486]
IOMMU Group 35 0e:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:149c]
IOMMU Group 36 0e:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Device [1022:1487]
IOMMU Group 37 0f:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 38 10:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1482]
IOMMU Group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1482]
IOMMU Group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1483]
IOMMU Group 6 00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1483]
IOMMU Group 7 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1482]
IOMMU Group 8 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1482]
IOMMU Group 9 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1482]

/etc/modprobe.d/vfio.conf

options vfio-pci ids=1002:67df,1002:aaf0,10ec:8125,1912:0015

/etc/modules

vfio vfio_iommu_type1 vfio_pci ids=1002:67df,1002:aaf0,10ec:8125,1912:0015

/etc/initramfs/tools/modules

vfio vfio_iommu_type1 vfio_pci ids=1002:67df,1002:aaf0,10ec:8125,1912:0015

/etc/modprobe.d/kvm.conf

vfio vfio_iommu_type1 vfio_pci ids=1002:67df,1002:aaf0,10ec:8125,1912:0015

/var/log/libvirt/qemu/win10.log

2020-07-13 21:45:18.556+0000: starting up libvirt version: 4.0.0, package: 1ubuntu8.17 (Marc Deslauriers <[email protected]> Wed, 06 May 2020 14:18:23 -0400), qemu version: 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.28), hostname: Optiplex
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name guest=win10,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-win10/master-key.aes -machine pc-i440fx-bionic,accel=kvm,usb=off,vmport=off,dump-guest-core=off -cpu EPYC,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -drive file=/usr/share/OVMF/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/win10_VARS.fd,if=pflash,format=raw,unit=1 -m 16384 -realtime mlock=off -smp 8,sockets=1,cores=4,threads=2 -uuid 253bfb95-2be8-407e-be75-c821d0a3efd5 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-win10/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device ahci,id=sata0,bus=pci.0,addr=0x6 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive file=/mnt/NVME/vbox/win10.raw,format=raw,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -drive file=/dev/disk/by-partuuid/89ee49c8-01,format=raw,if=none,id=drive-virtio-disk1,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk1,id=virtio-disk1 -drive if=none,id=drive-sata0-0-0,media=cdrom,readonly=on -device ide-cd,bus=sata0.0,drive=drive-sata0-0-0,id=sata0-0-0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device usb-host,hostbus=7,hostaddr=2,id=hostdev0,bus=usb.0,port=4 -device vfio-pci,host=0c:00.0,id=hostdev1,bus=pci.0,addr=0x2 -device vfio-pci,host=0c:00.1,id=hostdev2,bus=pci.0,addr=0xa -device vfio-pci,host=04:00.0,id=hostdev3,bus=pci.0,addr=0xb -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
2020-07-13T21:45:18.635521Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/1 (label charserial0)
2020-07-13T21:45:18.653481Z qemu-system-x86_64: AMD CPU doesn't support hyperthreading. Please configure -smp options properly.
2020-07-13T21:45:52.516924Z qemu-system-x86_64: warning: guest updated active QH
2020-07-13T21:46:58.626706Z qemu-system-x86_64: warning: guest updated active QH
2020-07-13T21:49:14.077545Z qemu-system-x86_64: warning: guest updated active QH
2020-07-13T21:49:20.203320Z qemu-system-x86_64: terminating on signal 15 from pid 1564 (/usr/sbin/libvirtd)
2020-07-13 21:49:22.606+0000: shutting down, reason=shutdown
2020-07-13 21:53:52.833+0000: starting up libvirt version: 4.0.0, package: 1ubuntu8.17 (Marc Deslauriers <[email protected]> Wed, 06 May 2020 14:18:23 -0400), qemu version: 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.28), hostname: Optiplex
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name guest=win10,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-win10/master-key.aes -machine pc-i440fx-bionic,accel=kvm,usb=off,vmport=off,dump-guest-core=off -cpu EPYC,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -drive file=/usr/share/OVMF/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/win10_VARS.fd,if=pflash,format=raw,unit=1 -m 16384 -realtime mlock=off -smp 8,sockets=1,cores=4,threads=2 -uuid 253bfb95-2be8-407e-be75-c821d0a3efd5 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-win10/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device ahci,id=sata0,bus=pci.0,addr=0x6 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive file=/mnt/NVME/vbox/win10.raw,format=raw,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -drive file=/dev/disk/by-partuuid/89ee49c8-01,format=raw,if=none,id=drive-virtio-disk1,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk1,id=virtio-disk1 -drive if=none,id=drive-sata0-0-0,media=cdrom,readonly=on -device ide-cd,bus=sata0.0,drive=drive-sata0-0-0,id=sata0-0-0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device vfio-pci,host=0c:00.0,id=hostdev0,bus=pci.0,addr=0x2 -device vfio-pci,host=0c:00.1,id=hostdev1,bus=pci.0,addr=0xa -device vfio-pci,host=04:00.0,id=hostdev2,bus=pci.0,addr=0xb -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
2020-07-13T21:53:52.907761Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/0 (label charserial0)
2020-07-13T21:53:52.925811Z qemu-system-x86_64: AMD CPU doesn't support hyperthreading. Please configure -smp options properly.
2020-07-13 22:15:34.142+0000: starting up libvirt version: 4.0.0, package: 1ubuntu8.17 (Marc Deslauriers <[email protected]> Wed, 06 May 2020 14:18:23 -0400), qemu version: 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.28), hostname: Optiplex
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name guest=win10,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-win10/master-key.aes -machine pc-i440fx-bionic,accel=kvm,usb=off,vmport=off,dump-guest-core=off -cpu EPYC,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -drive file=/usr/share/OVMF/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/win10_VARS.fd,if=pflash,format=raw,unit=1 -m 16384 -realtime mlock=off -smp 8,sockets=1,cores=4,threads=2 -uuid 253bfb95-2be8-407e-be75-c821d0a3efd5 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-win10/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device ahci,id=sata0,bus=pci.0,addr=0x6 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive file=/mnt/NVME/vbox/win10.raw,format=raw,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -drive file=/dev/disk/by-partuuid/89ee49c8-01,format=raw,if=none,id=drive-virtio-disk1,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk1,id=virtio-disk1 -drive if=none,id=drive-sata0-0-0,media=cdrom,readonly=on -device ide-cd,bus=sata0.0,drive=drive-sata0-0-0,id=sata0-0-0 -netdev tap,fd=21,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:f1:fb:53,bus=pci.0,addr=0xc -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device vfio-pci,host=0c:00.0,id=hostdev0,bus=pci.0,addr=0x2 -device vfio-pci,host=0c:00.1,id=hostdev1,bus=pci.0,addr=0xa -device vfio-pci,host=04:00.0,id=hostdev2,bus=pci.0,addr=0xb -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
2020-07-13T22:15:34.224139Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/0 (label charserial0)
2020-07-13T22:15:34.242223Z qemu-system-x86_64: AMD CPU doesn't support hyperthreading. Please configure -smp options properly.

Let me know if you need anything else.

Jothite · July 15, 2020, 10:22pm

I might have fixed it. I didn’t do much really. Just removed some of the unneeded virtual hardware and pinned the cores. I left the VM up for hours and played for hours after that and it’s never stayed up that long yet.
I do have one question. My host still seems to be able to see the USB audio device that is plugged into the passed through PCI-e USB card. (I did also turn that off in the sound settings in Ubuntu.) The card is definitely passing through because I’m using the bluetooth device in the guest. So how do I get Ubuntu to ignore things that get plugged in to that card?

Jothite · July 16, 2020, 5:54pm

Yes, something still isn’t quite right. The only device listed as using the vfio-pci driver is the RX570. I added the top three lines to the /etc/modprobe.d/vfio.conf file:

alias pci:v00001912d00000015sv0000FFFFsd0000FFFFbc0Csc03i30 vfio-pci
alias pci:v00001002d000067DFsv00001043sd000004C2bc03sc00i00 vfio-pci
alias pci:v00001002d0000AAF0sv00001043sd0000AAF0bc04sc03i00 vfio-pci
options vfio-pci ids=1002:67df,1002:aaf0,1912:0015

But Ubuntu can still see the USB sound card and the bluetooth adapter under lsusb. I’m not sure if that is the expected result. But I’m fairly positive I shouldn’t be able to select that USB device as an audio output on the host.

Jothite · July 22, 2020, 4:07pm

I’m a little stumped for real now. The usb card still isn’t using the virtio driver after a reboot. I can manually unbind and rebind it but should I have to be doing that? There’s got to be a way to get it to pass through automatically. Or, there’s got to be a better way than scripting it to unbind and rebind on boot.
Any help is appreciated.

Jothite · July 24, 2020, 11:38pm

Well I’m not getting much love here am I? Too much information or not enough? Too simple or too hard?

pantato · July 25, 2020, 4:45am

Try adding the USB device through the virt-manager GUI. Don’t passthrough the entire card. It’s probably needing the acs-override and the acs-override is not so great.
EDIT: Your IOMMU groupings look like you might be using the acs-override already. It might be the root of all this weirdness.

Jothite · July 25, 2020, 1:57pm

LOL, I don’t know what that means, but I’ll look into it, thanks.

EDIT: after a quick review, I don’t think I did anything with acs-overrides. That is an add in USB card and was already it’s own IOMMU group.

As far as I can tell (but I’m an idiot) the only thing not working correctly right now, is that USB card isn’t picking up the vfio-pci driver on boot. I’m having to manually unbind and bind it (which sudo can’t even do, who knew?). I might need to re-upload configuration files again, because I have made a few small changes, but mostly just cleaning up unneeded virtual hardware.
Just in case I’m a huge idiot, where would the acs overrides be? In virsh or in a vfio conf file somewhere? The post I read was not specific.

Pollomoolokki · July 25, 2020, 2:46pm

Hello and welcome to lvl1techs!

virt-manager is the application called “Virtual machine manager”. It can be found from the repos as “virt-manager” It’s used to manage libvirt via GUI.

acs-override is a patch for the kernel to separate the IOMMU groups. (Most new mobos have sane IOMMU groups - Some don’t. Mine needed the acs-patch to separate gpu1 and gpu2 to different IOMMU group. You might have heard wendell talking about IOMMU groups in any of the mobo reviews)

You have bound the vfio-drivers via modprobe - according to Archwiki_PCI_passthrough_via_OVMF#Isolating_the_GPU you need to regenerate the initramfs " Since new modules have been added to the initramfs configuration, you must regenerate the initramfs."

You can also bind the vfio-drivers via kernel parameters.

Looking at your modprobe files - all seem to be a-ok. What does “lspci -k” return? it should show the three(?) devices bound to vfio-pci (ids=1002:67df,1002:aaf0,1912:0015)

Jothite · July 25, 2020, 3:05pm

Thank you. I am using virt-manager mostly. I did edit a couple things manually with virsh.

I don’t remember installing the acs-override patch at any point. The first thing I did when beginning this was check IOMMU groups and they looked good to me as far as things being separate. Not saying for certain I did not install it though. I’m still a bit lost on it, but does the acs-override get installed into the kernel? I’m using the liquorix kernel. I don’t know if it’s on by default.

I have been running ‘sudo update-initramfs -u -k all’ after making changes to the modprobe files.

I will look for this next, but if you know where I can find out how to bind them via kernel parameters I’d be happy to try that.

Right now lspci -k lists vfio-pci as the driver. But only because I unbound it from xhci_pci and bound it to vfio-pci after I booted Ubuntu. I ran these commands.

echo '0000:04:00.0' > /sys/bus/pci/drivers/xhci_hcd/unbind
echo '0000:04:00.0' > /sys/bus/pci/drivers/vfio-pci/bind

This “works” but the problem is ubuntu is grabbing the USB devices that are plugged into that card and “holding” them until I unplug and replug them, after booting the guest.
Ideally I would think the behavior would be for it to load vfio-pci from the start and for the host to completely ignore devices plugged into the USB card. So it sounds like I need to pass kernel parameters through to do that?

Jothite · July 25, 2020, 3:25pm

I built and installed driverctl from upstream and used that but I’m not sure if it did anything I didn’t manually do already. I already added some driver overrides in /sys/bus/pci/devices/000:04:00.0/driver_override
I will try rebooting in a bit and see if that made any difference though.

# driverctl set-override 0000:04:00.0 vfio-pci
# driverctl -v list-overrides
0000:04:00.0 vfio-pci (uPD720202 USB 3.0 Host Controller)

Jothite · July 25, 2020, 3:38pm

Well I’ll be, that did it. lspci -k now lists vfio-pci driver being loaded to the USB card directly after boot without doing anything else. The audio device isn’t listed and bluetooth won’t sync with the devices. Looks good so far. Thanks a lot for the information. I didn’t know what else to search for.

Pollomoolokki · July 25, 2020, 3:48pm

The first thing I did when beginning this was check IOMMU groups and they looked good to me as far as things being separate.

Pretty sure you don’t have the acs-override enabled… You either have to install a kernel with the patch enabled or compile the kernel yourself with the patch. (A quick look at the liquorix kernel tells me that it does not have the acs-override)

Archwiki has a good article on Kernel_parameters

Have you tried adding the usb devices that are connected to your pcie-usb card to your guest? There is a button in virt-manager called “+ add hardware” and from there you can add your usb devices. example image

lspci -k now lists vfio-pci driver being loaded to the USB card directly

woohoo great to hear!

Jothite · July 25, 2020, 5:36pm

Things mostly appear to be operating as they should now. I’m getting a lock up on the guest, but I think that’s a windows problem, not a vfio problem. When it was having io issues it would take the host down with it. Thanks again. I really do prefer to have it done this way, passing through an entire USB card. That way I can just connect any peripheral I want to that USB and I’m good. Don’t have to worry about passing usb drives and things through which was always a huge annoyance with VMs for me. Really appreciate the help.

pantato · July 25, 2020, 8:12pm

Just so you know you can add and remove individual USB devices to your VM with virt-manager while the VM is running without passing through entire USB controllers / cards. A lot of people seem to not know this so my suggestion was for the purpose of kind of clarifying this and hopefully making it easier and skipping the headache of the controller binding issue you were having. Glad you got it figured out, though.

Jothite · July 25, 2020, 10:36pm

Thank you. I was doing that initially, and that totally would still work (although ubuntu still tried to play audio out the passed through USB sound card once and crashed everything). I just prefer to have a dedicated card/USB hub on my desk that is ready to go. It’s been kind of a dream since I started messing with virtualization long long ago. So now that the software is ready and I finally have hardware good enough to do it, I’m just super happy it’s working. Thanks again.

pantato · July 25, 2020, 10:40pm

Sure thing. Also, are you privy to evdev and scream?

Jothite · July 25, 2020, 10:52pm

I was not. I just did a quick search to see what it was. Looks like it’s device and pulseaudio sharing stuff?
Right now I’m using logitech lightspeed keyboard and mouse which both have bluetooth as well. So I just have a bluetooth adapter on the VM and the lightspeed adapter in my host PC.

I was also using barrier to share the keyboard and mouse, but it was causing my guest to lock up. I don’t know why, but uninstalling it fixed my problem. I reinstalled and it came back. Uninstalled and it went away again. So I’m guessing it was barrier at least.

For the audio I really only use a headset when I game. Nothing else I do in windows needs sound. But I might still be interested in scream though… I’ll have to look into that. Thanks

pantato · July 25, 2020, 11:02pm

Yeah evdev and scream are really nice. Evdev let’s you switch mouse and keyboard between guest and host by pressing l-ctrl and r-ctrl together. Scream is just seamless audio passthrough via network. All I gotta do to turn that on is type “scream” into a terminal which I have aliased to ‘scream-pulse -i virbr0"’ There’s virtually no latency involved with either. Those along with looking-glass allows me to use my Windows VM in bed where my TV faces me without having to get up and go to my desk with my gaming monitor. Which makes it ideal for games that involve console controllers like Breath of the Wild on the CEMU emulator or Sekiro. Windows literally just becomes a window on my XFCE/Arch desktop. But I’ll get up and go to the desk with my gaming monitor when I wanna play CS or Quake.