AMD Single GPU Passthrough Problems (Code 43)

sp33dy905 · March 20, 2021, 4:10am

Hello all!

I was following guides (Arch Wiki, Github, SomeOrdinaryGamers on YT) on setting up a single GPU passthrough, and while I was successful, I am stuck on the infamous Code 43 error.

Specs:
I5-10600KF
MSI Z490-A PRO
PowerColor RX5700 XT
64GB of RAM
Arch Linux on 5.11.7 on the Zen Kernel

Here’s what I’ve tried so far:

Doing the Vendor ID Trick
Turning on Hidden state (I know this is for NVIDIA cards, but I wanted to try just in case)
Checking that CSM is disabled (It’s set to UEFI)
Installing the vendor reset package from the AUR (My GPU still hangs after a shutdown of the VM, so I’m not sure if it’s related to this. I’ve only tried the dkms version, not the regular one)

Here are the information that you guys probably need in order to troubleshoot properly:

IOMMU groups:

IOMMU Group 0:
	00:00.0 Host bridge [0600]: Intel Corporation Device [8086:9b53] (rev 05)
IOMMU Group 1:
	00:01.0 PCI bridge [0604]: Intel Corporation 6th-9th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05)
	01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c1)
	02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
	03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] [1002:731f] (rev c1)
	03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio [1002:ab38]
IOMMU Group 2:
	00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model [8086:1911]
IOMMU Group 3:
	00:12.0 Signal processing controller [1180]: Intel Corporation Comet Lake PCH Thermal Controller [8086:06f9]
IOMMU Group 4:
	00:14.0 USB controller [0c03]: Intel Corporation Comet Lake USB 3.1 xHCI Host Controller [8086:06ed]
	00:14.2 RAM memory [0500]: Intel Corporation Comet Lake PCH Shared SRAM [8086:06ef]
IOMMU Group 5:
	00:16.0 Communication controller [0780]: Intel Corporation Comet Lake HECI Controller [8086:06e0]
IOMMU Group 6:
	00:17.0 SATA controller [0106]: Intel Corporation Device [8086:06d2]
IOMMU Group 7:
	00:1b.0 PCI bridge [0604]: Intel Corporation Comet Lake PCI Express Root Port #17 [8086:06c0] (rev f0)
IOMMU Group 8:
	00:1b.4 PCI bridge [0604]: Intel Corporation Comet Lake PCI Express Root Port #21 [8086:06ac] (rev f0)
IOMMU Group 9:
	00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:06b8] (rev f0)
IOMMU Group 10:
	00:1c.4 PCI bridge [0604]: Intel Corporation Device [8086:06bc] (rev f0)
IOMMU Group 11:
	00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:0685]
	00:1f.3 Audio device [0403]: Intel Corporation Comet Lake PCH cAVS [8086:06c8]
	00:1f.4 SMBus [0c05]: Intel Corporation Comet Lake PCH SMBus Controller [8086:06a3]
	00:1f.5 Serial bus controller [0c80]: Intel Corporation Comet Lake PCH SPI Controller [8086:06a4]
IOMMU Group 12:
	04:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation E16 PCIe4 NVMe Controller [1987:5016] (rev 01)
IOMMU Group 13:
	05:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Blue SN550 NVMe SSD [15b7:5009] (rev 01)
IOMMU Group 14:
	07:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 04)

start.sh script

#!/bin/bash
# Helpful to read output when debugging
set -x

source "/etc/libvirt/hooks/kvm.conf"

# Stop display manager
systemctl stop sddm.service
#I was experimenting with pulseaudio, disabling these didn't change anything
#pulse_pid=$(pgrep -u [username] pulseaudio)  
#pipewire_pid=$(pgrep -u [username] pipewire-media)
#kill $pulse_pid
#kill $pipewire_pid

# Unbind VTconsoles
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind

# Unbind EFI-Framebuffer
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

# Avoid a Race condition by waiting 2 seconds. This can be calibrated to be shorter or longer if required for your system
sleep 5

# Unload all Radeon drivers
modprobe -r amdgpu

# Unbind the GPU from display driver
virsh nodedev-detach $VIRSH_GPU_VIDEO
virsh nodedev-detach $VIRSH_GPU_AUDIO

# Load VFIO Kernel Module  
modprobe vfio-pci  
modprobe vfio_pci
modprobe vfio_iommu_type1

revert.sh script

#!/bin/bash
# Helpful to read output when debugging
set -x

# Load the config file with our environmental variables
source "/etc/libvirt/hooks/kvm.conf"

# Unload all the vfio modules
modprobe -r vfio_pci
modprobe -r vfio_iommu_type1
modprobe -r vfio

# Reattach the gpu
virsh nodedev-reattach $VIRSH_GPU_VIDEO
virsh nodedev-reattach $VIRSH_GPU_AUDIO

echo 1 > /sys/class/vtconsole/vtcon0/bind
echo 1 > /sys/class/vtconsole/vtcon1/bind


# Load all Radeon drivers
modprobe  amdgpu
modprobe  xhci_pci
modprobe  drm_kms_helper
modprobe  drm
modprobe  snd_hda_intel

#Start you display manager
systemctl start sddm.service

kvm.conf

VIRSH_GPU_VIDEO=pci_0000_03_00_0
VIRSH_GPU_AUDIO=pci_0000_03_00_1

lspci -v (Let me know if you want the full output)

03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] (rev c1) (prog-if 00 [VGA controller])
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0b36
	Flags: bus master, fast devsel, latency 0, IRQ 155, IOMMU group 1
	Memory at 4200000000 (64-bit, prefetchable) [size=8G]
	Memory at 4100000000 (64-bit, prefetchable) [size=2M]
	I/O ports at 4000 [size=256]
	Memory at a0000000 (32-bit, non-prefetchable) [size=512K]
	Expansion ROM at a0080000 [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu

03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio
	Flags: bus master, fast devsel, latency 0, IRQ 154, IOMMU group 1
	Memory at a00a0000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

Last log of VM launch

2021-03-20 00:33:54.260+0000: starting up libvirt version: 7.0.0, qemu version: 5.2.0, kernel: 5.11.7-zen1-1-zen, hostname: archyboi.localdomain
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-1-win10 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-win10/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-win10/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-win10/.config \
QEMU_AUDIO_DRV=spice \
/usr/bin/qemu-system-x86_64 \
-name guest=win10,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-win10/master-key.aes \
-blockdev '{"driver":"file","filename":"/usr/share/edk2-ovmf/x64/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/win10_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-5.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \
-cpu Cooperlake,ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,mpx=on,umip=on,md-clear=on,xsaves=on,ibpb=on,amd-stibp=on,amd-ssbd=on,hle=off,rtm=off,avx512f=off,avx512dq=off,clwb=off,avx512cd=off,avx512bw=off,avx512vl=off,avx512vnni=off,avx512-bf16=off,taa-no=off,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff \
-m 32000 \
-object memory-backend-ram,id=pc.ram,size=33554432000 \
-overcommit mem-lock=off \
-smp 10,sockets=1,dies=1,cores=5,threads=2 \
-uuid e9c1e261-069a-4a98-b6de-8c2b4ca79c12 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=31,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-reboot \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 \
-blockdev '{"driver":"file","filename":"/mnt/1TB/KVM-VM/win10.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null}' \
-device ide-hd,bus=ide.0,drive=libvirt-2-format,id=sata0-0-0,bootindex=2 \
-blockdev '{"driver":"file","filename":"/mnt/2TB/tmp/Win10_20H2_v2_English_x64.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-cd,bus=ide.1,drive=libvirt-1-format,id=sata0-0-1,bootindex=1 \
-netdev tap,fd=34,id=hostnet0 \
-device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:41:4b:13,bus=pci.1,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev spicevmc,id=charchannel0,name=vdagent \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 \
-device ich9-intel-hda,id=sound0,bus=pcie.0,addr=0x1b \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-chardev spicevmc,id=charredir0,name=usbredir \
-device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 \
-chardev spicevmc,id=charredir1,name=usbredir \
-device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 \
-device virtio-balloon-pci,id=balloon0,bus=pci.4,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/1 (label charserial0)
2021-03-20 00:36:28.260+0000: shutting down, reason=shutdown
2021-03-20 00:36:28.371+0000: starting up libvirt version: 7.0.0, qemu version: 5.2.0, kernel: 5.11.7-zen1-1-zen, hostname: archyboi.localdomain
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-2-win10 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-2-win10/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-2-win10/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-2-win10/.config \
QEMU_AUDIO_DRV=spice \
/usr/bin/qemu-system-x86_64 \
-name guest=win10,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-win10/master-key.aes \
-blockdev '{"driver":"file","filename":"/usr/share/edk2-ovmf/x64/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/win10_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-5.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \
-cpu Cooperlake,ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,mpx=on,umip=on,md-clear=on,xsaves=on,ibpb=on,amd-stibp=on,amd-ssbd=on,hle=off,rtm=off,avx512f=off,avx512dq=off,clwb=off,avx512cd=off,avx512bw=off,avx512vl=off,avx512vnni=off,avx512-bf16=off,taa-no=off,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff \
-m 32000 \
-object memory-backend-ram,id=pc.ram,size=33554432000 \
-overcommit mem-lock=off \
-smp 10,sockets=1,dies=1,cores=5,threads=2 \
-uuid e9c1e261-069a-4a98-b6de-8c2b4ca79c12 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=31,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 \
-blockdev '{"driver":"file","filename":"/mnt/1TB/KVM-VM/win10.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null}' \
-device ide-hd,bus=ide.0,drive=libvirt-2-format,id=sata0-0-0,bootindex=2 \
-blockdev '{"driver":"file","filename":"/mnt/2TB/tmp/Win10_20H2_v2_English_x64.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-cd,bus=ide.1,drive=libvirt-1-format,id=sata0-0-1,bootindex=1 \
-netdev tap,fd=34,id=hostnet0 \
-device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:41:4b:13,bus=pci.1,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev spicevmc,id=charchannel0,name=vdagent \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 \
-device ich9-intel-hda,id=sound0,bus=pcie.0,addr=0x1b \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-chardev spicevmc,id=charredir0,name=usbredir \
-device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 \
-chardev spicevmc,id=charredir1,name=usbredir \
-device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 \
-device virtio-balloon-pci,id=balloon0,bus=pci.4,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/1 (label charserial0)
2021-03-20T00:41:29.852115Z qemu-system-x86_64: terminating on signal 15 from pid 3224 (/usr/bin/libvirtd)
2021-03-20 00:41:30.072+0000: shutting down, reason=shutdown
2021-03-20 02:08:42.578+0000: starting up libvirt version: 7.0.0, qemu version: 5.2.0, kernel: 5.11.7-zen1-1-zen, hostname: archyboi.localdomain
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-1-win10 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-win10/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-win10/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-win10/.config \
QEMU_AUDIO_DRV=none \
/usr/bin/qemu-system-x86_64 \
-name guest=win10,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-win10/master-key.aes \
-blockdev '{"driver":"file","filename":"/usr/share/edk2-ovmf/x64/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/win10_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-5.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \
-cpu Cooperlake,ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,mpx=on,umip=on,md-clear=on,xsaves=on,ibpb=on,amd-stibp=on,amd-ssbd=on,hle=off,rtm=off,avx512f=off,avx512dq=off,clwb=off,avx512cd=off,avx512bw=off,avx512vl=off,avx512vnni=off,avx512-bf16=off,taa-no=off,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff \
-m 32000 \
-object memory-backend-ram,id=pc.ram,size=33554432000 \
-overcommit mem-lock=off \
-smp 10,sockets=1,dies=1,cores=5,threads=2 \
-uuid e9c1e261-069a-4a98-b6de-8c2b4ca79c12 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=30,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=0x8,chassis=6,id=pci.6,bus=pcie.0,addr=0x1 \
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 \
-blockdev '{"driver":"file","filename":"/mnt/1TB/KVM-VM/win10.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null}' \
-device ide-hd,bus=ide.0,drive=libvirt-2-format,id=sata0-0-0,bootindex=2 \
-blockdev '{"driver":"file","filename":"/mnt/2TB/tmp/Win10_20H2_v2_English_x64.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-cd,bus=ide.1,drive=libvirt-1-format,id=sata0-0-1,bootindex=1 \
-netdev tap,fd=33,id=hostnet0 \
-device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:41:4b:13,bus=pci.1,addr=0x0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-device ich9-intel-hda,id=sound0,bus=pcie.0,addr=0x1b \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-chardev spicevmc,id=charredir0,name=usbredir \
-device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 \
-chardev spicevmc,id=charredir1,name=usbredir \
-device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 \
-device usb-host,hostdevice=/dev/bus/usb/001/002,id=hostdev0,bus=usb.0,port=4 \
-device usb-host,hostdevice=/dev/bus/usb/001/003,id=hostdev1,bus=usb.0,port=5 \
-device usb-host,hostdevice=/dev/bus/usb/001/004,id=hostdev2,bus=usb.0,port=6 \
-device vfio-pci,host=0000:03:00.0,id=hostdev3,bus=pci.5,addr=0x0 \
-device vfio-pci,host=0000:03:00.1,id=hostdev4,bus=pci.6,addr=0x0 \
-device virtio-balloon-pci,id=balloon0,bus=pci.4,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2021-03-20T02:08:46.529636Z qemu-system-x86_64: vfio: Cannot reset device 0000:03:00.1, no available reset mechanism.
2021-03-20T02:08:46.533730Z qemu-system-x86_64: vfio: Cannot reset device 0000:03:00.1, no available reset mechanism.
2021-03-20T02:11:41.062673Z qemu-system-x86_64: terminating on signal 15 from pid 3215 (/usr/bin/libvirtd)
2021-03-20 02:11:43.703+0000: shutting down, reason=shutdown
2021-03-20 02:19:22.695+0000: starting up libvirt version: 7.0.0, qemu version: 5.2.0, kernel: 5.11.7-zen1-1-zen, hostname: archyboi.localdomain
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-1-win10 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-win10/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-win10/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-win10/.config \
QEMU_AUDIO_DRV=none \
/usr/bin/qemu-system-x86_64 \
-name guest=win10,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-win10/master-key.aes \
-blockdev '{"driver":"file","filename":"/usr/share/edk2-ovmf/x64/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/win10_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-5.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \
-cpu Cooperlake,ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,mpx=on,umip=on,md-clear=on,xsaves=on,ibpb=on,amd-stibp=on,amd-ssbd=on,hle=off,rtm=off,avx512f=off,avx512dq=off,clwb=off,avx512cd=off,avx512bw=off,avx512vl=off,avx512vnni=off,avx512-bf16=off,taa-no=off,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vendor-id=winvmboi \
-m 32000 \
-object memory-backend-ram,id=pc.ram,size=33554432000 \
-overcommit mem-lock=off \
-smp 10,sockets=1,dies=1,cores=5,threads=2 \
-uuid e9c1e261-069a-4a98-b6de-8c2b4ca79c12 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=31,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=0x8,chassis=6,id=pci.6,bus=pcie.0,addr=0x1 \
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 \
-blockdev '{"driver":"file","filename":"/mnt/1TB/KVM-VM/win10.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null}' \
-device ide-hd,bus=ide.0,drive=libvirt-2-format,id=sata0-0-0,bootindex=2 \
-blockdev '{"driver":"file","filename":"/mnt/2TB/tmp/Win10_20H2_v2_English_x64.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-cd,bus=ide.1,drive=libvirt-1-format,id=sata0-0-1,bootindex=1 \
-netdev tap,fd=33,id=hostnet0 \
-device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:41:4b:13,bus=pci.1,addr=0x0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-device ich9-intel-hda,id=sound0,bus=pcie.0,addr=0x1b \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-chardev spicevmc,id=charredir0,name=usbredir \
-device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 \
-chardev spicevmc,id=charredir1,name=usbredir \
-device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 \
-device usb-host,hostdevice=/dev/bus/usb/001/002,id=hostdev0,bus=usb.0,port=4 \
-device usb-host,hostdevice=/dev/bus/usb/001/003,id=hostdev1,bus=usb.0,port=5 \
-device usb-host,hostdevice=/dev/bus/usb/001/004,id=hostdev2,bus=usb.0,port=6 \
-device vfio-pci,host=0000:03:00.0,id=hostdev3,bus=pci.5,addr=0x0 \
-device vfio-pci,host=0000:03:00.1,id=hostdev4,bus=pci.6,addr=0x0 \
-device virtio-balloon-pci,id=balloon0,bus=pci.4,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2021-03-20T02:19:26.716607Z qemu-system-x86_64: vfio: Cannot reset device 0000:03:00.1, no available reset mechanism.
2021-03-20T02:19:26.720583Z qemu-system-x86_64: vfio: Cannot reset device 0000:03:00.1, no available reset mechanism.
2021-03-20T02:22:35.307020Z qemu-system-x86_64: terminating on signal 15 from pid 464 (/usr/bin/libvirtd)
2021-03-20 02:22:37.942+0000: shutting down, reason=shutdown
2021-03-20 02:29:19.978+0000: starting up libvirt version: 7.0.0, qemu version: 5.2.0, kernel: 5.11.7-zen1-1-zen, hostname: archyboi.localdomain
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-1-win10 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-win10/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-win10/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-win10/.config \
QEMU_AUDIO_DRV=none \
/usr/bin/qemu-system-x86_64 \
-name guest=win10,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-win10/master-key.aes \
-blockdev '{"driver":"file","filename":"/usr/share/edk2-ovmf/x64/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/win10_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-5.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \
-cpu Cooperlake,ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,mpx=on,umip=on,md-clear=on,xsaves=on,ibpb=on,amd-stibp=on,amd-ssbd=on,hle=off,rtm=off,avx512f=off,avx512dq=off,clwb=off,avx512cd=off,avx512bw=off,avx512vl=off,avx512vnni=off,avx512-bf16=off,taa-no=off,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vendor-id=winvmboi,kvm=off \
-m 32000 \
-object memory-backend-ram,id=pc.ram,size=33554432000 \
-overcommit mem-lock=off \
-smp 10,sockets=1,dies=1,cores=5,threads=2 \
-uuid e9c1e261-069a-4a98-b6de-8c2b4ca79c12 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=31,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=0x8,chassis=6,id=pci.6,bus=pcie.0,addr=0x1 \
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 \
-blockdev '{"driver":"file","filename":"/mnt/1TB/KVM-VM/win10.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null}' \
-device ide-hd,bus=ide.0,drive=libvirt-2-format,id=sata0-0-0,bootindex=2 \
-blockdev '{"driver":"file","filename":"/mnt/2TB/tmp/Win10_20H2_v2_English_x64.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-cd,bus=ide.1,drive=libvirt-1-format,id=sata0-0-1,bootindex=1 \
-netdev tap,fd=33,id=hostnet0 \
-device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:41:4b:13,bus=pci.1,addr=0x0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-device ich9-intel-hda,id=sound0,bus=pcie.0,addr=0x1b \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-chardev spicevmc,id=charredir0,name=usbredir \
-device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 \
-chardev spicevmc,id=charredir1,name=usbredir \
-device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 \
-device usb-host,hostdevice=/dev/bus/usb/001/002,id=hostdev0,bus=usb.0,port=4 \
-device usb-host,hostdevice=/dev/bus/usb/001/003,id=hostdev1,bus=usb.0,port=5 \
-device usb-host,hostdevice=/dev/bus/usb/001/004,id=hostdev2,bus=usb.0,port=6 \
-device vfio-pci,host=0000:03:00.0,id=hostdev3,bus=pci.5,addr=0x0 \
-device vfio-pci,host=0000:03:00.1,id=hostdev4,bus=pci.6,addr=0x0 \
-device virtio-balloon-pci,id=balloon0,bus=pci.4,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2021-03-20T02:29:23.990745Z qemu-system-x86_64: vfio: Cannot reset device 0000:03:00.1, no available reset mechanism.
2021-03-20T02:29:23.994723Z qemu-system-x86_64: vfio: Cannot reset device 0000:03:00.1, no available reset mechanism.
2021-03-20T02:36:14.021632Z qemu-system-x86_64: vfio: Cannot reset device 0000:03:00.1, no available reset mechanism.
2021-03-20T02:36:14.026748Z qemu-system-x86_64: vfio: Cannot reset device 0000:03:00.1, no available reset mechanism.
2021-03-20T02:38:03.880538Z qemu-system-x86_64: vfio: Cannot reset device 0000:03:00.1, no available reset mechanism.
2021-03-20T02:38:03.885726Z qemu-system-x86_64: vfio: Cannot reset device 0000:03:00.1, no available reset mechanism.
2021-03-20T02:41:33.957888Z qemu-system-x86_64: terminating on signal 15 from pid 447 (/usr/bin/libvirtd)
2021-03-20 02:41:36.594+0000: shutting down, reason=shutdown
2021-03-20 03:08:19.741+0000: starting up libvirt version: 7.0.0, qemu version: 5.2.0, kernel: 5.11.7-zen1-1-zen, hostname: archyboi.localdomain
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-1-win10 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-win10/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-win10/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-win10/.config \
QEMU_AUDIO_DRV=none \
/usr/bin/qemu-system-x86_64 \
-name guest=win10,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-win10/master-key.aes \
-blockdev '{"driver":"file","filename":"/usr/share/edk2-ovmf/x64/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/win10_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-5.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \
-cpu Cooperlake,ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,mpx=on,umip=on,md-clear=on,xsaves=on,ibpb=on,amd-stibp=on,amd-ssbd=on,hle=off,rtm=off,avx512f=off,avx512dq=off,clwb=off,avx512cd=off,avx512bw=off,avx512vl=off,avx512vnni=off,avx512-bf16=off,taa-no=off,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vendor-id=123456789ab,kvm=off \
-m 32000 \
-object memory-backend-ram,id=pc.ram,size=33554432000 \
-overcommit mem-lock=off \
-smp 10,sockets=1,dies=1,cores=5,threads=2 \
-uuid e9c1e261-069a-4a98-b6de-8c2b4ca79c12 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=31,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=0x8,chassis=6,id=pci.6,bus=pcie.0,addr=0x1 \
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 \
-blockdev '{"driver":"file","filename":"/mnt/1TB/KVM-VM/win10.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null}' \
-device ide-hd,bus=ide.0,drive=libvirt-2-format,id=sata0-0-0,bootindex=2 \
-device ide-cd,bus=ide.1,id=sata0-0-1,bootindex=1 \
-netdev tap,fd=33,id=hostnet0 \
-device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:41:4b:13,bus=pci.1,addr=0x0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-device ich9-intel-hda,id=sound0,bus=pcie.0,addr=0x1b \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-chardev spicevmc,id=charredir0,name=usbredir \
-device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 \
-chardev spicevmc,id=charredir1,name=usbredir \
-device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 \
-device usb-host,hostdevice=/dev/bus/usb/001/002,id=hostdev0,bus=usb.0,port=4 \
-device usb-host,hostdevice=/dev/bus/usb/001/003,id=hostdev1,bus=usb.0,port=5 \
-device usb-host,hostdevice=/dev/bus/usb/001/004,id=hostdev2,bus=usb.0,port=6 \
-device vfio-pci,host=0000:03:00.0,id=hostdev3,bus=pci.5,addr=0x0 \
-device vfio-pci,host=0000:03:00.1,id=hostdev4,bus=pci.6,addr=0x0 \
-device virtio-balloon-pci,id=balloon0,bus=pci.4,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2021-03-20T03:08:23.688625Z qemu-system-x86_64: vfio: Cannot reset device 0000:03:00.1, no available reset mechanism.
2021-03-20T03:08:23.693726Z qemu-system-x86_64: vfio: Cannot reset device 0000:03:00.1, no available reset mechanism.
2021-03-20T03:21:49.465639Z qemu-system-x86_64: vfio: Cannot reset device 0000:03:00.1, no available reset mechanism.
2021-03-20T03:21:49.469733Z qemu-system-x86_64: vfio: Cannot reset device 0000:03:00.1, no available reset mechanism.
2021-03-20T03:23:46.992714Z qemu-system-x86_64: terminating on signal 15 from pid 2693 (/usr/bin/libvirtd)
2021-03-20 03:23:49.629+0000: shutting down, reason=shutdown
2021-03-20 03:26:29.532+0000: starting up libvirt version: 7.0.0, qemu version: 5.2.0, kernel: 5.11.7-zen1-1-zen, hostname: archyboi.localdomain
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-1-win10 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-win10/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-win10/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-win10/.config \
QEMU_AUDIO_DRV=none \
/usr/bin/qemu-system-x86_64 \
-name guest=win10,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-win10/master-key.aes \
-blockdev '{"driver":"file","filename":"/usr/share/edk2-ovmf/x64/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/win10_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-5.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \
-cpu Cooperlake,ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,mpx=on,umip=on,md-clear=on,xsaves=on,ibpb=on,amd-stibp=on,amd-ssbd=on,hle=off,rtm=off,avx512f=off,avx512dq=off,clwb=off,avx512cd=off,avx512bw=off,avx512vl=off,avx512vnni=off,avx512-bf16=off,taa-no=off,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vendor-id=123456789ab,kvm=off \
-m 32000 \
-object memory-backend-ram,id=pc.ram,size=33554432000 \
-overcommit mem-lock=off \
-smp 10,sockets=1,dies=1,cores=5,threads=2 \
-uuid e9c1e261-069a-4a98-b6de-8c2b4ca79c12 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=31,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=0x8,chassis=6,id=pci.6,bus=pcie.0,addr=0x1 \
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 \
-blockdev '{"driver":"file","filename":"/mnt/1TB/KVM-VM/win10.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null}' \
-device ide-hd,bus=ide.0,drive=libvirt-2-format,id=sata0-0-0,bootindex=2 \
-device ide-cd,bus=ide.1,id=sata0-0-1,bootindex=1 \
-netdev tap,fd=33,id=hostnet0 \
-device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:41:4b:13,bus=pci.1,addr=0x0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-device ich9-intel-hda,id=sound0,bus=pcie.0,addr=0x1b \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-chardev spicevmc,id=charredir0,name=usbredir \
-device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 \
-chardev spicevmc,id=charredir1,name=usbredir \
-device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 \
-device usb-host,hostdevice=/dev/bus/usb/001/002,id=hostdev0,bus=usb.0,port=4 \
-device usb-host,hostdevice=/dev/bus/usb/001/003,id=hostdev1,bus=usb.0,port=5 \
-device usb-host,hostdevice=/dev/bus/usb/001/004,id=hostdev2,bus=usb.0,port=6 \
-device vfio-pci,host=0000:03:00.0,id=hostdev3,bus=pci.5,addr=0x0 \
-device vfio-pci,host=0000:03:00.1,id=hostdev4,bus=pci.6,addr=0x0 \
-device virtio-balloon-pci,id=balloon0,bus=pci.4,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2021-03-20T03:26:33.485771Z qemu-system-x86_64: vfio: Cannot reset device 0000:03:00.1, no available reset mechanism.
2021-03-20T03:26:33.490753Z qemu-system-x86_64: vfio: Cannot reset device 0000:03:00.1, no available reset mechanism.
2021-03-20T03:28:04.092756Z qemu-system-x86_64: terminating on signal 15 from pid 431 (/usr/bin/libvirtd)
2021-03-20 03:28:06.715+0000: shutting down, reason=shutdown

Any assistance would be appreciated, as this is my first time experimenting with VFIO!

Thanks. Apologies for a long post.

sp33dy905 · March 20, 2021, 3:51pm

Update 1:

Here’s what I’ve also just tried with no success so far:

Adding a few more kernel parameters

iommu=pt
iommu=1
video=efifb:off

Dumping the GPU ROM and loading it on VM initialization

Do I need to do the ACS patch? I looked at my IOMMU grouping and they seem to be valid

Kosutka · March 21, 2021, 4:48pm

I have the same issue as you. Seems like the GPU doesn’t reset correctly in the first place. I see some strange pink artifacts during VM boot and then have the error 43.
My specs are:
Ryzen 3900x
Gigabyte X570 Aorus Master
Gigabyte RX5700 XT
64GB of RAM
Arch Linux on 5.11.8 xanmod-cacule kernel

Kosutka · March 22, 2021, 2:16pm

Got it working! Added this to my hooks after unloading the gpu: rtcwake -m mem -s 5, which puts the computer to sleep.

sp33dy905 · March 22, 2021, 8:53pm

Got my solution partially working, I can shut down now and return to host, and I don’t have the pink artifacts anymore when I boot. The Dreaded Code 43 is the one that’s stopping me. So close, yet so far.

Here’s what I’ve tried so far:

Added the rtcwake command (helped in boot and shutdown of vm)
Tried the ACS patch (didn’t work)
Enabled Hyper-V and installed it (didn’t work)
Installed Windows updates and GPU Driver (Did install (Beta version), but refuse to launch)
Tried all the VBIOS ROMS (TechPower Dump, GPU-Z, Linux Dump) that I could use, no luck

I’m honestly at a shamble here. It’s soo close to working, and yet that’s just stopping me.

coldfire7 · March 22, 2021, 9:39pm

I don’t know about AMD GPU but this is what worked for me with a NVIDIA GPU:
gist[dot]github[dot]com/coldfire7/c0759efb190f8d786e449ccd7896ec9c

Kosutka · March 23, 2021, 5:21pm

Can you show me your start hook script?

sp33dy905 · March 23, 2021, 6:30pm

Here’s an updated version of it, with changes applied so far:

#!/bin/bash
# Helpful to read output when debugging
set -x
 
source "/etc/libvirt/hooks/kvm.conf"
 
# Stop display manager
systemctl stop sddm.service
 
# Unbind VTconsoles
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind
 
# Unbind EFI-Framebuffer
#echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
 
# Avoid a Race condition by waiting 2 seconds. This can be calibrated to be shorter or longer if required for your system
sleep 5
 
# Unload all Radeon drivers
modprobe -r gpu_sched
modprobe -r ttm
modprobe -r drm_kms_helper
modprobe -r i2c_algo_bit
modprobe -r drm
modprobe -r snd_hda_intel
modprobe -r amdgpu
 
# Unbind the GPU from display driver
virsh nodedev-detach $VIRSH_GPU_VIDEO
virsh nodedev-detach $VIRSH_GPU_AUDIO

rtcwake -m mem -s 5
 
# Load VFIO Kernel Module  
modprobe vfio
modprobe vfio_pci  
modprobe vfio_iommu_type1

Kosutka · March 23, 2021, 7:42pm

Looks almost same as mine. Have you tried adding ‘<vendor_id state=“on” value=“xxx”/>’ in the hyperv section of your xml?

sp33dy905 · March 23, 2021, 9:11pm

Yep, Done every xml trick (that I could find) to it:

<domain type='kvm'>
  <name>win10</name>
  <uuid>105dcbce-2be4-4874-9b5d-ba5cfd553c73</uuid>
  <metadata/>
  <memory unit='KiB'>32768000</memory>
  <currentMemory unit='KiB'>32768000</currentMemory>
  <vcpu placement='static'>10</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-5.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vpindex state='on'/>
      <synic state='on'/>
      <stimer state='on'/>
      <reset state='on'/>
      <vendor_id state='on' value='6jIdXGrJdG2O'/>
      <frequencies state='on'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
    <ioapic driver='kvm'/>
  </features>
  <cpu mode='host-passthrough' check='partial' migratable='on'>
    <topology sockets='1' dies='1' cores='5' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='disable' name='hypervisor'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/mnt/1TB/KVM-VM/win10.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <boot order='2'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='9' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:e5:d3:3d'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc08b'/>
      </source>
      <address type='usb' bus='0' port='4'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x2516'/>
        <product id='0x0014'/>
      </source>
      <address type='usb' bus='0' port='5'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0x0ab7'/>
      </source>
      <address type='usb' bus='0' port='6'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <rom file='/usr/share/libvirt/Navi-10.rom'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <rom file='/usr/share/libvirt/Navi-10.rom'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
</domain>

rainmanlv1 · March 6, 2022, 7:43pm

Have you found solution to it yet, im working with fedora server here same issue but I have no need for it to revert back to linux desktop, i got code 43 in guest(vm) on boot each time on my HD7970 but the sound part works fine only the video part, anyone has solution to this? @wendell or anyone ? Please save us!!!