Hello everyone!
This is my first time using GPU passthrough, I had a bit of trouble setting it up but got it working perfectly just a few days ago, or so I thought.
- Gigabyte B450 AORUS M
- AMD Ryzen 5 2600 Six-Core Processor
- ADATA SX8100NP 512GB M.2 2280 NVMe
- Host: Linux 5.11.14-1-MANJARO Ornara 21.0.2
- GALAX GeForce GTX 1050 Ti EXOC 4GB GDDR5 128Bit
- Guest: Windows 10 Pro x64 20H2 April
- ASUS Geforce RTX 3060 Ti OC TUF Gaming 8GB GDDR6 256bit
I’m using the 3060 for the win10 VM, everything was working fine through multiple startups, driver update, gaming. After an otherwise normal shutdown (not sure if related but noticed a windows update notification, which I later uninstalled attempting to fix the problem, no luck), I haven’t been able to get it working again. The GPU is detected normally in virtmanager but at the VM startup it fails to start, no output or fan spin. If I use Spice Graphics instead I can see win10 boots up just fine, no GPU detected though. Any thoughts?
Diagnostics
$ lspci -nnk
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
Kernel driver in use: pcieport
00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
Kernel driver in use: pcieport
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
Kernel driver in use: pcieport
00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
Kernel driver in use: pcieport
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
Kernel driver in use: pcieport
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
Kernel driver in use: pcieport
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
Subsystem: Gigabyte Technology Co., Ltd Device [1458:5001]
Kernel modules: i2c_piix4, sp5100_tco
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
Subsystem: Gigabyte Technology Co., Ltd Device [1458:5001]
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
Kernel driver in use: k10temp
Kernel modules: k10temp
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
01:00.0 Non-Volatile memory controller [0108]: Realtek Semiconductor Co., Ltd. RTS5763DL NVMe SSD Controller [10ec:5762] (rev 01)
Subsystem: Realtek Semiconductor Co., Ltd. RTS5763DL NVMe SSD Controller [10ec:5762]
Kernel driver in use: nvme
02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller [1022:43d5] (rev 01)
Subsystem: ASMedia Technology Inc. Device [1b21:1142]
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
Subsystem: ASMedia Technology Inc. Device [1b21:1062]
Kernel driver in use: ahci
02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
Kernel driver in use: pcieport
03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
Kernel driver in use: pcieport
03:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
Kernel driver in use: pcieport
03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
Kernel driver in use: pcieport
04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 16)
DeviceName: Broadcom 5762
Subsystem: Gigabyte Technology Co., Ltd Onboard Ethernet [1458:e000]
Kernel driver in use: r8169
Kernel modules: r8169
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] [10de:1c82] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:11bf]
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia_drm, nvidia
06:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:11bf]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104 [GeForce RTX 3060 Ti] [10de:2486] (rev a1)
Subsystem: ASUSTeK Computer Inc. Device [1043:87c6]
Kernel driver in use: vfio-pci
Kernel modules: nouveau, nvidia_drm, nvidia
07:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:228b] (rev a1)
Subsystem: ASUSTeK Computer Inc. Device [1043:87c6]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
09:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
Kernel driver in use: ccp
Kernel modules: ccp
09:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 Host controller [1022:145f]
Subsystem: Gigabyte Technology Co., Ltd Device [1458:5007]
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
0a:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
Subsystem: Gigabyte Technology Co., Ltd Device [1458:b002]
Kernel driver in use: ahci
0a:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller [1022:1457]
Subsystem: Gigabyte Technology Co., Ltd Device [1458:a182]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
VM xml
<domain type='kvm'>
<name>win10</name>
<uuid>51062533-bfc9-4dc6-8d97-ef9ae3261783</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/10"/>
</libosinfo:libosinfo>
</metadata>
<memory unit='KiB'>12582912</memory>
<currentMemory unit='KiB'>12582912</currentMemory>
<vcpu placement='static'>8</vcpu>
<os>
<type arch='x86_64' machine='pc-q35-5.2'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
<bootmenu enable='yes'/>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
<vendor_id state='on' value='randomid'/>
</hyperv>
<kvm>
<hidden state='on'/>
</kvm>
<vmport state='off'/>
</features>
<cpu mode='host-passthrough' check='partial' migratable='on'>
<topology sockets='1' dies='1' cores='4' threads='2'/>
</cpu>
<clock offset='localtime'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/win10.qcow2'/>
<target dev='sda' bus='sata'/>
<boot order='2'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</controller>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='1' port='0x8'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='2' port='0x9'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='3' port='0xa'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='4' port='0xb'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='5' port='0xc'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
</controller>
<controller type='pci' index='6' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='6' port='0xd'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
</controller>
<controller type='pci' index='7' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='7' port='0xe'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
</controller>
<controller type='pci' index='8' model='pcie-to-pci-bridge'>
<model name='pcie-pci-bridge'/>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
</controller>
<interface type='network'>
<mac address='52:54:00:02:70:30'/>
<source network='default'/>
<model type='e1000e'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<hostdev mode='subsystem' type='usb' managed='yes'>
<source>
<vendor id='0x0d8c'/>
<product id='0x0171'/>
</source>
<address type='usb' bus='0' port='3'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</hostdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</memballoon>
</devices>
</domain>
QEMU Log
2021-04-27 03:41:21.985+0000: starting up libvirt version: 7.1.0, qemu version: 5.2.0, kernel: 5.11.14-1-MANJARO, hostname: thejournalist-b450aorusm
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/var/lib/snapd/snap/bin \
HOME=/var/lib/libvirt/qemu/domain-1-win10 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-win10/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-win10/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-win10/.config \
QEMU_AUDIO_DRV=none \
/usr/bin/qemu-system-x86_64 \
-name guest=win10,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-win10/master-key.aes \
-blockdev '{"driver":"file","filename":"/usr/share/edk2-ovmf/x64/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/win10_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-5.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \
-cpu host,migratable=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vendor-id=randomid,kvm=off \
-m 12288 \
-object memory-backend-ram,id=pc.ram,size=12884901888 \
-overcommit mem-lock=off \
-smp 8,sockets=1,dies=1,cores=4,threads=2 \
-uuid 51062533-bfc9-4dc6-8d97-ef9ae3261783 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=32,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot menu=on,strict=on \
-device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \
-device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \
-device pcie-root-port,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 \
-device pcie-root-port,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x3 \
-device pcie-root-port,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x1.0x4 \
-device pcie-root-port,port=0xd,chassis=6,id=pci.6,bus=pcie.0,addr=0x1.0x5 \
-device pcie-root-port,port=0xe,chassis=7,id=pci.7,bus=pcie.0,addr=0x1.0x6 \
-device pcie-pci-bridge,id=pci.8,bus=pci.6,addr=0x0 \
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/win10.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device ide-hd,bus=ide.0,drive=libvirt-1-format,id=sata0-0-0,bootindex=2 \
-netdev tap,fd=34,id=hostnet0 \
-device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:02:70:30,bus=pci.1,addr=0x0 \
-device usb-host,hostdevice=/dev/bus/usb/003/002,id=hostdev0,bus=usb.0,port=3 \
-device vfio-pci,host=0000:07:00.0,id=hostdev1,bus=pci.3,addr=0x0 \
-device vfio-pci,host=0000:07:00.1,id=hostdev2,bus=pci.4,addr=0x0 \
-device virtio-balloon-pci,id=balloon0,bus=pci.5,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2021-04-27T03:41:22.170148Z qemu-system-x86_64: warning: This family of AMD CPU doesn't support hyperthreading(2)
Please configure -smp options properly or try enabling topoext feature.
2021-04-27T03:41:25.453517Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
2021-04-27T03:41:25.466836Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
2021-04-27T03:42:37.032037Z qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:07:00.0
Device option ROM contents are probably invalid (check dmesg).
Skip option ROM probe with rombar=0, or load from file with romfile=
2021-04-27T03:44:20.242524Z qemu-system-x86_64: terminating on signal 15 from pid 11707 (/usr/bin/libvirtd)
2021-04-27 03:45:06.350+0000: shutting down, reason=shutdown
2021-04-27 03:45:36.536+0000: shutting down, reason=failed
$ dmesg | grep -i vfio
(after failed startup)
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.11-x86_64 root=UUID=c35ba447-0aaf-4d75-8bd7-dfc406374195 rw quiet amd_iommu=on rd.driver.pre=vfio-pci kvm.ignore_msrs=1 cryptdevice=UUID=68f7c2b2-f971-406e-a046-1efe50988589:luks-68f7c2b2-f971-406e-a046-1efe50988589 root=/dev/mapper/luks-68f7c2b2-f971-406e-a046-1efe50988589 apparmor=1 security=apparmor resume=/dev/mapper/luks-9d65bdc0-7c91-42eb-a553-44531e694e3a udev.log_priority=3
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.11-x86_64 root=UUID=c35ba447-0aaf-4d75-8bd7-dfc406374195 rw quiet amd_iommu=on rd.driver.pre=vfio-pci kvm.ignore_msrs=1 cryptdevice=UUID=68f7c2b2-f971-406e-a046-1efe50988589:luks-68f7c2b2-f971-406e-a046-1efe50988589 root=/dev/mapper/luks-68f7c2b2-f971-406e-a046-1efe50988589 apparmor=1 security=apparmor resume=/dev/mapper/luks-9d65bdc0-7c91-42eb-a553-44531e694e3a udev.log_priority=3
[ 1.896779] VFIO - User Level meta-driver version: 0.3
[ 1.903808] vfio_pci: unknown parameter 'allow_unsafe_interrupts' ignored
[ 1.903901] vfio-pci 0000:07:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 62.983351] vfio-pci 0000:07:00.0: enabling device (0000 -> 0003)
[ 63.093513] vfio-pci 0000:07:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258
[ 63.093536] vfio-pci 0000:07:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[ 63.093544] vfio-pci 0000:07:00.0: vfio_ecap_init: hiding ecap 0x26@0xc1c
[ 63.093546] vfio-pci 0000:07:00.0: vfio_ecap_init: hiding ecap 0x27@0xd00
[ 63.093548] vfio-pci 0000:07:00.0: vfio_ecap_init: hiding ecap 0x25@0xe00
[ 63.116674] vfio-pci 0000:07:00.1: enabling device (0000 -> 0002)
[ 63.116812] vfio-pci 0000:07:00.1: vfio_ecap_init: hiding ecap 0x25@0x160
[ 64.326737] vfio-pci 0000:07:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 64.353440] vfio-pci 0000:07:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 65.096858] vfio-pci 0000:07:00.0: timed out waiting for pending transaction; performing function level reset anyway
[ 66.323362] vfio-pci 0000:07:00.0: not ready 1023ms after FLR; waiting
[ 67.363373] vfio-pci 0000:07:00.0: not ready 2047ms after FLR; waiting
[ 69.576775] vfio-pci 0000:07:00.0: not ready 4095ms after FLR; waiting
[ 73.846680] vfio-pci 0000:07:00.0: not ready 8191ms after FLR; waiting
[ 82.164097] vfio-pci 0000:07:00.0: not ready 16383ms after FLR; waiting
[ 98.591597] vfio-pci 0000:07:00.0: not ready 32767ms after FLR; waiting
[ 132.725255] vfio-pci 0000:07:00.0: not ready 65535ms after FLR; giving up
[ 135.296238] vfio-pci 0000:07:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 135.302310] vfio-pci 0000:07:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 135.326585] vfio-pci 0000:07:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 135.329261] vfio-pci 0000:07:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 135.345676] vfio-pci 0000:07:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 135.347582] vfio-pci 0000:07:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 135.372502] vfio-pci 0000:07:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 135.383839] vfio-pci 0000:07:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff
[ 135.386194] vfio-pci 0000:07:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 135.394336] vfio-pci 0000:07:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 135.394632] vfio-pci 0000:07:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 135.394653] vfio-pci 0000:07:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 135.394819] vfio-pci 0000:07:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 135.395741] vfio-pci 0000:07:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 135.395921] vfio-pci 0000:07:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 135.395942] vfio-pci 0000:07:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 135.396094] vfio-pci 0000:07:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 135.430585] vfio-pci 0000:07:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 135.431024] vfio-pci 0000:07:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 135.445289] vfio-pci 0000:07:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 135.445598] vfio-pci 0000:07:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 135.446520] vfio-pci 0000:07:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 135.446715] vfio-pci 0000:07:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 135.458912] vfio-pci 0000:07:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 135.459759] vfio-pci 0000:07:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 447.922500] vfio-pci 0000:07:00.1: can't change power state from D0 to D3hot (config space inaccessible)
[ 448.664261] vfio-pci 0000:07:00.0: timed out waiting for pending transaction; performing function level reset anyway
[ 449.890875] vfio-pci 0000:07:00.0: not ready 1023ms after FLR; waiting
[ 450.930892] vfio-pci 0000:07:00.0: not ready 2047ms after FLR; waiting
[ 453.144154] vfio-pci 0000:07:00.0: not ready 4095ms after FLR; waiting
[ 457.410766] vfio-pci 0000:07:00.0: not ready 8191ms after FLR; waiting
[ 465.730780] vfio-pci 0000:07:00.0: not ready 16383ms after FLR; waiting
[ 482.583753] vfio-pci 0000:07:00.0: not ready 32767ms after FLR; waiting
[ 516.716938] vfio-pci 0000:07:00.0: not ready 65535ms after FLR; giving up
[ 517.812063] vfio-pci 0000:07:00.1: can't change power state from D0 to D3hot (config space inaccessible)
[ 517.812073] vfio-pci 0000:07:00.0: can't change power state from D0 to D3hot (config space inaccessible)
[ 661.103793] vfio-pci 0000:07:00.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=io+mem:owns=none
$ virsh start win10
(attempt to restart VM)
error: Failed to start domain 'win10'
error: internal error: Unknown PCI header type '127' for device '0000:07:00.0'