I am using Qemu to virtualise Windows 10 with GPU passthrough from Arch Linux. My secondary is an RX580 my CPU is an i9-10980XE on the X299 platform. When booting the system the RX’s fans ramp up (kind of like they do when you power a computer on), but the display shows nothing. I am using displayport to connect to my monitor.
This find /sys/kernel/iommu_groups/ -type l command returns:
/sys/kernel/iommu_groups/0/devices/0000:40:00.0
/sys/kernel/iommu_groups/57/devices/0000:40:0a.4
/sys/kernel/iommu_groups/19/devices/0000:00:14.2
/sys/kernel/iommu_groups/19/devices/0000:00:14.0
/sys/kernel/iommu_groups/47/devices/0000:20:1e.5
/sys/kernel/iommu_groups/47/devices/0000:20:1e.3
/sys/kernel/iommu_groups/47/devices/0000:20:1e.1
/sys/kernel/iommu_groups/47/devices/0000:20:1e.6
/sys/kernel/iommu_groups/47/devices/0000:20:1e.4
/sys/kernel/iommu_groups/47/devices/0000:20:1e.2
/sys/kernel/iommu_groups/47/devices/0000:20:1e.0
/sys/kernel/iommu_groups/75/devices/0000:40:0d.2
/sys/kernel/iommu_groups/37/devices/0000:20:05.0
/sys/kernel/iommu_groups/9/devices/0000:00:04.4
/sys/kernel/iommu_groups/65/devices/0000:40:0c.0
/sys/kernel/iommu_groups/27/devices/0000:00:1c.4
The GPU falls into the 21 IOMMU group according to this (the GPU is also in one IOMMU group only, and there is no other hardware in the group (aside from the sound output device)):
IOMMU Group 3:
21:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7)
21:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
lsmod | grep vfio returns:
vfio_pci 16384 0
vfio_pci_core 98304 1 vfio_pci
vfio_iommu_type1 49152 0
vfio 77824 4 vfio_pci_core,vfio_iommu_type1,vfio_pci
iommufd 110592 1 vfio
lspci -nn | grep -i vga returns:
21:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7)
41:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] [10de:1b06] (rev a1)
dmesg | grep -i vfio returns:
[ 3.077607] VFIO - User Level meta-driver version: 0.3
[ 3.103592] vfio-pci 0000:21:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
[ 3.103852] vfio_pci: add [1002:67df[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.151233] vfio_pci: add [1002:aaf0[ffffffff:ffffffff]] class 0x000000/00000000
[ 2962.118778] vfio-pci 0000:21:00.0: enabling device (0100 -> 0103)
I am also going to include my VM logs:
2024-08-27 16:13:20.322+0000: starting up libvirt version: 10.6.0, qemu version: 9.0.2, kernel: 6.10.6-arch1-1, hostname: H-E-R-A-N
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin \
USER=root \
HOME=/var/lib/libvirt/qemu/domain-12-win10 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-12-win10/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-12-win10/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-12-win10/.config \
/usr/bin/qemu-system-x86_64 \
-name guest=win10,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-12-win10/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/edk2/x64/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/win10_VARS.fd","node-name":"libvirt-pflash1-storage","read-only":false}' \
-machine pc-q35-9.0,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-storage,hpet=off,acpi=on \
-accel kvm \
-cpu host,migratable=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff \
-m size=8388608k \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":8589934592}' \
-overcommit mem-lock=off \
-smp 8,sockets=8,cores=1,threads=1 \
-uuid dfa1146c-ed8b-4d6e-8ca7-867a6c22d8a2 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=30,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":8,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x1"}' \
-device '{"driver":"pcie-root-port","port":9,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x1.0x1"}' \
-device '{"driver":"pcie-root-port","port":10,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x1.0x2"}' \
-device '{"driver":"pcie-root-port","port":11,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x1.0x3"}' \
-device '{"driver":"pcie-root-port","port":12,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x1.0x4"}' \
-device '{"driver":"pcie-root-port","port":13,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x1.0x5"}' \
-device '{"driver":"pcie-root-port","port":14,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x1.0x6"}' \
-device '{"driver":"pcie-root-port","port":15,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x1.0x7"}' \
-device '{"driver":"pcie-root-port","port":16,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":12,"id":"pci.12","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"pcie-root-port","port":20,"chassis":13,"id":"pci.13","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"pcie-root-port","port":21,"chassis":14,"id":"pci.14","bus":"pcie.0","addr":"0x2.0x5"}' \
-device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.2","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/mnt/BA6029B160297573/KVMs/win10.qcow2","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":false,"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.3","addr":"0x0","drive":"libvirt-3-format","id":"virtio-disk0","bootindex":1}' \
-blockdev '{"driver":"file","filename":"/mnt/BA6029B160297573/Downloads/Win10_22H2_EnglishInternational_x64.iso","node-name":"libvirt-2-storage","read-only":true}' \
-device '{"driver":"ide-cd","bus":"ide.1","drive":"libvirt-2-storage","id":"sata0-0-1"}' \
-blockdev '{"driver":"file","filename":"/mnt/BA6029B160297573/Downloads/virtio-win-0.1.262.iso","node-name":"libvirt-1-storage","read-only":true}' \
-device '{"driver":"ide-cd","bus":"ide.2","drive":"libvirt-1-storage","id":"sata0-0-2"}' \
-netdev '{"type":"tap","fd":"31","id":"hostnet0"}' \
-device '{"driver":"e1000e","netdev":"hostnet0","id":"net0","mac":"52:54:00:bc:7e:dc","bus":"pci.1","addr":"0x0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"2"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-global ICH9-LPC.noreboot=off \
-watchdog-action reset \
-device '{"driver":"vfio-pci","host":"0000:21:00.0","id":"hostdev0","bus":"pci.4","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:21:00.1","id":"hostdev1","bus":"pci.5","addr":"0x0"}' \
-device '{"driver":"usb-host","hostdevice":"/dev/bus/usb/001/004","id":"hostdev2","bus":"usb.0","port":"1"}' \
-device '{"driver":"usb-host","hostdevice":"/dev/bus/usb/001/005","id":"hostdev3","bus":"usb.0","port":"3"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.6","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
I have not found anything immediately wrong in these, so I am not able to diagnose this properly. I am also new to the KVM stuff, so I would love to see your solutions.
If you have any further questions about the configuration please ask me.
Thanks a lot for reading.
PS:
One user there recommends blacklisting the GPU drivers, but that is not a solution for me because I will likely swap to an AMD card soon. I cannot have a temporary solution here, I need to blacklist it only.
I suspect the issue is because how I load the drivers in (the order).
Here is how I load the modules in through mkinitcpio.conf:
MODULES=(nvidia nvidia_modeset nvidia_uvm nvidia_drm vfio_pci vfio vfio_iommu_type1)
And here is my systemdboot configuration:
title Arch
linux /vmlinuz-linux
initrd /initramfs-linux.img
options root=PARTUUID=e4bccff7-9cca-499b-94a0-da275bed16c4 rw nvidia-drm.modeset=1 module_blacklist=hid_logitech_hidpp quiet splash intel_iommu=on
The module_blacklist=hid_logitech_hidpp part is to block the logitech driver from loading in (check my previous posts for more info on that).
I also tried vendor-reset, but it had errors while installing. That ultimately killed my Arch system (it did not recognise ext4, lol) (which I fixed in 15 minutes). If there is documentation I should follow, please hand it over.