Hello all, wondering if anyone has a similar setup. Having issues with my VFIO setup when using the LVL1 KVM hardware switch.
Hardware Overview
- Primary Monitor - Asus Swift PG348Q 34" 3440 x 1440, 100 Hz - Gsync
- Second Monitor - LG 32UD60-B 4K 60Hz
- GTX 1080: For Pop-OS 20.04 host
- GTX 1080TI: For Win10 VM(passthrough)
- Dedicated nvme for Win10 VM
- Asus X99 Deluxe II
- i7-6950X
- LVL1 KVM Switch DP 1.2
Want to state, I do have a stable\reliable VFIO setup when NOT using the KVM switch. My typical routine… Physically unplug primary monitor from GTX 1080 and plug it into the GTX 1080Ti, boot VM. Works every time, no issues.
However, when I use the KVM switch to toggle my primary monitor over to the 1080TI. I can’t get the VM to boot up and I have to reboot my entire computer because the VM is hung up and can’t force it off. I don’t have anything else plugged into the KVM switch. Just the primary monitor and both nvidia cards.
One thing to note, after the reboot of my host. I can get my VM to start, pretty much have to reboot my host every time I want to start the VM. Even while keeping the KVM switched to the 1080TI. I don’t have to do this when I physically move the DP cable.
When I run lspci -vnn while the VM is running, it takes a moment but eventually it will list my devices and my 1080TI does indeed show Kernel Driver in use: vfio-pci
Anyone have any idea what could be going on? Below are my configs, not sure if it’s needed…
I used this guide when setting everything up, https://github.com/bryansteiner/gpu-passthrough-tutorial
Along with Wendell’s section about using similar GPU
lspci -vnn 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80] (rev a1) (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. GP104 [GeForce GTX 1080] [1043:8592] Physical Slot: 4 Flags: bus master, fast devsel, latency 0, IRQ 78, NUMA node 0 Memory at fa000000 (32-bit, non-prefetchable) [size=16M] Memory at c0000000 (64-bit, prefetchable) [size=256M] Memory at d0000000 (64-bit, prefetchable) [size=32M] I/O ports at e000 [size=128] Expansion ROM at 000c0000 [virtual] [disabled] [size=128K] Capabilities: <access denied> Kernel driver in use: nvidia Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia 01:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1) Subsystem: ASUSTeK Computer Inc. GP104 High Definition Audio Controller [1043:8592] Physical Slot: 4 Flags: bus master, fast devsel, latency 0, IRQ 77, NUMA node 0 Memory at fb080000 (32-bit, non-prefetchable) [size=16K] Capabilities: <access denied> Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] [10de:1b06] (rev a1) (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. GP102 [GeForce GTX 1080 Ti] [1043:85e4] Physical Slot: 6 Flags: bus master, fast devsel, latency 0, IRQ 27, NUMA node 0 Memory at f8000000 (32-bit, non-prefetchable) [size=16M] Memory at a0000000 (64-bit, prefetchable) [size=256M] Memory at b0000000 (64-bit, prefetchable) [size=32M] I/O ports at d000 [size=128] Expansion ROM at f9000000 [disabled] [size=512K] Capabilities: <access denied> Kernel driver in use: nvidia Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia 02:00.1 Audio device [0403]: NVIDIA Corporation GP102 HDMI Audio Controller [10de:10ef] (rev a1) Subsystem: ASUSTeK Computer Inc. GP102 HDMI Audio Controller [1043:85e4] Physical Slot: 6 Flags: bus master, fast devsel, latency 0, IRQ 41, NUMA node 0 Memory at f9080000 (32-bit, non-prefetchable) [size=16K] Capabilities: <access denied> Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel 03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804] (prog-if 02 [NVM Express]) Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a801] Physical Slot: 6-1 Flags: bus master, fast devsel, latency 0, IRQ 60, NUMA node 0 Memory at fb400000 (64-bit, non-prefetchable) [size=16K] Capabilities: <access denied> Kernel driver in use: nvme Kernel modules: nvme
sudo cat /boot/efi/loader/entries/Pop_os-current.conf title Pop!_OS linux /EFI/Pop_OS-c531160a-11f6-40d3-a7ec-20534a9c27ad/vmlinuz.efi initrd /EFI/Pop_OS-c531160a-11f6-40d3-a7ec-20534a9c27ad/initrd.img options root=UUID=c531160a-11f6-40d3-a7ec-20534a9c27ad ro quiet loglevel=0 systemd.show_status=false splash intel_iommu=on usbcore.autosuspend=-1
/etc/initramfs-tools/scripts/init-top/bind_vfio.sh #!/bin/sh PREREQS="" DEVS="0000:02:00.0 0000:02:00.1" for DEV in $DEVS; do echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override done modprobe -i vfio-pci
/etc/initramfs-tools/modules # List of modules that you want to include in your initramfs. # They will be loaded at boot time in the order below. # # Syntax: module_name [args ...] # # You must run update-initramfs(8) to effect this change. # # Examples: # # raid1 # sd_mod vfio-pci
cat /etc/libvirt/hooks/kvm.conf ## Virsh devices VIRSH_GPU_VIDEO=pci_0000_02_00_0 VIRSH_GPU_AUDIO=pci_0000_02_00_1 VIRSH_NVME_SSD=pci_0000_03_00_0
cat /etc/libvirt/hooks/qemu.d/vmwin01/prepare/begin/bind_vfio.sh #!/bin/bash ## Load the config file source "/etc/libvirt/hooks/kvm.conf" ## Load vfio modprobe vfio modprobe vfio_iommu_type1 modprobe vfio_pci ## Unbind gpu from nvidia and bind to vfio virsh nodedev-detach $VIRSH_GPU_VIDEO virsh nodedev-detach $VIRSH_GPU_AUDIO ## Unbind ssd from nvme and bind to vfio virsh nodedev-detach $VIRSH_NVME_SSD
cat /etc/libvirt/hooks/qemu.d/vmwin01/release/end/unbind_vfio.sh #!/bin/bash ## Load the config file source "/etc/libvirt/hooks/kvm.conf" ## Unbind gpu from vfio and bind to nvidia virsh nodedev-reattach $VIRSH_GPU_VIDEO virsh nodedev-reattach $VIRSH_GPU_AUDIO ## Unbind ssd from vfio and bind to nvme virsh nodedev-reattach $VIRSH_NVME_SSD ## Unload vfio modprobe -r vfio_pci modprobe -r vfio_iommu_type1 modprobe -r vfio
Virt Manager XML <domain type="kvm"> <name>win10</name> <uuid>43ee43fb-a0b6-46bb-987b-5212077d7605</uuid> <metadata> <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0"> <libosinfo:os id="http://microsoft.com/win/10"/> </libosinfo:libosinfo> </metadata> <memory unit="KiB">16777216</memory> <currentMemory unit="KiB">16777216</currentMemory> <memoryBacking> <hugepages/> </memoryBacking> <vcpu placement="static">10</vcpu> <iothreads>4</iothreads> <cputune> <vcpupin vcpu="0" cpuset="5"/> <vcpupin vcpu="1" cpuset="15"/> <vcpupin vcpu="2" cpuset="6"/> <vcpupin vcpu="3" cpuset="16"/> <vcpupin vcpu="4" cpuset="7"/> <vcpupin vcpu="5" cpuset="17"/> <vcpupin vcpu="6" cpuset="8"/> <vcpupin vcpu="7" cpuset="18"/> <vcpupin vcpu="8" cpuset="9"/> <vcpupin vcpu="9" cpuset="19"/> <emulatorpin cpuset="0-10"/> <iothreadpin iothread="1" cpuset="1-11"/> <iothreadpin iothread="2" cpuset="2-12"/> <iothreadpin iothread="3" cpuset="3-13"/> <iothreadpin iothread="4" cpuset="4-14"/> </cputune> <os> <type arch="x86_64" machine="pc-q35-4.0">hvm</type> <loader readonly="yes" type="pflash">/usr/share/OVMF/OVMF_CODE.fd</loader> <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram> <bootmenu enable="no"/> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state="on"/> <vapic state="on"/> <spinlocks state="on" retries="8191"/> <vendor_id state="on" value="kvm hyperv"/> </hyperv> <kvm> <hidden state="on"/> </kvm> <vmport state="off"/> <ioapic driver="kvm"/> </features> <cpu mode="host-passthrough" check="partial"> <topology sockets="1" cores="5" threads="2"/> </cpu> <clock offset="localtime"> <timer name="rtc" tickpolicy="catchup"/> <timer name="pit" tickpolicy="delay"/> <timer name="hpet" present="no"/> <timer name="hypervclock" present="yes"/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled="no"/> <suspend-to-disk enabled="no"/> </pm> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <controller type="usb" index="0" model="qemu-xhci" ports="15"> <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/> </controller> <controller type="sata" index="0"> <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/> </controller> <controller type="pci" index="0" model="pcie-root"/> <controller type="pci" index="1" model="pcie-root-port"> <model name="pcie-root-port"/> <target chassis="1" port="0x8"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0" multifunction="on"/> </controller> <controller type="pci" index="2" model="pcie-root-port"> <model name="pcie-root-port"/> <target chassis="2" port="0x9"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/> </controller> <controller type="pci" index="3" model="pcie-root-port"> <model name="pcie-root-port"/> <target chassis="3" port="0xa"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/> </controller> <controller type="pci" index="4" model="pcie-root-port"> <model name="pcie-root-port"/> <target chassis="4" port="0xb"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x3"/> </controller> <controller type="pci" index="5" model="pcie-root-port"> <model name="pcie-root-port"/> <target chassis="5" port="0xc"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x4"/> </controller> <controller type="pci" index="6" model="pcie-root-port"> <model name="pcie-root-port"/> <target chassis="6" port="0xd"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x5"/> </controller> <controller type="pci" index="7" model="pcie-root-port"> <model name="pcie-root-port"/> <target chassis="7" port="0xe"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x6"/> </controller> <controller type="pci" index="8" model="pcie-root-port"> <model name="pcie-root-port"/> <target chassis="8" port="0xf"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x7"/> </controller> <interface type="bridge"> <mac address="52:54:00:a6:c4:ee"/> <source bridge="bridge0"/> <model type="virtio"/> <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/> </interface> <input type="mouse" bus="ps2"/> <input type="keyboard" bus="ps2"/> <hostdev mode="subsystem" type="pci" managed="yes"> <source> <address domain="0x0000" bus="0x02" slot="0x00" function="0x0"/> </source> <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/> </hostdev> <hostdev mode="subsystem" type="pci" managed="yes"> <source> <address domain="0x0000" bus="0x02" slot="0x00" function="0x1"/> </source> <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/> </hostdev> <hostdev mode="subsystem" type="pci" managed="yes"> <source> <address domain="0x0000" bus="0x03" slot="0x00" function="0x0"/> </source> <boot order="1"/> <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/> </hostdev> <hostdev mode="subsystem" type="pci" managed="yes"> <source> <address domain="0x0000" bus="0x07" slot="0x00" function="0x0"/> </source> <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/> </hostdev> <hostdev mode="subsystem" type="pci" managed="yes"> <source> <address domain="0x0000" bus="0x08" slot="0x00" function="0x0"/> </source> <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/> </hostdev> <memballoon model="virtio"> <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/> </memballoon> </devices> </domain>
