[SOLVED] Help with Dual Nvidia GPU and Looking Glass

All:

This has been resolved, see: [SOLVED] Help with Dual Nvidia GPU and Looking Glass - #17 by wallacebw

I was playing with looking glass with a Arch system this week with a 3090 (Primary) and a 780TI (assigned to guest).

I received my 4090 today (ASUS 4090TUF-OC) and when i remove the 780 from the system and replace it with the 4090, it is recognized by arch is working find within linux, however, the VM does not boot and no errors are thrown in VMM.

I was thinking that this was because the 780TI IS NOT supported by the latest NVIDIA drivers, while the 4090 is. To troubleshoot, I updated my xorg.conf to add:

Section "ServerFlags"
    Option "AutoAddGPU" "off"
EndSection

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "NVIDIA GeForce RTX 3090"
    BusID          "PCI:1:0:0"
EndSection

After making this change, nvidis-smi shows no processes running on the 4090 (card 1):

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06    Driver Version: 520.56.06    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
|  0%   26C    P8    35W / 350W |    663MiB / 24576MiB |      8%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:4B:00.0 Off |                  Off |
|  0%   34C    P8     6W / 450W |      1MiB / 24564MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      3329      G   /usr/lib/Xorg                     160MiB |
|    0   N/A  N/A      3486      G   /usr/bin/gnome-shell              130MiB |
|    0   N/A  N/A      4166      G   /usr/lib/firefox/firefox          369MiB |
+-----------------------------------------------------------------------------+

The VM is still failing to boot, what else needs to be done?

NOTE: I do not want to blacklist the device, as I use it for work (CUDA loads) during the day.

Thanks

I found this link:

Which made the nvidia drivers ‘skip’ the 4090.

EDIT: I’ve adjusted to a similar, but more ARCH native approach. Same issues, VM Powers on, but no display and the OS Doesn’t load

Now the VM ‘powers on’ to a black-screen and never boots. No uefi splash, the IP of the windows guest never comes online, etc.

@gnif Any thoughts?

Thanks for anyone that can help

What GPU are you passing through to the VM, the 3090 or the 4090?

When you run “lspci -nnk”, what kernel driver is in use for the 3090 and 4090? The card you’re passing to the guest should be vfio.

After you replaced the card did you update the initrd / grub with the new PCI id for the VFIO module to bind to?

Do you get any output when plugging a screen directly into the passed through GPU?

1 Like

I’m trying to pass through the 4090. Here are more details on where I’m at as of now:

  • No Video on the VM
  • No Video on the attached Monitor
  • no IP connectivity with the Guest
  • as soon as I remove the pci passthrough devices it boots

One Interesting Note:

if I set the video type to Ramfb, I get the UEFI screen with a

map: No Mapping Found

This makes me think it may be a uefi /vbios thing, but need to look more into that.

PCI Info

The 4090 is mapped to the vfio-pci driver.


lspci -nnk | grep -i -A3 vga
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204] (rev a1)
	Subsystem: ASUSTeK Computer Inc. Device [1043:87b3]
	Kernel driver in use: nvidia
	Kernel modules: nouveau, nvidia_drm, nvidia
4b:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD102 [GeForce RTX 4090] [10de:2684] (rev a1)
	Subsystem: ASUSTeK Computer Inc. Device [1043:889a]
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau, nvidia_drm, nvidia

lspci -nnk | grep  -A3 '^01'
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204] (rev a1)
	Subsystem: ASUSTeK Computer Inc. Device [1043:87b3]
	Kernel driver in use: nvidia
	Kernel modules: nouveau, nvidia_drm, nvidia
01:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
	Subsystem: ASUSTeK Computer Inc. Device [1043:87b3]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

lspci -nnk | grep  -A3 '^4b'
4b:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD102 [GeForce RTX 4090] [10de:2684] (rev a1)
	Subsystem: ASUSTeK Computer Inc. Device [1043:889a]
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau, nvidia_drm, nvidia
4b:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:22ba] (rev a1)
	Subsystem: ASUSTeK Computer Inc. Device [1043:889a]
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

OS Settings

mkinitcpio.conf

MODULES=(vfio_pci vfio vfio_iommu_type1 vfio_virqfd)

Bootloader setup (systemd-boot):

title     Arch Linux (GPU Passthrough)
linux     /vmlinuz-linux
initrd    /amd-ucode.img
initrd    /initramfs-linux.img
options   zfs=rpool/ROOT/default rw
options   amd_iommu=on
options   iommu=py
options   vfio-pci.ids=10de:2684,10de:22ba

Guest Setup

I built a new, clean windows 10 VM based on the below. (no optimizations)

With the below the VM appears to power on, but the VM does not boot:

  • No Video on the VM
  • No Video on the attached Monotor
  • no IP connectivity with the guest
<domain type="kvm">
  <name>win10-2</name>
  <uuid>23e9f019-4bfe-4746-989b-be785ebde42c</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">33554432</memory>
  <currentMemory unit="KiB">33554432</currentMemory>
  <vcpu placement="static">16</vcpu>
  <os firmware="efi">
    <type arch="x86_64" machine="pc-q35-7.1">hvm</type>
    <boot dev="hd"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode="custom">
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
    </hyperv>
    <vmport state="off"/>
  </features>
  <cpu mode="host-passthrough" check="none" migratable="on">
    <topology sockets="1" dies="1" cores="8" threads="2"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" discard="unmap"/>
      <source file="/var/lib/libvirt/images/win10-2.qcow2"/>
      <target dev="sda" bus="sata"/>
      <address type="drive" controller="0" bus="0" target="0" unit="0"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/var/lib/libvirt/images/ISO/Win10_20H2_v2_English_x64.iso"/>
      <target dev="sdb" bus="sata"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="1"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x16"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0x17"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0x18"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="10" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="10" port="0x19"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
    </controller>
    <controller type="pci" index="11" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="11" port="0x1a"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
    </controller>
    <controller type="pci" index="12" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="12" port="0x1b"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
    </controller>
    <controller type="pci" index="13" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="13" port="0x1c"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
    </controller>
    <controller type="pci" index="14" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="14" port="0x1d"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
    </controller>
    <controller type="pci" index="15" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="15" port="0x8"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
    </controller>
    <controller type="pci" index="16" model="pcie-to-pci-bridge">
      <model name="pcie-pci-bridge"/>
      <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </controller>
    <interface type="bridge">
      <mac address="52:54:00:1a:08:fd"/>
      <source bridge="kvmbridge"/>
      <model type="e1000e"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </interface>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial"/>
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
    </console>
    <channel type="spicevmc">
      <target type="virtio" name="com.redhat.spice.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="1"/>
    </channel>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <input type="keyboard" bus="virtio">
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
    </input>
    <graphics type="spice" autoport="yes">
      <listen type="address"/>
      <image compression="off"/>
    </graphics>
    <sound model="ich9">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
    </sound>
    <audio id="1" type="spice"/>
    <video>
      <model type="none"/>
    </video>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x4b" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x4b" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </hostdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="2"/>
    </redirdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="3"/>
    </redirdev>
    <memballoon model="none"/>
    <shmem name="looking-glass">
      <model type="ivshmem-plain"/>
      <size unit="M">32</size>
      <address type="pci" domain="0x0000" bus="0x10" slot="0x01" function="0x0"/>
    </shmem>
  </devices>
</domain>


It looks like your VM is booted in BIOS/MBR mode, maybe try UEFI mode? I’m not sure if it makes a difference, but IIRC every example I’ve seen uses UEFI in the VM. (I’ve got an AMD GPU passthrough using UEFI as well). Ignore me, apparently I can’t read.

It sounds like it’s failing at a very early stage. Do you get any PCIe errors in dmesg? And are the GPU fans spinning?

The 4090 fans don’t run unless under load, so not a good indicator

It’s like the pci devices are messing with UEFI and it’s not able to parse the device list and/or identify a bootable device

I see the following when using video mode ramfb, but I cannot type in the UEFI menu, boot into bios or otherwise interact with the device. I even tried passing a second usb keyboard device into the VM, nada. :confused:

image

no useful dmesg output to my eyes.

DMESG

Startup:
[13861.735365] audit: type=2502 audit(1666195396.750:704): pid=2548 uid=0 auid=4294967295 ses=4294967295 msg='virt=kvm vm="win10-2" uuid=23e9f019-4bfe-4746-989b-be785ebde42c vm-ctx=+967:+967 img-ctx=+967:+967 model=dac exe="/usr/bin/libvirtd" hostname=? addr=? terminal=? res=success'
[13861.768526] kvmbridge: port 2(vnet18) entered blocking state
[13861.768553] kvmbridge: port 2(vnet18) entered disabled state
[13861.768639] device vnet18 entered promiscuous mode
[13861.768683] audit: type=1700 audit(1666195396.783:705): dev=vnet18 prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
[13861.768814] audit: type=1300 audit(1666195396.783:705): arch=c000003e syscall=16 success=yes exit=0 a0=23 a1=89a2 a2=7f45b221a0c0 a3=7f45b2219e16 items=0 ppid=1 pid=2548 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rpc-libvirtd" exe="/usr/bin/libvirtd" key=(null)
[13861.768819] audit: type=1327 audit(1666195396.783:705): proctitle=2F7573722F62696E2F6C69627669727464002D2D74696D656F757400313230
[13861.768981] kvmbridge: port 2(vnet18) entered blocking state
[13861.768986] kvmbridge: port 2(vnet18) entered forwarding state
[13861.769049] audit: type=2501 audit(1666195396.786:706): pid=2548 uid=0 auid=4294967295 ses=4294967295 msg='virt=kvm resrc=net reason=open vm="win10-2" uuid=23e9f019-4bfe-4746-989b-be785ebde42c net=52:54:00:1a:08:fd path="/dev/net/tun" rdev=0A:C8 exe="/usr/bin/libvirtd" hostname=? addr=? terminal=? res=success'
[13861.889634] audit: type=1334 audit(1666195396.906:707): prog-id=60 op=LOAD
[13861.889644] audit: type=1300 audit(1666195396.906:707): arch=c000003e syscall=321 success=yes exit=33 a0=5 a1=7f45b221a120 a2=90 a3=40000 items=0 ppid=1 pid=2548 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rpc-libvirtd" exe="/usr/bin/libvirtd" key=(null)
[13861.889649] audit: type=1327 audit(1666195396.906:707): proctitle=2F7573722F62696E2F6C69627669727464002D2D74696D656F757400313230
[13861.889704] audit: type=2501 audit(1666195396.906:708): pid=2548 uid=0 auid=4294967295 ses=4294967295 msg='virt=kvm resrc=cgroup reason=deny vm="win10-2" uuid=23e9f019-4bfe-4746-989b-be785ebde42c cgroup="/sys/fs/cgroup/machine.slice/machine-qemu\x2d20\x2dwin10\x2d2.scope/" class=all exe="/usr/bin/libvirtd" hostname=? addr=? terminal=? res=success'
[13861.889744] audit: type=2501 audit(1666195396.906:709): pid=2548 uid=0 auid=4294967295 ses=4294967295 msg='virt=kvm resrc=cgroup reason=allow vm="win10-2" uuid=23e9f019-4bfe-4746-989b-be785ebde42c cgroup="/sys/fs/cgroup/machine.slice/machine-qemu\x2d20\x2dwin10\x2d2.scope/" class=path path="/dev/null" rdev=01:03 acl=rw exe="/usr/bin/libvirtd" hostname=? addr=? terminal=? res=success'
[13864.968900] vfio-pci 0000:4b:00.0: vfio_ecap_init: hiding ecap [email protected]
[13864.968922] vfio-pci 0000:4b:00.0: vfio_ecap_init: hiding ecap [email protected]
[13864.968930] vfio-pci 0000:4b:00.0: vfio_ecap_init: hiding ecap [email protected]
[13864.968931] vfio-pci 0000:4b:00.0: vfio_ecap_init: hiding ecap [email protected]
[13864.968933] vfio-pci 0000:4b:00.0: vfio_ecap_init: hiding ecap [email protected]

---------------------
Shutdown

[13891.548880] kvmbridge: port 2(vnet18) entered disabled state
[13891.549172] device vnet18 left promiscuous mode
[13891.549180] kvmbridge: port 2(vnet18) entered disabled state
[13891.549196] kauditd_printk_skb: 20 callbacks suppressed
[13891.549200] audit: type=1700 audit(1666195426.566:730): dev=vnet18 prom=0 old_prom=256 auid=4294967295 uid=967 gid=967 ses=4294967295
[13892.169918] audit: type=1334 audit(1666195427.186:731): prog-id=0 op=UNLOAD
[13892.175665] audit: type=2500 audit(1666195427.193:732): pid=2548 uid=0 auid=4294967295 ses=4294967295 msg='virt=kvm op=stop reason=destroyed vm="win10-2" uuid=23e9f019-4bfe-4746-989b-be785ebde42c vm-pid=0 exe="/usr/bin/libvirtd" hostname=? addr=? terminal=? res=success'

Ok, my PC is starting to look like a mad science experiment.

I now have:

  • 3090 (primary display)
  • 4090 (assigned vfio-pci ‘driver’)
  • 780ti (not supported by any currently installed drivers)

New testing

  • I reinstalled the 780Ti and verified it still worked with my new test VM
  • I tested assigning the 780 to vfio and pass-through sill worked as expected (in case vfio was causing the issue.)
  • I testing making the 4090 my primary device and passing through the 3090, same symptoms as listed above for the 4090
  • I tried unloading (modprobe -r) the vfio modules after boot and then starting the VM, same issues.

@wendell (Disregard) Have you ever tried pci passthrough / looking glass with a 3000 or 4000 series card?

SOLVED

it was related to resizable bar

REF: [Solved] RTX 3090 GPU passthrough just displays a black screen with qemu - #5 by bambinone

2 Likes

does this mean you’re able to use your 4090 inside arch, then start a windows 10 vm and use it inside there and then when you exit that vm; it hands the 4090 back to arch?

I’ve only ever fully passed pci devices to a vm in the past so I’m not sure how this works yet

No, sorry…

Arch is using the 3090 as it’s primary device and I am assigning the 4090 to the guest.

I can reboot to a different bootloader config and use both GPUs for arch (CUDA stuff) by removing this kernel option.

options   vfio-pci.ids=10de:2684,10de:22ba

gotcha thanks. Glad your issue is sorted

thanks. Still better than dual boot, because windows always changes either the UEFI boot order (WHY!!!) or overwrites the linux bootloader (GRRR)

yeah I think this is going to be what I do on my next computer I’ll just reboot like you are to either pass my dgpu through or not to windows and hopefully use the integrated gpu for my display

Alternatively you could run linux and windows in VMs, and stop/start VMs to switch GPUs between workloads.

1 Like

It’s possible to do that. I have a 3080Ti I use in my VM, but if my VM is not used, I unbind it from the vfio driver and bind it to the Linux Nvidia driver. This allows me to run CUDA loads without taxing the 3060Ti I have on my host. If I then need to use the VM, I’ll do the reverse; unbind it from the Nvidia driver, and re-bind it to the vfio driver.

3 Likes

Good thought. If I boot the PC with the passthrough device bound to vfio, then X11 is not using the device and I don’t have to make a lot of other misc changes to unbind it from host. When I want to pass it through to a VM, I now just run:

echo  -n "0000:4b:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
echo  -n "0000:4b:00.0" > /sys/bus/pci/drivers/nvidia/bind

and to free it up for the guest just reverse it…

echo  -n "0000:4b:00.0" > /sys/bus/pci/drivers/nvidia/unbind
echo  -n "0000:4b:00.0" > /sys/bus/pci/drivers/vfio-pci/bind

even better than a reboot, Thanks!

1 Like

I wrote myself a little guide and a few scripts to manage GPU passthrough and thought I would share.

NOTE: This is for arch and seems to work for me, YMMV

Configuration for VMM PCI Passthrough with NVidia GPU for host and VM

Setup Host


mkinitpcio.conf

Early Load VFIO before Nvidia drivers

REF: https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF

MODULES=(vfio_pci vfio vfio_iommu_type1 vfio_virqfd nvidia nvidia_modeset nvidia_uvm nvidia_drm)

Boot Loader

Add Device ID(s) of passthrough device(s) to VFIO driver in options.

REF: https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF

options   vfio-pci.ids=10de:2684,10de:22ba

Enable Hugepages

REF: https://looking-glass.io/wiki/Performance_Optimization

  • create a sysctl entry (/etc/sysctl.d/10-hugepages.conf) to create the hugepages (see REF for sizing)
vm.nr_hugepages=33000
vm.hugetlb_shm_group=992
sys.kernel.mm.transparent_hugepage.defrag='never'
sys.kernel.mm.transparent_hugepage.enabled='never'
  • add Hugepages to /etc/fstab to mount with appropriate permissions
 # HUGEPAGES
 hugetlbfs           /dev/hugepages  hugetlbfs       mode=01770,gid=kvm        0 0
  • Create a file /etc/tmpfiles.d/10-looking-glass.conf
# Shared Memory
f /dev/shm/looking-glass 0660 brian kvm -

Install Looking Glass Client on arch

REF: https://aur.archlinux.org/packages/looking-glass

 

Setup VMM (libvirt) VM


Pre-Install

After creating the VM update it’s XML

EDITOR=vim virsh edit VM_Name

# Change Domain tag to include Namespace
# REF: https://www.reddit.com/r/VFIO/comments/qx4rg7/can_anyone_else_confirm_that_vfio_doesnt_work_w/
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

# Add qemu command line parameters to increase the mmio address space to 64MB -- ReBar Support (within Domain, after devices)
# REF: https://www.reddit.com/r/VFIO/comments/qx4rg7/can_anyone_else_confirm_that_vfio_doesnt_work_w/
<qemu:commandline>
 <qemu:arg value='-fw_cfg'/>
 <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/>
</qemu:commandline>

The following can also be done within VMM XML Editor

# Pin the VCPUs to PCPUs in the same CCD (within devices)
# REF: https://looking-glass.io/wiki/Performance_Optimization
  <cputune>
    <vcpupin vcpu='0' cpuset='8'/>
    <vcpupin vcpu='1' cpuset='9'/>
    <vcpupin vcpu='2' cpuset='10'/>
    <vcpupin vcpu='3' cpuset='11'/>
    <vcpupin vcpu='4' cpuset='12'/>
    <vcpupin vcpu='5' cpuset='13'/>
    <vcpupin vcpu='6' cpuset='14'/>
    <vcpupin vcpu='7' cpuset='15'/>
    <vcpupin vcpu='8' cpuset='40'/>
    <vcpupin vcpu='9' cpuset='41'/>
    <vcpupin vcpu='10' cpuset='42'/>
    <vcpupin vcpu='11' cpuset='43'/>
    <vcpupin vcpu='12' cpuset='44'/>
    <vcpupin vcpu='13' cpuset='45'/>
    <vcpupin vcpu='14' cpuset='46'/>
    <vcpupin vcpu='15' cpuset='47'/>
  </cputune>

# Create the shared memory space for looking glass (within devices)
# REF: https://looking-glass.io/docs/B5.0.1/install/
<shmem name='looking-glass'>
  <model type='ivshmem-plain'/>
  <size unit='M'>32</size>
</shmem>

# Enable HugePages memory backing (within devices)
# REF: https://looking-glass.io/wiki/Performance_Optimization
  <memoryBacking>
    <hugepages/>
  </memoryBacking>

# Enable shared clipboard (within devices)
# REF: https://looking-glass.io/docs/B5.0.1/install/#client-determining-memory
<channel type="spicevmc">
  <target type="virtio" name="com.redhat.spice.0"/>
  <address type="virtio-serial" controller="0" bus="0" port="1"/>
</channel>

# Disable membaloon (within devices)
# REF: https://looking-glass.io/docs/B5.0.1/install/
<memballoon model="none"/>

Install Guest

Post Install

  • Find your <video> device, and set <model type=‘none’/>
  • Remove the <input type=‘tablet’/> device, if you have one
  • Create an <input type=‘keyboard’ bus=‘virtio’/> device to improve keyboard usage
    • This requires the vioinput driver from virtio-win to be installed in the guest

 

Automate GPU transition to/from host/guest


To simplify transitioning from GPU host usage to VM usage, a couple scripts may help. Should be easiy enough to modify for your use case. (review the VARS)

Assign Device to VFIO and Launch VM

#! /bin/bash
###############
# DEFINE VARS #
###############
PCI_ID='0000:4b:00.0'
VM_NAME='win11'
DRIVER_GPU='nvidia'
DRIVER_VFIO='vfio-pci'

######################
# ASSIGN GPU TO VFIO #
######################
echo "Assigning ${PCI_ID} to Host:"
if [[ -e "/sys/bus/pci/devices/${PCI_ID}/driver" ]] ; then
    CURRENT_DRIVER=`readlink "/sys/bus/pci/devices/${PCI_ID}/driver" | sed 's/.*\/drivers\///'`
    case ${CURRENT_DRIVER} in
        ${DRIVER_VFIO})
            echo " -- ${PCI_ID} is already assigned driver (${DRIVER_VFIO}).  Nothing to do."
            ;;
        ${DRIVER_GPU})
            sudo tee /sys/bus/pci/drivers/${DRIVER_GPU}/unbind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to unassign ${PCI_ID} from ${DRIVER_GPU}."
            sudo tee /sys/bus/pci/drivers/${DRIVER_VFIO}/bind <<< ${PCI_ID} > /dev/null    || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_VFIO}."
            echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_VFIO})."
            ;;
        *)
            echo " -- ERROR: Unexpected driver (${CURRENT_DRIVER}) assigned, no action taken."
            ;;
    esac
else
    sudo tee /sys/bus/pci/drivers/${DRIVER_VFIO}/bind <<< ${PCI_ID} > /dev/null    || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_VFIO}."
    echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_VFIO})."
fi

#####################################
# START VM AND LAUNCH LOOKING-GLASS #
#####################################
echo "Launching VM (${VM_NAME}):"
echo " -- Starting VM (${VM_NAME}):"
VM_STATUS=`sudo virsh list --name | grep -xc '${VM_NAME}'`
if [[ ${VM_STATUS} == 0  ]]; then
    sudo virsh start ${VM_NAME} || echo " -- ERROR: Unable to start ${VM_NAME}"
else
    echo " -- ${VM_NAME} is already running."
fi
echo " -- Launching Looking Glass:"
looking-glass-client -F -k -Q -S  spice:captureOnStart > /dev/null 2>&1

######################
# ASSIGN GPU TO HOST #
######################
echo -n 'Reassign GPU to host? [Y,N]:'
read ASSIGN_TO_HOST
case ${ASSIGN_TO_HOST} in
    'y' | 'Y')
        echo "Assigning ${PCI_ID} to Host"
        echo " -- Waiting on VM to shutdown (MAX 60 Seconds)"
        i=0
        while [[ '`sudo virsh list --name | grep -xc '${VM_NAME}'`' == 1 ]]; do
            if [[ $i == 60 ]]; then
                echo "${VM_NAME} has not stopped, skipping assigning GPU (${PCI_ID}) to host"
                exit
            fi
            sleep 1
            ((i++))
        done

        if [[ -e "/sys/bus/pci/devices/${PCI_ID}/driver" ]] ; then
            CURRENT_DRIVER=`readlink "/sys/bus/pci/devices/${PCI_ID}/driver" | sed 's/.*\/drivers\///'`
            case ${CURRENT_DRIVER} in
                ${DRIVER_GPU})
                    echo " -- ${PCI_ID} is already assigned driver (${DRIVER_GPU}).  Nothing to do."
                    ;;
                ${DRIVER_VFIO})
                    sudo tee /sys/bus/pci/drivers/${DRIVER_VFIO}/unbind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to unassign ${PCI_ID} from ${DRIVER_VFIO}."
                    sudo tee /sys/bus/pci/drivers/${DRIVER_GPU}/bind <<< ${PCI_ID} > /dev/null    || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_GPU}."
                    echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_GPU})."
                    ;;
                *)
                    echo " -- ERROR: Unexpected driver (${CURRENT_DRIVER}) assigned, no action taken."
                    ;;
            esac
        else
            sudo tee /sys/bus/pci/drivers/${DRIVER_GPU}/bind <<< ${PCI_ID} > /dev/null    || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_GPU}."
            echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_GPU})."
        fi
        ;;
    'n' | 'N')
        echo " -- Skipping GPU resaaignment to host"
        ;;
    *)
        echo " -- Response not understood, skipping GPU resaaignment to host"
        ;;
esac

Assign Device to HOST

#! /bin/bash
###############
# DEFINE VARS #
###############
PCI_ID='0000:4b:00.0'
VM_NAME='win11'
DRIVER_GPU='nvidia'
DRIVER_VFIO='vfio-pci'

######################
# ASSIGN GPU TO HOST #
######################
echo "Assigning ${PCI_ID} to Host:"
if [[ -e "/sys/bus/pci/devices/${PCI_ID}/driver" ]] ; then
    CURRENT_DRIVER=`readlink "/sys/bus/pci/devices/${PCI_ID}/driver" | sed 's/.*\/drivers\///'`
    case ${CURRENT_DRIVER} in
        ${DRIVER_GPU})
            echo " -- ${PCI_ID} is already assigned driver (${DRIVER_GPU}).  Nothing to do."
            ;;
        ${DRIVER_VFIO})
            sudo tee /sys/bus/pci/drivers/${DRIVER_VFIO}/unbind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to unassign ${PCI_ID} from ${DRIVER_VFIO}."
            sudo tee /sys/bus/pci/drivers/${DRIVER_GPU}/bind <<< ${PCI_ID} > /dev/null    || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_GPU}."
            echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_GPU})."
            ;;
        *)
            echo " -- ERROR: Unexpected driver (${CURRENT_DRIVER}) assigned, no action taken."
            ;;
    esac
else
    sudo tee /sys/bus/pci/drivers/${DRIVER_GPU}/bind <<< ${PCI_ID} > /dev/null    || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_GPU}."
    echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_GPU})."
fi

Assign Device to VFIO

#! /bin/bash
###############
# DEFINE VARS #
###############
PCI_ID='0000:4b:00.0'
VM_NAME='win11'
DRIVER_GPU='nvidia'
DRIVER_VFIO='vfio-pci'

######################
# ASSIGN GPU TO VFIO #
######################
echo "Assigning ${PCI_ID} to Host:"
if [[ -e "/sys/bus/pci/devices/${PCI_ID}/driver" ]] ; then
    CURRENT_DRIVER=`readlink "/sys/bus/pci/devices/${PCI_ID}/driver" | sed 's/.*\/drivers\///'`
    case ${CURRENT_DRIVER} in
        ${DRIVER_VFIO})
            echo " -- ${PCI_ID} is already assigned driver (${DRIVER_VFIO}).  Nothing to do."
            ;;
        ${DRIVER_GPU})
            sudo tee /sys/bus/pci/drivers/${DRIVER_GPU}/unbind <<< ${PCI_ID} > /dev/null || echo " -- ERROR: Unable to unassign ${PCI_ID} from ${DRIVER_GPU}."
            sudo tee /sys/bus/pci/drivers/${DRIVER_VFIO}/bind <<< ${PCI_ID} > /dev/null    || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_VFIO}."
            echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_VFIO})."
            ;;
        *)
            echo " -- ERROR: Unexpected driver (${CURRENT_DRIVER}) assigned, no action taken."
            ;;
    esac
else
    sudo tee /sys/bus/pci/drivers/${DRIVER_VFIO}/bind <<< ${PCI_ID} > /dev/null    || echo " -- ERROR: Unable to assign ${PCI_ID} to ${DRIVER_VFIO}."
    echo " -- ${PCI_ID} successfully assigned driver (${DRIVER_VFIO})."

I hope this helps someone and let me know if there are any changes or improvments to the guide / scripts (i’m not a programmer by trade:))

Thanks,
Brian

4 Likes