How can I make the 2nd GPU (RTX 3060 Ti) be usable by the host system (but disable display output) for CUDA work, and still be usable for GPU passthrough?

Operating Systems & Open Source Virtualization
,
1

I have a 3060 Ti as secondary on Ubuntu, and I want it to be used by the host system for stuff that can only be done by an NVIDIA GPU without using it as display (disable it completely), such as CUDA (i.e. Stable Diffusion). However, I also want to be able to use that GPU for passthrough seamlessly to my VMs if needed, and be able to use it again for CUDA after I’m done using it for VM work.

In short, I still need a display cable plugged into the GPU for the VMs, while still be able to use it for CUDA stuff, like Stable Diffusion, when not being used for passthrough.

I have my GPU currently using vfio-pci driver:

vfio-pci.ids=10de:2486,10de:228b

Would the NVIDIA headless driver work?

EDIT: Just tried with headless driver, and here’s what happens:

  • No output, except for boot splash screen
  • It screwed my desktop layout
  • I can see the monitor plugged into the NVIDIA GPU, but no display gets shown (as expected)
  • I tried to start one of my VMs with GPU passthrough and Virt Manager just hung.
2

It is possible to make it work, but it is finicky. I had it working previously but now I have trouble rebinding to vfio after using my nvidia card on the host system (I suspect the cause is the driver update to 530).

Which steps did you use exactly? Which distro, kernel, nvidia drivers are you using?

3 
Operating System: Kubuntu 23.04
KDE Plasma Version: 5.27.4
KDE Frameworks Version: 5.104.0
Qt Version: 5.15.8
Kernel Version: 6.2.0-19-generic (64-bit)
Graphics Platform: X11
Processors: 24 × AMD Ryzen 9 5900X 12-Core Processor
Memory: 62.7 GiB of RAM
Graphics Processor: AMD Radeon RX 6900 XT
Manufacturer: Gigabyte Technology Co., Ltd.
Product Name: X570S AORUS MASTER
System Version: -CF

I tried to use the NVIDIA 525 headless driver (nvidia-headless-525).

4

I’m not familiar with the headless driver (I use fedora with the rpmfusion driver), but here is how I reattach the gpu after binding it to vfio on boot (or after shutting down the VM):

#! /bin/sh

virsh nodedev-reattach pci_0000_01_00_1
virsh nodedev-reattach pci_0000_01_00_0

echo 0000:01:00.0 > /sys/bus/pci/drivers/vfio-pci/unbind
echo 0000:01:00.1 > /sys/bus/pci/drivers/vfio-pci/unbind

modprobe nvidia_drm modeset=off
modprobe nvidia_modeset
modprobe nvidia_uvm
modprobe nvidia

echo 0000:01:00.1 > /sys/bus/pci/drivers/snd_hda_intel/bind

Device xx.0 is the gpu, xx.1 is the gpu audio.

Binding to the nvidia driver for xx.0 is not necessary, probing the nvidia module already does this. This is on Wayland, I don’t know if Xorg behaves differently here.

6

So I’m looking into single GPU passthrough right now (since that’s the closest I can find that will deal with my situation).

A lot of them seems to be doing a lot of killing the display manager, which I don’t want.

Would this affect my current session? I still want to use my current session when I use my VM.

7

Yes it would affect your current session… The desktop environment attaches to the GPU and the GPU can’t be passed through as long as it is being used. That’s why these guides have you unbind the vtconsole and efi-framebuffer and kill the display manager.

If you try passthrough while some process is using the GPU virtmanager will hang. You can check this with
sudo fuser -v /dev/nvidia0

8

So maybe one of the things I may need to look for is how to make sure Plasma and SDDM doesn’t attach itself to the 2nd GPU.

9

The approach I use for this, is to bind to vfio on boot, then if I need the card in linux I reattach to host using the script I posted above. Important is to reattach only AFTER the display manager has started. Otherwise it sees the GPU and tries to use it.

This works on gnome/gdm with wayland…

1 Like
10

Thanks for the insight! :smiley_cat:

I’ll give this a spin when I get the chance. I wanna use OBS or Stable Diffusion with my host gaming PC :slight_smile: