How can I make the 2nd GPU (RTX 3060 Ti) be usable by the host system (but disable display output) for CUDA work, and still be usable for GPU passthrough?

Operating Systems & Open Source Virtualization
,
1

I have a 3060 Ti as secondary on Ubuntu, and I want it to be used by the host system for stuff that can only be done by an NVIDIA GPU without using it as display (disable it completely), such as CUDA (i.e. Stable Diffusion). However, I also want to be able to use that GPU for passthrough seamlessly to my VMs if needed, and be able to use it again for CUDA after I’m done using it for VM work.

In short, I still need a display cable plugged into the GPU for the VMs, while still be able to use it for CUDA stuff, like Stable Diffusion, when not being used for passthrough.

I have my GPU currently using vfio-pci driver:

vfio-pci.ids=10de:2486,10de:228b

Would the NVIDIA headless driver work?

EDIT: Just tried with headless driver, and here’s what happens:

  • No output, except for boot splash screen
  • It screwed my desktop layout
  • I can see the monitor plugged into the NVIDIA GPU, but no display gets shown (as expected)
  • I tried to start one of my VMs with GPU passthrough and Virt Manager just hung.
2

It is possible to make it work, but it is finicky. I had it working previously but now I have trouble rebinding to vfio after using my nvidia card on the host system (I suspect the cause is the driver update to 530).

Which steps did you use exactly? Which distro, kernel, nvidia drivers are you using?

3 
Operating System: Kubuntu 23.04
KDE Plasma Version: 5.27.4
KDE Frameworks Version: 5.104.0
Qt Version: 5.15.8
Kernel Version: 6.2.0-19-generic (64-bit)
Graphics Platform: X11
Processors: 24 × AMD Ryzen 9 5900X 12-Core Processor
Memory: 62.7 GiB of RAM
Graphics Processor: AMD Radeon RX 6900 XT
Manufacturer: Gigabyte Technology Co., Ltd.
Product Name: X570S AORUS MASTER
System Version: -CF

I tried to use the NVIDIA 525 headless driver (nvidia-headless-525).

4

I’m not familiar with the headless driver (I use fedora with the rpmfusion driver), but here is how I reattach the gpu after binding it to vfio on boot (or after shutting down the VM):

#! /bin/sh

virsh nodedev-reattach pci_0000_01_00_1
virsh nodedev-reattach pci_0000_01_00_0

echo 0000:01:00.0 > /sys/bus/pci/drivers/vfio-pci/unbind
echo 0000:01:00.1 > /sys/bus/pci/drivers/vfio-pci/unbind

modprobe nvidia_drm modeset=off
modprobe nvidia_modeset
modprobe nvidia_uvm
modprobe nvidia

echo 0000:01:00.1 > /sys/bus/pci/drivers/snd_hda_intel/bind

Device xx.0 is the gpu, xx.1 is the gpu audio.

Binding to the nvidia driver for xx.0 is not necessary, probing the nvidia module already does this. This is on Wayland, I don’t know if Xorg behaves differently here.

6

So I’m looking into single GPU passthrough right now (since that’s the closest I can find that will deal with my situation).

A lot of them seems to be doing a lot of killing the display manager, which I don’t want.

Would this affect my current session? I still want to use my current session when I use my VM.

7

Yes it would affect your current session… The desktop environment attaches to the GPU and the GPU can’t be passed through as long as it is being used. That’s why these guides have you unbind the vtconsole and efi-framebuffer and kill the display manager.

If you try passthrough while some process is using the GPU virtmanager will hang. You can check this with
sudo fuser -v /dev/nvidia0

8

So maybe one of the things I may need to look for is how to make sure Plasma and SDDM doesn’t attach itself to the 2nd GPU.

9

The approach I use for this, is to bind to vfio on boot, then if I need the card in linux I reattach to host using the script I posted above. Important is to reattach only AFTER the display manager has started. Otherwise it sees the GPU and tries to use it.

This works on gnome/gdm with wayland…

1 Like
10

Thanks for the insight! :smiley_cat:

I’ll give this a spin when I get the chance. I wanna use OBS or Stable Diffusion with my host gaming PC :slight_smile:

11

Is it normal this gets spammed?

Feb 06 18:57:29 Y4M1-II kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 507
Feb 06 18:57:29 Y4M1-II kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 507
Feb 06 18:57:29 Y4M1-II kernel: NVRM: GPU 0000:0f:00.0 is already bound to vfio-pci.
Feb 06 18:57:29 Y4M1-II kernel: NVRM: The NVIDIA probe routine was not called for 1 device(s).
Feb 06 18:57:29 Y4M1-II kernel: NVRM: This can occur when another driver was loaded and 
                                NVRM: obtained ownership of the NVIDIA device(s).
Feb 06 18:57:29 Y4M1-II kernel: NVRM: Try unloading the conflicting kernel module (and/or
                                NVRM: reconfigure your kernel without the conflicting
                                NVRM: driver(s)), then try loading the NVIDIA kernel module
                                NVRM: again.
Feb 06 18:57:29 Y4M1-II kernel: NVRM: No NVIDIA devices probed.
Feb 06 18:57:29 Y4M1-II kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 507
Feb 06 18:57:29 Y4M1-II kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 507
Feb 06 18:57:29 Y4M1-II kernel: NVRM: GPU 0000:0f:00.0 is already bound to vfio-pci.
Feb 06 18:57:29 Y4M1-II kernel: NVRM: The NVIDIA probe routine was not called for 1 device(s).
Feb 06 18:57:29 Y4M1-II kernel: NVRM: This can occur when another driver was loaded and 
                                NVRM: obtained ownership of the NVIDIA device(s).
Feb 06 18:57:29 Y4M1-II kernel: NVRM: Try unloading the conflicting kernel module (and/or
                                NVRM: reconfigure your kernel without the conflicting
                                NVRM: driver(s)), then try loading the NVIDIA kernel module
                                NVRM: again.
Feb 06 18:57:29 Y4M1-II kernel: NVRM: No NVIDIA devices probed.
Feb 06 18:57:29 Y4M1-II kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 507
Feb 06 18:57:29 Y4M1-II kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 507

Also, I can’t seem to probe the driver with nvidia-smi

0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD107 [GeForce RTX 4060] [10de:2882] (rev a1)
        Subsystem: Gigabyte Technology Co., Ltd AD107 [GeForce RTX 4060] [1458:4116]
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
0f:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:22be] (rev a1)
        Subsystem: Gigabyte Technology Co., Ltd Device [1458:4116]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

> nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

> cat /usr/local/bin/nvidia-reattach.sh 
#! /usr/bin/bash

virsh nodedev-reattach pci_0000_0f_00_1
virsh nodedev-reattach pci_0000_0f_00_0

echo 0000:0f:00.0 > /sys/bus/pci/drivers/vfio-pci/unbind
echo 0000:0f:00.1 > /sys/bus/pci/drivers/vfio-pci/unbind

modprobe nvidia_drm modeset=off
modprobe nvidia_modeset
modprobe nvidia_uvm
modprobe nvidia

echo 0000:0f:00.1 > /sys/bus/pci/drivers/snd_hda_intel/bind

Detaching seems to work as expected:

> cat /usr/local/bin/nvidia-detach.sh
#! /usr/bin/bash

virsh nodedev-detach pci_0000_0f_00_1
virsh nodedev-detach pci_0000_0f_00_0

0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD107 [GeForce RTX 4060] [10de:2882] (rev a1)
        Subsystem: Gigabyte Technology Co., Ltd AD107 [GeForce RTX 4060] [1458:4116]
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
0f:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:22be] (rev a1)
        Subsystem: Gigabyte Technology Co., Ltd Device [1458:4116]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
12

Yes I get this too when the GPU is bound to vfio. Could not find a way around it sadly.

Normal and expected. If the nvidia driver is not loaded for the card nvidia-smi can’t communicate with it.

13 
> sudo nvidia-reattach.sh
Device pci_0000_0f_00_1 re-attached

Device pci_0000_0f_00_0 re-attached

/usr/local/bin/nvidia-reattach.sh: line 7: echo: write error: No such device
modprobe: ERROR: could not insert 'nvidia_drm': No such device
/usr/local/bin/nvidia-reattach.sh: line 14: echo: write error: Device or resource busy