GPU Passthrough - Can't enable nvidia on host dynimically

Operating Systems & Open Source Virtualization
1

Hello, I am currently using a virtual machine and passing through my NVIDIA graphics card 3070 Ti to a Windows guest or also to a Linux guest, which works wonderfully for me.

My goal now is to be able to dynamically use the NVIDIA graphics card on my host after I have shut down the virtual machine, without having to restart the host to reload the NVIDIA drivers.

As I have often seen, this is resolved with custom alias commands to, for example, activate or deactivate the NVIDIA driver if I want to use the virtual machine again with GPU passthrough.

The ID of my graphics card is: 01:00.0

To activate the NVIDIA on the host, I thought the following would work:

sudo virsh nodedev-reattach pci_0000_01_00_0 \
sudo rmmod vfio_pci vfio_pci_core vfio_iommu_type1 \
sudo modprobe -i nvidia_modeset nvidia_uvm nvidia

However, I get the following output when using modprobe: modprobe: ERROR: could not insert 'nvidia_modeset': No such device Also, with sudo modprobe -i nvidia, I get: modprobe: ERROR: could not insert 'nvidia': No such device

The output of lspci -nnk shows me that the vfio-pci driver is in use:

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104 [GeForce RTX 3070 Ti]
       Kernel driver in use: vfio-pci  
       Kernel modules: nouveau, nvidia_drm, nvidia  
01:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller
       Kernel driver in use: vfio-pci  
       Kernel modules: snd_hda_intel

I am using EndeavourOS on the host system, and the installed driver is nvidia 560.35.03-1.

I hope someone can help me, thank you in advance and best regards

1 Like
2

Try reattaching also the GPU’s audio

sudo virsh nodedev-reattach pci_0000_01_00_0

Why do you use the line continuation character \ ?

This is my script:

#! /bin/sh

virsh nodedev-reattach pci_0000_01_00_1
virsh nodedev-reattach pci_0000_01_00_0

echo 0000:01:00.0 > /sys/bus/pci/drivers/vfio-pci/unbind
echo 0000:01:00.1 > /sys/bus/pci/drivers/vfio-pci/unbind

modprobe nvidia-drm modeset=1 fbdev=0
modprobe nvidia nvidia-modeset nvidia-uvm

echo -n "0000:01:00.0" > /sys/bus/pci/drivers/nvidia/bind
echo -n "0000:01:00.1" > /sys/bus/pci/drivers/snd_hda_intel/bind

systemctl start nvidia-persistenced.service

Note also that your desktop environment/window manager may try to latch on the GPU (meaning you won’t be able to unbind without killing it), but you can avoid that. Look here for hints:

1 Like
3

Thank you for sharing your script! it works very well! I can enable the NVIDIA driver on the host side. However, I’m struggling to switch to vfio-pci. It works for the GPU’s audio, as I verified with lspci -nnk, which shows the value changing from snd_hda_intel to vfio-pci.

The problem stays with the VGA GPU. When I run virsh nodedev-detach pci_0000_01_00_0, it just hangs without responses in terminal. Unfortunately, I’m not very familiar with modprobe, so my guess is that the GPU might still be in use by something what causes the hang. The output of nvidia-smi shows only one process: /usr/lib/Xorg.

I’ve been experimenting and thought it might work similarly, but I’m not sure yet.


#! /bin/sh  
    
virsh nodedev-detach pci_0000_01_00_1  
virsh nodedev-detach pci_0000_01_00_0  #Freeze and hangs here?.
  
echo 0000:01:00.0 > /sys/bus/pci/drivers/vfio-pci/bind  
echo 0000:01:00.1 > /sys/bus/pci/drivers/vfio-pci/bind  
  
rmmod nvidia_drm  
rmmod nvidia_modeset  
rmmod nvidia_uvm  
rmmod nvidia  
  
echo -n "0000:01:00.0" > /sys/bus/pci/drivers/nvidia/unbind  
echo -n "0000:01:00.1" > /sys/bus/pci/drivers/snd_hda_intel/unbind
4

Even one process using the GPU will hang the detach. I use the method in the github repo to make sure the desktop environment doesn’t grab the GPU (so I can still use it using prime offload, but it doesn’t draw the DE). I’m using wayland though.

On x.org it must be possible somehow to, probably involving changing the x.org config? I haven’t done this but I’d guess you want to have only the iGPU in the x.org conf and set up prime to render select applications on the dGPU (similar to laptops) or compute.

5

Funnily enough, I just watched this and it might give you some tips?

1 Like