Need help with dynamically binding and un-binding Nvidia GPU

Tree_McGee · March 1, 2020, 10:41pm

The other day I say a post from Bryan Steiner detailing steps to dynamically bind and unbind a GPU from VFIO in Linux. However I believe there’s an issue with it. To use this system you must have two GPUs using two different drivers.

Bryan Steiner’s guide (can’t include links): github - bryansteiner

For my workflow this doesn’t work. I would prefer to have two Nvidia GPUs for use in a CUDA application. OpenCL is supported on AMD GPUs but it has a limited feature set in the application I use, an issue I kept running into over the last two weeks when I had a RX 570 for testing.

Bryan Steiner has pointed out that a dual Nvidia system may be possible with help from PCI override. But I’m unable to figure out how to get it work due to two reasons.

PCI override guide (can’t include links): Arch linux wiki - PCI passthrough via OVMF - Section “Using identical guest and host GPUs”.

I’m fairly new to VFIO and this side of Linux and thus don’t have experience in this area.
I don’t have a second Nvidia GPU to test with as of right now.

So can anyone help? Thanks in advanced - Tree McGee

What I’m trying to achieve.
Have two Nvidia GPUs with the Nvidia proprietary drivers (for CUDA) available in Linux. When I open a my Windows VM, one GPU binds to VFIO for use in Windows. The other GPU I would hope to still be available on Linux if I need it for CUDA or something else. Once I close the VM, the VFIO GPU is bound back to Linux and I can start to use both of them for CUDA again.

I should note that the GPU that I wish to not to bind to VFIO will always be the display out for Linux.

Specs:
CPU: Ryzne 9 3900X
Motherboard: Gigabyte Aorus Elite Wifi (Great IOMMU groups)
RAM: 2x16GB 3200Mhz Corsair Vengeance Pro RGB
GPU 1 (VFIO GPU): Hopefully a RTX 2060 Super
GPU 2 (Linux GPU): GTX 1050ti
Distro: Pop!_OS 18.04 LTS

gordonthree · March 1, 2020, 10:47pm

What have you tried so far? What is the “PCI override?” do you mean ACS override?

You should be able to unbind an UNUSED card from the driver using the sysfs … like echo 1 > /sys/bus/pci/…/unbind as root (no sudo), then bind it to vfio in a similar fashion.

You must give the kernel a console of some sort, or it gets sad. Even a serial console will work, even a junky card in a x1 slot, just something for it to complain about things to. Then you can unbind your expensive cards, and assign them as needed.

Tree_McGee · March 1, 2020, 10:59pm

I haven’t been able to try anything yet because I currently don’t have a second Nvidia GPU to test with. I also no longer have access to my RX 570 so can’t run tests with that.

The PCI override is this script run at boot time as part of initramfs:

#!/bin/sh

DEVS="0000:03:00.0 0000:03:00.1"

if [ ! -z "$(ls -A /sys/class/iommu)" ]; then
    for DEV in $DEVS; do
        echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
    done
fi

From what I’m reading from your post and information from the script, you’re saying I can run
echo "vfio-pci" > /sys/bus/pci/devices/GPU ID/driver_override
echo "vfio-pci" > /sys/bus/pci/devices/GPU AUDIO ID/driver_override
or some variant of that and the Nvidia GPU will bind to VFIO. Then I can run the same commands but replace vfio-pci with the nvidia driver and it should shift back to Linux for use in CUDA applications?

gordonthree · March 2, 2020, 12:05am

I haven’t had any luck with driver_override … But the two step process of unbinding from the driver and binding to VFIO works for me.

gordonthree · March 2, 2020, 12:38am

here’s a script I use for one of my cards, which the host normally uses, but sometimes I assign to a guest. i admit, I haven’t tried running this in reverse.

#!/bin/bash

# unbind from drivers
echo 0000:0a:00.0 > /sys/bus/pci/devices/0000\:0a\:00.0/driver/unbind
echo 0000:0a:00.1 > /sys/bus/pci/devices/0000\:0a\:00.1/driver/unbind

# bind to vfio
echo 0000:0a:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
echo 0000:0a:00.1 > /sys/bus/pci/drivers/vfio-pci/bind

Tree_McGee · March 2, 2020, 2:37am

Thanks for sharing your input. I’ll be sure to give this a try once I get a second GPU.

chonitoe · March 2, 2020, 6:10am

Hey Tree, that would be my guide you’re talking about

Like gordonthree said, you could run a script like this:

#!/bin/bash

# VGA Controller: unbind nvidia and bind vfio-pci
echo '0000:0f:00.0' > /sys/bus/pci/devices/0000:0f:00.0/driver/unbind
echo '10de 100a'    > /sys/bus/pci/drivers/vfio-pci/new_id
echo '0000:0f:00.0' > /sys/bus/pci/devices/0000:0f:00.0/driver/bind
echo '10de 100a'    > /sys/bus/pci/drivers/vfio-pci/remove_id

# Audio Controller: unbind snd_hda_intel and bind vfio-pci 
echo '0000:0f:00.1' > /sys/bus/pci/devices/0000:0f:00.1/driver/unbind
echo '10de 0e1a'    > /sys/bus/pci/drivers/vfio-pci/new_id
echo '0000:0f:00.1' > /sys/bus/pci/devices/0000:0f:00.1/driver/bind
echo '10de 0e1a'    > /sys/bus/pci/drivers/vfio-pci/remove_id

I’m using the virsh tool in my scripts but this works and is a manual way of accomplishing the same task.

Also, it doesn’t matter if you’re using GPUs with the same drivers or not. The only thing to be careful about is this:

## Unload nvidia
modprobe -r nvidia_drm
modprobe -r nvidia_uvm
modprobe -r nvidia_modeset

^^ Since you’ll be using two Nvidia GPUs, you wont be able to unload these modules because they’ll have non-zero usage count (this wasn’t an issue for me because I had an AMD GPU for my host). So go ahead and just delete those lines for your setup.

Tree_McGee · March 2, 2020, 7:20am

So what you’re telling me to do is follow your guide until I get to point where I create bind_vfio.sh. Inside bind_vfio.sh it should look like this?

#!/bin/bash

## Load vfio
modprobe vfio
modprobe vfio_iommu_type1
modprobe vfio_pci

## VGA Controller: unbind nvidia and bind vfio-pci
echo '0000:0f:00.0' > /sys/bus/pci/devices/0000:0f:00.0/driver/unbind
echo '10de 100a'    > /sys/bus/pci/drivers/vfio-pci/new_id
echo '0000:0f:00.0' > /sys/bus/pci/devices/0000:0f:00.0/driver/bind
echo '10de 100a'    > /sys/bus/pci/drivers/vfio-pci/remove_id

## Audio Controller: unbind snd_hda_intel and bind vfio-pci 
echo '0000:0f:00.1' > /sys/bus/pci/devices/0000:0f:00.1/driver/unbind
echo '10de 0e1a'    > /sys/bus/pci/drivers/vfio-pci/new_id
echo '0000:0f:00.1' > /sys/bus/pci/devices/0000:0f:00.1/driver/bind
echo '10de 0e1a'    > /sys/bus/pci/drivers/vfio-pci/remove_id

Do I need to do something like this
echo '10de 100a' > /sys/bus/pci/drivers/nvidia/remove_id
before binding to VFIO or does Linux do a smart thing where a device can only have it’s ID bound to one driver and thus removes the GPU from the nvidia driver ID list when I add it to VFIO?

and for unbind_vfio.sh?

#!/bin/bash

## VGA Controller: unbind vfio-pci and bind nvidia
echo '0000:0f:00.0' > /sys/bus/pci/devices/0000:0f:00.0/driver/unbind
echo '10de 100a'    > /sys/bus/pci/drivers/nvidia/new_id
echo '0000:0f:00.0' > /sys/bus/pci/devices/0000:0f:00.0/driver/bind
echo '10de 0e1a'    > /sys/bus/pci/drivers/nvidia/remove_id

## Audio Controller: unbind vfio-pci and bind snd_hda_intel 
echo '0000:0f:00.1' > /sys/bus/pci/devices/0000:0f:00.1/driver/unbind
echo '10de 0e1a'    > /sys/bus/pci/drivers/snd_hda_intel/new_id
echo '0000:0f:00.1' > /sys/bus/pci/devices/0000:0f:00.1/driver/bind
echo '10de 0e1a'    > /sys/bus/pci/drivers/snd_hda_intel/remove_id

## Unload vfio
modprobe -r vfio_pci
modprobe -r vfio_iommu_type1
modprobe -r vfio

Also, from the looks of it, this setup wouldn’t work with identical GPU models or GPUs with identical controllers (E.G. Audio, Serial, etc) due to the use of identifiers like “10de 100a” and “10de 0e1a”.

In that case, wouldn’t it be better to run something like this for the bind_vfio.sh? as suggested by gordanthree (I’m concerned about this issue as I believe Nvidia may not change their Virtual Link controllers with next gen RTX. And considering I may be upgrading to a RTX 2060 Super and RTX ??70 I really want to make sure my method accounts for this)

#!/bin/bash

## Load vfio
modprobe vfio
modprobe vfio_iommu_type1
modprobe vfio_pci

## Unbind VGA and audio from "standard" drivers
echo 0000:0f:00.0 > /sys/bus/pci/devices/0000:0f:00.0/driver/unbind
echo 0000:0f:00.1 > /sys/bus/pci/devices/0000:0f:00.1/driver/unbind

##Bind VGA and audio to vfio-pci
echo 0000:0f:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
echo 0000:0f:00.1 > /sys/bus/pci/drivers/vfio-pci/bind

Then this for unbind_vfio.sh?

#!/bin/bash

##Unbind VGA and audio to vfio-pci
echo 0000:0f:00.0 > /sys/bus/pci/devices/0000:0f:00.0/driver/unbind
echo 0000:0f:00.1 > /sys/bus/pci/devices/0000:0f:00.1/driver/unbind

##Bind VGA and audio to "standard" drivers
echo 0000:0f:00.0 > /sys/bus/pci/drivers/nvidia/bind
echo 0000:0f:00.1 > /sys/bus/pci/drivers/snd_hda_intel/bind

## Unload vfio
modprobe -r vfio_pci
modprobe -r vfio_iommu_type1
modprobe -r vfio

Sorry. Just trying to wrap my head around all of this. It’s kind of hard for me to picture how all this works when I’m not well versed in VFIO and when I don’t have the hardware on hand to test with.

chonitoe · March 2, 2020, 10:08pm

Yes that should work… The best way of course is to test it out once you have access to the hardware you’ll be using. Until then, try not to get wrapped up in the details. Once the problems and errors present themselves, it’ll be much easier to work out the causes.

enMadz · October 16, 2021, 10:43am

Hello sorry for digging up this thread if its destiny was to rest in peace

I’m using gpu_passthrough for over a year with blacklisting dGPU, and binding it do vfio drivers before system boots on Manjaro.

Now on, being inspired with turorial mentioned in first post (Bryan Steiner’s guide), I would like to try dynamic binding GPU to vfio_pci driver.

There are few other tutorials on the internet claiming that dynamic device binding is doable, but i never had success with that.

My testing platform is new POP OS 20.04 LTS, 5.14.0-12.1-liquorix-amd64 kernel (needed ACS override), with system76-power graphics is set to hybrid, intel iGPU for the host, and nvidia dGPU for the guest.

To this moment, only thing i gained is “fake” unbinding dGPU from nvidia and snd_hda_intel drivers using commands:

echo 0000:01:00.0 > /sys/bus/pci/devices/0000\:01\:00.0/driver/unbind &
echo 0000:01:00.1 > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind &

After executing above commands, symlinks from sysfs /sys/bus/pci/drivers/{nvidia,snd_hda_intel} pointing to dGPU and its sound output are being removed, but if i switch display’s input source to dGPU output it seems it still works.

Using “virsh nodedev-detach … &” seems to work in the same way.

Next step would be binding devices with vfio drivers with command like:

echo '0000:01:00.0' > /sys/bus/pci/drivers/vfio-pci/bind

Unfotunately I get:
‘bash: echo: write error: No such device’

After many tries and failures I’m almost at give up phase

According to that i would like to ask you about your experiences with dynamic gpu binding. Is onyone succeed with scenario described by mentioned tutorial’s author - binding and unbinding dGPU with starting and stopping virtual machine?