Can't unbind GPU

I want to switch over to Linux as my primary OS but I’m having trouble getting my test setup working.

I’m running an AMD Ryzen 3700X with 2 GPUs, 1 GTX 780 and 1 GTX 980. Monitors hooked up to the GTX 780 only.

So far in my initial testing, it seems like I’m able to use Nvidia Prime/Optimus so that I get the better performance of the 980. Though on games that detect and allow you to select the GPU it refuses to work (at least when I tested with Proton).

I then moved on to trying to set up a Windows VM with GPU passthrough. Unfortunately, it always hangs on unbinding the GTX 980 before I can give it to the VM.

Looking at Nvidia-smi and dmesg when it happens seems to suggest that Xorg is running on the 980 and refuses to let the driver eject it.

I’m not sure of how I can get Xorg to leave the 980 alone and doubly unsure of if Prime/Optimus will work afterwards.

My goal here is to have a setup where I can use the 980 for any applications/games that work on Linux, then on VM launch the VM gets the 980 for Windows exclusive games and when I shut it down Linux gets the 980 back.

You could always trying using a mediated VFIO-mdev driver on your host and share the GPU between host and guest.
Full disclosure: I co-wrote this tool

I’ve seen the tool and I really like the idea however, I’m not sure I can handle losing half my VRAM since my 980 only has 4GB.

I’ve managed to get vfio-pci to load in before the nvidia driver grabs my 980 and starts Xorg on it, however doing that broke my ability to use Prime. Also the fan runs loud when the VM is off because the nvidia driver refuses to monitor the fan curve if nothing is running on the card.

So if I could use libvf.io to work around that in a way that lets me devote my full VRAM to either Host or Guest without having to restart and driver issues, I’d switch in a heartbeat.

Actually you can do that if you want. Here’s an example of a full GPU passthrough with LibVF.IO:

You just close the gpus: later with [] and then add the device ID under commands as seen here:
image

This does full passthrough rather than a mediated partition. :slight_smile:

If you need any help setting this up I’m happy to lend a hand in our Discord server in the #techsupport channel or on GitHub if you post an issue. I can also try my best to help here if that’s easier.