Vfio-pci not working on nvidia gtx 960

Software & Operating Systems Linux
1

I’m following this tutorial and I am stuck on the part where I want the vfio-pci driver for the gpu i’m trying to pass through. The problem is that i’m unable to change the driver from nvidia to vfio-pci.

I’m running Ubuntu 18.04 with the 4.18.5-041805-generic kernel.

PC setup:
AMD FX-8350
ASUS M5A97 EVO R2.0
GTX 970 - For the host OS
GTX 960 2GB - To pass through

The graphics cards are in their own IOMMU group with nothing else than the graphics cards in them:

IOMMU Group 13 06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401] (rev a1)
IOMMU Group 13 06:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fba] (rev a1)

IOMMU Group 15 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
IOMMU Group 15 01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)

My GRUB cmd line:

GRUB_CMDLINE_LINUX_DEFAULT="iommu=1 amd_iommu=on vfio_pci.ids=10de:0fba,10de:1401"

My /etc/initramfs-tools/modules file:

softdep nvidiafb pre: vfio vfio_pci

vfio
vfio_iommu_type1
vfio_virqfd
options vfio_pci ids=10de:1401,10de:0fba
vfio_pci ids=10de:1401,10de:0fba
vfio_pci
nvidiafb

My /etc/modules file:

vfio
vfio_iommu_type1
vfio_pci ids=10de:0fba,10de:1401

My /etc/modprobe.d/vfio_pci.conf file:

options vfio_pci ids=10de:0fba,10de:1401

When i run lspci -nnv I still get:

06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: ASUSTeK Computer Inc. GM206 [GeForce GTX 960] [1043:8520]
	Flags: bus master, fast devsel, latency 0, IRQ 51, NUMA node 0
	Memory at f4000000 (32-bit, non-prefetchable) [size=16M]
	Memory at a0000000 (64-bit, prefetchable) [size=256M]
	Memory at b0000000 (64-bit, prefetchable) [size=32M]
	I/O ports at c000 [size=128]
	[virtual] Expansion ROM at f5000000 [disabled] [size=512K]
	Capabilities: <access denied>
	Kernel driver in use: nvidia
	Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

06:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fba] (rev a1)
	Subsystem: ASUSTeK Computer Inc. Device [1043:8520]
	Flags: fast devsel, IRQ 14, NUMA node 0
	Memory at f5080000 (32-bit, non-prefetchable) [disabled] [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

So the vfio-pci thing works on the audio part of the gpu but not the gpu part…

I’m pretty much a Linux noob so any help would be greatly appreciated :slight_smile:

2

This is setup a little differently to how I have done mine. So I’m a bit unfamiliar with it.

But try this for /etc/initramfs-tools/modules

softdep nvidiafb pre: vfio vfio_pci

vfio
vfio_iommu_type1
vfio_virqfd
options vfio_pci ids=10de:1401,10de:0fba
vfio_pci ids=10de:1401,10de:0fba
vfio_pci
nvidiafb
nvidia

The issue might be that the nvidia driver is loading before the vfio_pci driver. Hence grabbing the card first.

Let me know how you get on :slight_smile:

3

didn’t work, still shows the nvidia driver :confused:

4

Damn. What module does your host GPU use?

Maybe try softdep nvidia pre: vfio vfio_pci

instead of softdep nvidiafb pre: vfio vfio_pci

5

I tried with nvidia insted of nvidiafb but it still shows the same thing

my host gpu says the same thing as the one i want to use for my guest

6

Try adding the softdep nvidiafb pre: vfio vfio_pci to the modprobe file.
Problem is the nvidia driver module is loaded before vfio_pci. The modules file doesn’t load stuff before late in the boot process, so using the modprobe file instead might work.

7

Did you update initramfs and grub before reboot?

Also, you want nvidia rather then nvidiafb in all config files. The kernel driver is what you want to change out, not the kernel module(s), therefore use nvidia.

8

This guide

https://scottlinux.com/2016/08/28/gpu-passthrough-with-kvm-and-debian-linux/

Plus adding this to your VM xml after creating it in virt-manager( sudo virsh edit vmname ):

" domain type=‘kvm’ xmlns:qemu=‘http://libvirt.org/schemas/domain/qemu/1.0’ "

(Replace this with the first line) ^

qemu:commandline
<qemu:arg value=’-cpu’/>
<qemu:arg value=‘host,hv_time,kvm=off,hv_vendor,hv_vendor_id=null’/>
</qemu:commandline>
"

(Add this right before the end of (or domain?, I think devices) ^

Works flawlessly for me every time.

Also make sure when you first create the VM to select the UEFI bios in the “Overview” tab in virt-manager. Remove all the other video devices except for your nvidia cards.

Let me know if this helped

EDIT/P.S.: I basically combined the guide linked above and this guide https://www.youtube.com/watch?v=dsDUtzMkxFk . I basically only used the magic virsh edit lines given in this video and the UEFI bios suggestion to make everything work (I have no idea why he has you compile your own kernel, maybe cause the video is older?). The rest was the first guide linked. Took me weeks to figure out but it finally happened!

P.S.S. My 980 ti would NOT show up on my screen if I did not select the OVMF UEFI bios when creating the VM. And if you don’t select it during creation, it does not let you change it after making it. You have to make a whole new one over again.

9

That first line for the virsh edit needs < > on either side. This forum wouldn’t let me input it. But yeah.

10

You can also remove the Nvidia driver from the initial ram fs. I’ve had trouble with the Nvidia driver somehow ignoring the softdep line on some configs/distros.

Ls the initial ramdisk and verify the VFIO drivers are present on the ramdisk?

11

Just tried it, didn’t help, thanks for the help anyways :slight_smile:

12

Thanks for the help, I changed nvidiafb to nvidia and updated grub and initramfb and now it is using the vfio_pci driver.