GPU pass-through: vfio-pci not claiming device (Nvidia & Intel)

Software & Operating Systems Linux
, ,
1

After @wendell did another live stream on the whole GPU pass-through topic, I decided to finally give it a shot on my Intel system. Since my hardware (4790K, Z97) supports VT-d and IOMMU is working, I thought there shouldn’t be an issue, if I follow the Level1Techs tutorial and adjust it to my hardware.

I managed to get IOMMU working after setting the kernel parameters in GRUB

GRUB_CMDLINE_LINUX="[...] iommu=1 intel_iommu=on rd.driver.pre=vfio-pci"

and I was able to identify the grouping of my PCI devices. My Nvidia GTX 770 video card is in a separate IOMMU group together with the PCIe x16 controller that it is seated in.

IOMMU group 1
	00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
	01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1)
	01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)

Since I remember Wendell once mentioning that all devices in an IOMMU group have to be passed-through, I wrote down all three IDs for the next steps.

I added the options option to my /etc/modprobe.d/vfio.conf file as follows:

options vfio-pci ids=8086:0c01,10de:1184,10de:0e0a

To write the changes to the initram-image I ran dracut as per the tutorial:

dracut -f --kver `uname –r`

which ran without any error message. (I also updated GRUB with grub2-mkconfig).

Now, after doing a reboot the nouveau driver is still loaded for the Nvidia card. If I blacklist it in a separate modprobe.d file, the boot process freezes during the systemd messages.

My assumption is that the vfio-pci driver couldn’t be found/loaded and that is why it falls back to the nouveau driver and why it can’t continue booting if I blacklist that one. However, the vfio-pci module does exist:

# modinfo vfio-pci
filename:       /lib/modules/4.13.9-200.fc26.x86_64/kernel/drivers/vfio/pci/vfio-pci.ko.xz
description:    VFIO PCI - User Level meta-driver
author:         Alex Williamson <[email protected]>
license:        GPL v2
version:        0.2
srcversion:     DB8F55EC2187EC83F7E71EA
depends:        vfio,irqbypass,vfio_virqfd
intree:         Y
name:           vfio_pci
vermagic:       4.13.9-200.fc26.x86_64 SMP mod_unload 
signat:         PKCS#7
signer:         
sig_key:        
sig_hashalgo:   md4
parm:           ids:Initial PCI IDs to add to the vfio driver, format is "vendor:device[:subvendor[:subdevice[:class[:class_mask]]]]" and multiple comma separated entries can be specified (string)
parm:           nointxmask:Disable support for PCI 2.3 style INTx masking.  If this resolves problems for specific devices, report lspci -vvvxxx to [email protected] so the device can be fixed automatically via the broken_intx_masking flag. (bool)
parm:           disable_vga:Disable VGA resource access through vfio-pci (bool)
parm:           disable_idle_d3:Disable using the PCI D3 low power state for idle, unused devices (bool)

I’d also like to add that I am using Fedora’s built-in full-disk-encryption (LVM on LUKS, unencrypted /boot) but I don’t think that should be an issue here.

Does someone know what could be wrong here or how I can go about to debug this?

EDIT: I went through the journalctl and found the following line, which doesn’t make sense to me currently:

dracut-pre-udev[385]: modprobe: FATAL: Module vfio-pci not found in directory /lib/modules/4.13.9-200.fc26.x86_64

Because when I run find in the mentioned directory there is the corresponding module:

# find /lib/modules/4.13.9-200.fc26.x86_64 | grep vfio    
/lib/modules/4.13.9-200.fc26.x86_64/kernel/drivers/vfio
/lib/modules/4.13.9-200.fc26.x86_64/kernel/drivers/vfio/vfio_iommu_type1.ko.xz
/lib/modules/4.13.9-200.fc26.x86_64/kernel/drivers/vfio/vfio_virqfd.ko.xz
/lib/modules/4.13.9-200.fc26.x86_64/kernel/drivers/vfio/pci
/lib/modules/4.13.9-200.fc26.x86_64/kernel/drivers/vfio/pci/vfio-pci.ko.xz
/lib/modules/4.13.9-200.fc26.x86_64/kernel/drivers/vfio/vfio.ko.xz
/lib/modules/4.13.9-200.fc26.x86_64/kernel/drivers/vfio/mdev
/lib/modules/4.13.9-200.fc26.x86_64/kernel/drivers/vfio/mdev/vfio_mdev.ko.xz
/lib/modules/4.13.9-200.fc26.x86_64/kernel/drivers/vfio/mdev/mdev.ko.xz

It seems like that line from journalctl was logged before the root partition was unlocked and mounted, but shouldn’t have dracut copied the module to the boot partition, so it can be loaded before the LUKS container is unlocked?

EDIT2: After running lsinitrd | less I noticed that the vfio-pci kernel module was indeed missing from the initram-image, so I ran dracut again, now adding --add-drivers vfio-pci to the command. I verified the process by running lsinitrd | less again, and now the above mentioned directories are copied over too.

However, when I try rebooting with the card installed the system freezes at a black screen after I entered the password at the LUKS prompt. journalctl shows that vfio-pci now claims the device IDs, but there is still something not working right.

2

What’s the second GPU your leaving for the host?

3

That’t the integrated graphics of my CPU.

4

Is the igpu the one you are booting with? If not change igpu to that one

5

Ok, some things come to mind, that bit me last time I set this up:

  1. Does grub.cfg in /boot have the intel_iommu=on line on the kernel? If not run grub2-mkconfig -o <path_to_grub.cfg>
  2. Check lspci -vv output if the driver is really in use. Should be something like this:
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/580] (rev cf) (prog-if 00 [VGA controller])
        [...]
	Kernel driver in use: vfio-pci
	Kernel modules: amdgpu
  1. Check in the BIOS if you’re booting with the IGPU.
6

Remove this from the following line:

You don’t want vfio-pci on your processor.

1 Like
7

I checked that and it does. When I press ‘e’ during boot to inspect the kernel parameters, I can confirm it’s there.

The problem is that I can’t boot completely to check that.

I went into the BIOS/UEFI and changed that option to the IGPU, but that had no visible effect by itself.

However, what did somehow work was to unplug the monitor that is connected to the Nvidia card. Now I was able to boot normally again. Now I am not sure if that is a temporary problem or something else.

Since it was GDM (or maybe Xorg) that wasn’t able to start up during boot and I saw messages about nouveau from Xorg in journalctl, I assume that it wants to load nouveau since the “first” monitor I had before (while using only the Nvidia card alone) was connected over the Nvidia card. Maybe changing some configuration there (e.g. known monitors or order of monitors) might help?

I don’t think that 8086:0c01 is the processor, that’s most likely just a falsely detected label. After all, I don’t have a Xeon processors installed. The end of label also states “PCI Express x16 Controller”. But would it even work, if I don’t claim all devices in the IOMMU group?

Either way, I will continue following the tutorial now to see if it actually works. :thinking:

8

https://pci-ids.ucw.cz/read/PC/8086/0c01
It’s the PCI express controller on your processor, specifically. Definitely not your GPU, to be sure. You only want vfio-pci on your GPU, and that has two parts, the graphics, and the audio, which are the two other devices there.

9

Ah so there is a “feature” of modern cards that they will defer being the primary GPU if they detect no monitor is attached

If you get any output at all on the Nvidia card something has gone wrong because that output is meant to be on your igpu

10

Before I unplugged the monitor, I did get systemd messages there until I hit the LUKS prompt. From there the systemd messages froze on the monitor connected to the Nvidia card and the monitor connected to the IGPU continued with the Fedora boot logo until it was “full” and the screen went black.

11

Looks like initrd is initializing your GPU. You could try blacklisting the nouveau driver in the initrd using this in grub.cfg:

GRUB_CMDLINE_LINUX="rd.driver.blacklist=nouveau modprobe.blacklist=nouveau ...

It might be Wayland behaving badly. If blacklisting doesn’t work try turning Xorg on by modifying /etc/gdm/custom.conf:

[daemon]
# Uncoment the line below to force the login screen to use Xorg
WaylandEnable=false
1 Like
12

@orbit: I will try that when I need to reboot the next time :wink:

Since the GPU was claimed by the vfio-pci driver, I opened up virt-manager to test if I can pass-through the GPU now. Via the GUI I added both devices from the Nvidia card (adding the PCIe controller gave me an error). Now the VM still starts and I can hear the fans of the GPU doing a power cycle but the (now reconnected) monitor remains dark with no signal.

13

Something else I had to do was use the default VM options (for Spice and QXL) to install and set up the VM. Then I could install the drivers for my card and was able to remove the default VM graphics and continue using the passthrough card. Depending on your VM’s OS, your mileage may vary.

2 Likes
14

Okay, something went wrong with my VM configuration.

I created a new VM and did the same steps like before to add the GPU and now I got a signal on the monitor connected to the Nvidia card.

Let’s hope this stays like that and I can complete the Windows setup :smiley:

Edit: So the signal is indeed still there, but my Windows 7 install ISO (which worked fine on the previous setup) gets stuck at “Starting Windows”, right before the little animation appears.

15

Good to hear!

I can suggest trying different virtual video cards instead of QXL and SATA/IDE controllers. You can switch both of them to virtio/QXL later on. Another thing to keep in mind is to create a UEFI VM instead of a legacy one.

16

So, after installing some updates and drivers, I checked Windows device manager and the GPU was detected, however it was listed as stopped or inactive.

After that, I shut the VM down and removed the two devices in virt-manager (Spice and QXL), so the VM can use the Nvidia card now, but when I boot the VM again, the monitor connected to the Nvidia card stays dark without a signal.

When I add the two devices back, the VM will start booting again but mystically crash after the Windows booting animation, right before the blue welcome screen.

I installed it in BIOS mode, so no UEFI. Is UEFI absolutely necessary? If so, can I somehow convert it?

17

GPU passthrough is broken without UEFI more often than not. You will have to create a new VM with OVMF and reattach storage devices if you want to keep them. There’s no way to convert an existing VM and you will probably have to reinstall Windows.

18

So I opened virt-manager and created a fresh VM with default settings (except for changing the drives to SATA and of course changing it to UEFI) and installed Windows 7 from an installation DVD iso. Since I didn’t attach the GPU yet, I wasn’t able to install the Nvidia driver yet. So I rebooted the VM with the Nvidia card (and its audio device) attached. After that I ran the Nvidia setup assistant to install the driver and rebooted once again. Everything still works up to here.

Now I removed the SPICE and QXL devices, added a USB keyboard and mouse through the standard USB-forwarding and rebooted the machine. It is successful in the way, that the VM actually boots and starts loading windows (I can see the colored dots animation). But after a few seconds into that animations everything seems to freeze and the animation stops as well. I am not sure if it’s just the graphics and the rest of Windows is up, but it doesn’t really matter at this point, since I can’t use the OS.

How can I diagnose or fix this problem?

EDIT: I hit F7 during the Windows boot to start safe boot, but that didn’t make a difference either. I ran another mode which generated a TXT file called ntbtlog.txt that lists all the loaded and not loaded drivers, but those don’t tell me much.

If I grep the file for “nvidia” I get lines that read something like this:

Did not load driver @oem2.inf,%nvidia_dev.1184%;NVIDIA GeForce GTX 770
Did not load driver @oem3.inf,%nvidia.devicedesc%;NVIDIA High Definition Audio
19

I’m wondering if you might be experiencing something with Nvidia’s drivers not loading because it detects that you’re using a VM. If you want more info on that, they generally refer to it as Error 43.
I don’t have an Nvidia card to test with, but I found the following potential fix, if that’s what you’re experiencing:

To solve this there is a kvm option ‘hv_vendor_id=Nvidia43FIX’ and bam everything works like a charm again.
So my whole -cpu line looks like:

-cpu host,kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_vendor_id=Nvidia43FIX

It was tucked in the comments here:
https://bufferoverflow.io/gpu-passthrough/

You can edit the xml of your VM using:

nano virsh edit [vmname]

To save and close in nano, use ‘CTRL+X’, ‘Y’ to save changes, ‘Enter’ to confirm the name.

1 Like
20

You will also need to set the XMLS declaration at the top of the virsh edit or it will kick it back for a malformed XML.