Trouble with graphics card passthrough in Fedora 36

Software & Operating Systems VFIO
, ,
1

Have been scratching my head trying to pass through a graphics card to a VM.

My goal:
I have some Windows only applications I want to run, and would prefer to run the system in a VM. At least one of the applications can be GPU intensive, hence why I want to pass through a card to the VM.

I have installed KVM, QEMU and Virt-Manager.

Relevant hardware I have:
Motherboard: ASUS ROG Strix X570-F
CPU: AMD Ryzen 9 3950X
GPU 1: XFX Radeon RX 6800 XT Speedster MERC 319
GPU 2: ASUS GeForce RTX 2070 DUAL EVO V2 (intended for VM)

Currently my system is Fedora 36 with KDE.

I have tried several guides, but ran into some distribution (application) differences that didn’t “translate” to something that worked with what is available by default in Fedora. Could also be due to my inexperience.
So far my most successful attempt has been a guide wendell wrote. I also applied the corrections mentioned in the thread. Looking around I also came across a video on YouTube that is almost identical and confirms the steps at least worked in Fedora 35.

However, when running dracut -fv I get the following output:

dracut: Executing: /usr/bin/dracut -fv
dracut: dracut module ‘busybox’ will not be installed, because command ‘busybox’ could not be found!
dracut: dracut module ‘rngd’ will not be installed, because command ‘rngd’ could not be found!
dracut: dracut module ‘network-wicked’ will not be installed, because command ‘wicked’ could not be found!
dracut: dracut module ‘tpm2-tss’ will not be installed, because command ‘tpm2’ could not be found!
dracut: dracut module ‘nvmf’ will not be installed, because command ‘nvme’ could not be found!
dracut: dracut module ‘biosdevname’ will not be installed, because command ‘biosdevname’ could not be found!
dracut: dracut module ‘busybox’ will not be installed, because command ‘busybox’ could not be found!
dracut: dracut module ‘rngd’ will not be installed, because command ‘rngd’ could not be found!
dracut: dracut module ‘network-wicked’ will not be installed, because command ‘wicked’ could not be found!
dracut: dracut module ‘tpm2-tss’ will not be installed, because command ‘tpm2’ could not be found!
dracut: dracut module ‘nvmf’ will not be installed, because command ‘nvme’ could not be found!
dracut: *** Including module: bash ***
dracut: *** Including module: systemd ***
dracut: *** Including module: systemd-initrd ***
dracut: *** Including module: systemd-sysusers ***
dracut: *** Including module: nss-softokn ***
dracut: *** Including module: dbus-broker ***
dracut: *** Including module: dbus ***
dracut: *** Including module: i18n ***
dracut: *** Including module: vfio ***
dracut: Cannot install a hook (“/usr/lib/dracut/modules.d/20vfio/vfio-pci-override.sh”) that does not exist.
dracut: Aborting initrd creation.

I have tried the vfio-pci-override.sh script from wendell and the one from YouTube with the same result. They seem to function identically, but tried regardless.

My grub file has the following line: GRUB_CMDLINE_LINUX="quiet amd_iommu=on rd.driver.pre=vfio-pci" Docs from Fedora say that at least amd_iommu=on isn’t necessary, but it is added. Shouldn’t hurt anything? I have made sure IOMMU and SVM is set to ON and not Auto/Disabled in my BIOS.

The nVidia card I want to pass through is located at 0000:0c:00.0 to 0000:0c:00.3.

The vfio-pci-override.sh script located in /usr/sbin/ is made executable and is as follows:

#!/bin/sh
PREREQS=""
DEVS=“0000:0c:00.0 0000:0c:00.1 0000:0c:00.2 0000:0c:00.3”

for DEV in $DEVS; do
echo “vfio-pci” > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci

module-setup.sh is made executable, located in /usr/lib/dracut/modules.d/20vfio/ and is as follows:

#!/usr/bin/bash
check() {
return 0
}
depends() {
return 0
}
install() {
declare moddir=${moddir}
inst_hook pre-udev 00 “$moddir/vfio-pci-override.sh”
}

I also added vfio-pci-override.sh as a symbolic link to the same directory.

vfio.conf has been created in /etc/dracut.conf.d/ and is as follows:

add_dracutmodules+=" vfio "
force_drivers+=" vfio vfio-pci vfio_iommu_type1 "
install_items="/usr/sbin/vfio-pci-override.sh /usr/bin/find /usr/bin/dirname"

Any suggestions as to what failed and how to fix it?

1 Like
2

Never tried this script route before, I do the following on Fedora 36.

Add amd_iommu=on to /etc/default/grub:

GRUB_CMDLINE_LINUX="... amd_iommu=on ..."

Create /etc/modules-load.d/vfio.conf to enable VFIO kernel modules:

vfio
vfio-pci
vfio_virqfd
vfio_iommu_type1

Create /etc/modprobe.d/vfio.conf with PCI ids of devices you would like to passthrough (10de:1b80 and 10de:10f0 are my gtx1080 and its audio controller):

options vfio-pci ids=10de:1b80,10de:10f0

Then rebuild grub and initramfs:

grub2-mkconfig > /etc/grub2-efi.cfg
dracut -fv

Finally reboot system.

1 Like
3

Thank you for the reply!

I removed my previous attempt and performed your steps.

However, I regret to inform that I can’t confirm if it worked…

After the reboot, I were greeted with the following:

Booting ‘Fedora Linux (5.19.11-200.fc36.x86_64) 36 (KDE Plasma)’

error: …/…/grub-core/fs/fshelp.c:257:file ‘/initramfs-5.19.11-200.fc36.x86_64.img’ not found.

Press any key to continue…

If I press a key, it just freezes afterwards. Getting some “no irq handler for vector” lines and nothing more.

Seems like something went really wrong eventhough both grub and dracut didn’t give any errors.

I need to spend some time investigating this. Probably better to create a new thread on this issue.

4

Seems to be a grub issue, I wouldn’t worry about the no irq handler for vector errors they are just motherboard errors common on some boards.

My command to update grub was for EFI based installs perhaps you are using legacy/BIOS?

I would take a look at the Fedora docs for how to update grub config based on your system.

5

Apologies for the late reply.

After some searching and poking around, I concluded with that the kernel were present, but somehow lost the init image. I were able to boot into an older kernel in the boot menu. Used dracut to force build a new one for the latest version. Booted up again as normal after that.

Now, back to the main topic!

Running lspci -nnk I find this in the output:

0c:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU106 [GeForce RTX 2070 Rev. A] [10de:1f07] (rev a1)
Subsystem: ASUSTeK Computer Inc. Device [1043:875c]
Kernel driver in use: vfio-pci
Kernel modules: nouveau

Looks like a success! Did a quick Win10 install on a VM and were successful at installing the nVidia drivers. I suppose it works as it should now.

Thank you so much, @CaptClock !

6

Glad to hear :smiley: