Trouble with GPU passthrough on Fedora 36

Communities Gaming
, ,
1

I should first mention that I have two NVIDIA GPU Cards, so I want to keep the NVIDIA drivers installed. I had a GPU passthrough system setup on ubuntu 23.04, and am attempting to recreate the same gaming VM for windows games on Fedora 36.

Currently running

lspci -nnk

shows:

28:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116 [GeForce GTX 1660 Ti] [10de:2182] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3750]
Kernel driver in use: vfio-pci
Kernel modules: nouveau, nvidia_drm, nvidia
28:00.1 Audio device [0403]: NVIDIA Corporation TU116 High Definition Audio Controller [10de:1aeb] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3750]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
28:00.2 USB controller [0c03]: NVIDIA Corporation TU116 USB 3.1 Host Controller [10de:1aec] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3750]
Kernel driver in use: xhci_hcd
28:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 USB Type-C UCSI Controller [10de:1aed] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3750]
Kernel driver in use: vfio-pci
Kernel modules: i2c_nvidia_gpu

For some reason only one of the PCI devies is grabbed by the nvidia driver?

I have created the following files with their respective contents as follows:

/etc/modules-load.d/vfio.conf

vfio
vfio-pci
vfio_virqfd
vfio_iommu_type1

/etc/dracut.conf.d/vfio.conf

add_dracutmodules+=" vfio "
force_drivers+=" vfio vfio-pci vfio_iommu_type1 "

/etc/modprobe.d/nvidia.conf:

softdep nouveau pre: vfio-pci
softdep nvidia pre: vfio-pci
softdep nvidia* pre: vfio-pci
softdep snd_hda_intel pre:vfio-pci
softdep i2c_nvidia_gpu pre: vfio-pci
softdep nvidia-gpu pre: vfio-pci
softdep xhci_hcd pre: vfio vfio_pci vfio-pci
softdep xhci_pci pre: vfio vfio_pci vfio-pci
softdep pcieport pre: vfio vfio_pci vfio-pci
softdep nvidia_drm pre: vfio vfio_pci vfio-pci

as well as /etc/default/grub containing this line:

GRUB_CMDLINE_LINUX=“rd.driver.pre=vfio-pci rd.driver.blacklist=nouveau modprobe.blacklist=nouveau rhgb quiet video=efifb:off amd_iommu=on kvm.ignore_msrs=1 vfio-pci.ids=10de:2182,10de:1aeb,10de:1aec,10de:1aed”

I have been updating intrafms with
sudo dracut -f --kver $(uname -r)

as well as updating grub via:
sudo grub2-mkconfig -o /etc/grub2-efi.cfg

I have literally no idea what i am doing. This was relatively straight forward on ubuntu, and maybe I am doing something wrong given this is fedora. I’m not sure. Any help would be greatly appreciated. :slight_smile:

2

Fedora 36 is now EOL since May 16, 2023.

I am unsure if your problems stem from a non-updating distro. Any particular reason you are staying on 36? Are you using newer binaries/packages sourced elsewhere?

3

Nothing works better than mutter rounded. I mostly use flatpaks for apps so I am not concerned about old software, and as far as security goes linux is significantly more secure in general so if there is a flaw found in fedora 36, I’ll consider updating. I know there are alternatives to mutter-rounded, as far as the gnome extension is concerned, but it has weird artifacts with its shadow when using certain gnome window effects from compwiz. So I am hopefully waiting until those issues are resolved.

4

I feel like you are approaching the problem wrong.

You are bound to have complex and compounded issues with old unmaintained, insecure, Fedora 36 with an up to date flatpak on top. Maybe try using Fedora 38 (37 is still supported, btw) and see if you could downgrade to rounded mutter instead?

5

I understand that perspective, which is why I original wanted to switch to Ubuntu 22.10 LTS, which still uses mutter 42 and has 3ish years of updates left, as well as security updates for an additional five years with Ubuntu Pro (free for personal use). However, I had issues compiling mutter-rounded. I am not experienced enough with programing to begin to understand how to fix, nor was i able to get older versions of mutter working on newer gnome versions. Fedora 38 ships with gnome 44, and it doesn’t play nice (or at least I couldn’t get it to) with mutter 42 (even the non patched mutter). That’s actually the reason the developer abandoned the mutter-rounded project and switched to an extension, patching each version with significant changes became increasingly more complex. I don’t intend on staying on fedora 36 forever, just until the extension is more mature. Fedora 36 just barely is EOL, so its not really insecure YET. Even so, linux is very secure as is. I’m not worried about it.

6

For people who come back and see this thread with similar problems, I wasn’t able to force vfio-pci to load before the xhci-hcd driver using modprod.d configurations.

I also ended up trying to over-ride the pci drivers during boot with a script using wendel’s tutorial for fedora 33. (Forum wont let me link the url sorry)

Those didn’t work for me. I ended up using the newer Linux kernel (6.2.13 & up) feature to just unbind the driver in userspace after system boot as root with crontab. Any user account crontab will not work as you will need root permissions to unbind and bind device drivers.

sudo EDITOR=nano crontab -e

with the following line inside:

@reboot sh -c "echo -n \"0000:28:00.2\" > /sys/bus/pci/drivers/xhci_hcd/unbind && echo -n \"0000:28:00.2\" > /sys/bus/pci/drivers/vfio-pci/bind"

This is not ideal as if I install newer hardware or anything like that, these pci-ids could change. I am still trying to bind them during boot. I will update this thread If I ever find a solution.

1 Like