Ryzen GPU Passthrough Setup Guide: Fedora 26 + Windows Gaming on Linux | Level One Techs

dr_ew · September 10, 2017, 2:21am

fedora26
kernel 4.12.9-300.fc26.x86_64

[root@localhost ~]# lsinitrd | grep vfio
-rw-r–r-- 1 root root 41 Aug 11 08:19 etc/modprobe.d/vfio.conf
[root@localhost ~]# find /var |grep vfio
[root@localhost ~]#

I can grep on something else in lsinitrd if it helps. its a huge output

I’m basically just following your guide; brand new fedora install and all. no prior experience with vfio / gpu passthrough

wendell · September 10, 2017, 2:30am

Try find / grep vfio kernel is slightly newer than the guide

We are looking for the name(s) of the vfio modules to tell dracut to include in the initial ramdisk

dr_ew · September 10, 2017, 2:32am

[root@localhost ~]# find / | grep vfio

Summary

/etc/modprobe.d/vfio.conf
/etc/dracut.conf.d/vfio.conf
find: ‘/run/user/1000/gvfs’: Permission denied
/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio
/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/vfio.ko.xz
/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/mdev
/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/mdev/vfio_mdev.ko.xz
/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/mdev/mdev.ko.xz
/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/pci
/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/pci/vfio-pci.ko.xz
/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/vfio_virqfd.ko.xz
/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/vfio_iommu_type1.ko.xz
/usr/lib/modules/4.11.8-300.fc26.x86_64/kernel/drivers/vfio
/usr/lib/modules/4.11.8-300.fc26.x86_64/kernel/drivers/vfio/vfio.ko.xz
/usr/lib/modules/4.11.8-300.fc26.x86_64/kernel/drivers/vfio/mdev
/usr/lib/modules/4.11.8-300.fc26.x86_64/kernel/drivers/vfio/mdev/vfio_mdev.ko.xz
/usr/lib/modules/4.11.8-300.fc26.x86_64/kernel/drivers/vfio/mdev/mdev.ko.xz
/usr/lib/modules/4.11.8-300.fc26.x86_64/kernel/drivers/vfio/pci
/usr/lib/modules/4.11.8-300.fc26.x86_64/kernel/drivers/vfio/pci/vfio-pci.ko.xz
/usr/lib/modules/4.11.8-300.fc26.x86_64/kernel/drivers/vfio/vfio_virqfd.ko.xz
/usr/lib/modules/4.11.8-300.fc26.x86_64/kernel/drivers/vfio/vfio_iommu_type1.ko.xz
/usr/src/kernels/4.12.9-300.fc26.x86_64/samples/vfio-mdev
/usr/src/kernels/4.12.9-300.fc26.x86_64/samples/vfio-mdev/Makefile
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/platform
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/platform/Makefile
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/platform/Kconfig
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/platform/reset
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/platform/reset/Makefile
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/platform/reset/Kconfig
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/Makefile
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/Kconfig
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/mdev
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/mdev/Makefile
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/mdev/Kconfig
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/pci
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/pci/Makefile
/usr/src/kernels/4.12.9-300.fc26.x86_64/drivers/vfio/pci/Kconfig
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/uapi/linux/vfio.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/uapi/linux/vfio_ccw.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/pci.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/virqfd.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/mdev.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/mdev
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/mdev/device.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/iommu
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/iommu/type1.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/pci
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/pci/vga.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/pci/intx.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/pci/mmap.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/vfio/pci/igd.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/config/kvm/vfio.h
/usr/src/kernels/4.12.9-300.fc26.x86_64/include/linux/vfio.h
/usr/include/linux/vfio.h
/usr/include/linux/vfio_ccw.h
/dev/vfio
/dev/vfio/vfio

True, about the kernel. I can downgrade if that will be easier.

wendell · September 10, 2017, 2:43am

There it is

/usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/vfio_iommu_type1.ko.xz

Try dracut --add-drivers “vfio_iommu_type1 vfio vfio-pci” ? Then the dracut -f command. Maybe a typo in your vfio.conf ?

dr_ew · September 10, 2017, 2:45am

cant be a typo, used copy paste

[root@localhost ~]# dracut --add-drivers “vfio_iommu_type1 vfio vfio-pci”
dracut: Cannot find module directory /lib/modules/vfio-pci”/
dracut: and --no-kernel was not specified

EDIT: editing my foot as we speak. Typed out the "add_drivers… " and it worked fine. must be a copy /paste error with vim??

updating grub2 now. fingers crossed

wendell · September 10, 2017, 2:50am

Ls -l /lib/modules ?

The quotation marks sometimes get messed up on copy paste.

usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/pci/vfio-pci.ko.xz

There is vfio-pci so … maybe the /lib/modules is the wrong symlink or something? Weird it says it can’t find it

dr_ew · September 10, 2017, 3:00am

good news: the dracut command worked after i typed it (stupid c/p error)

bad news is that the card is still using the nvidia drivers after getting the dracut command to work

im wondering if this has to do with the fact that i blacklisted nouveau when i installed the nvidia drivers: https://www.if-not-true-then-false.com/2015/fedora-nvidia-guide/
performance on nouveau was unbearable

wendell · September 10, 2017, 3:01am

Html quotes are not simple quotes they are utf. You can edit your .vimrc to translate tho

wendell · September 10, 2017, 3:03am

So do you see the drivers on lsinitrd ? If so the vfio conf isn’t being read in time. You can remove the Nvidia driver from your initial ramdisk or add the PCi IDs to the kernel line so your module config is right on the kernel line. Don’t forget to retina grub2 if you update the kernel command line

dr_ew · September 10, 2017, 3:10am

whats the syntax for that? I dont think i want to block the nvidia drivers because my gtx 1030 (primary / fedora gpu) wouldnt see them then.

[root@localhost ~]# lsinitrd | grep vfio
-rw-r–r-- 1 root root 41 Aug 11 08:19 etc/modprobe.d/vfio.conf
drwxr-xr-x 3 root root 0 Aug 11 08:19 usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/pci
-rw-r–r-- 1 root root 21116 Aug 11 08:19 usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/pci/vfio-pci.ko.xz
-rw-r–r-- 1 root root 9680 Aug 11 08:19 usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/vfio_iommu_type1.ko.xz
-rw-r–r-- 1 root root 10736 Aug 11 08:19 usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/vfio.ko.xz
-rw-r–r-- 1 root root 2884 Aug 11 08:19 usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/vfio_virqfd.ko.xz

wendell · September 10, 2017, 3:11am

In afk for a bit but man dracut to remove Nvidia it’s fine because it’ll load later in boot

dr_ew · September 10, 2017, 3:39am

i tried 2 methods:
#1 omitting the driver using:
dracut -f --omit-drivers “nvidia” --kver `uname -r`
then checked the driver in use for with lspci -vnn, still saying nvidia

#2 blacklisting driver by
vim /etc/default/grub
adding: rd.driver.blacklist=nvidia
then checked the driver in use for with lspci -vnn, still saying nvidia

to recap: I cant get my 2nd GPU to use the vfio drivers, keeps loading up the nvidia drivers. i do see the vfio drivers in lsinitrd though…

thanks for your help. where can i donate to your channel? regardless of getting this working, im learning a ton about boot procedure (stuff i didnt learn during RHCSA training too :D)

wendell · September 10, 2017, 5:48am

Does lsinitrd have Nvidia still? If so that’s what we gotta fix. If it doesn’t something is up with your vfio config because there is no way vfio can’t not grab the device if Nvidia is not part of your initrd. ? Make sense?

I don’t actually know what in doing I just reason my way out of all these situations hence mixing up mentally lib and var earlier. Been doing lots of manual labor today so that always makes the old noodle a bit foggy.

Anyway, if Nvidia is not part of initrd what’s your vfio conf, the PCi IDs of your card and your grub command line?

Hit e to edit at boot and confirm your kernel line is changing when you change default and run grub2-mkconfig ?

Gixxa · September 10, 2017, 11:41am

Hi so I just setup my Threadripper build this is the iommu groups on an ASRock X399 Taichi:

IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 10 00:19.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1460]
IOMMU Group 10 00:19.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1461]
IOMMU Group 10 00:19.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1462]
IOMMU Group 10 00:19.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1463]
IOMMU Group 10 00:19.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1464]
IOMMU Group 10 00:19.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1465]
IOMMU Group 10 00:19.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1466]
IOMMU Group 10 00:19.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1467]
IOMMU Group 11 01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ba] (rev 02)
IOMMU Group 11 01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b6] (rev 02)
IOMMU Group 11 01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b1] (rev 02)
IOMMU Group 11 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 11 02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 11 02:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 11 02:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 11 02:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 11 04:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 11 05:00.0 Network controller [0280]: Intel Corporation Device [8086:24fb] (rev 10)
IOMMU Group 11 06:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 12 08:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 660 Ti] [10de:1183] (rev a1)
IOMMU Group 12 08:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
IOMMU Group 13 40:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 14 40:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 15 40:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 16 40:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 17 40:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 18 40:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 18 40:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 18 42:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]
IOMMU Group 18 42:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1456]
IOMMU Group 18 42:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB3 Host Controller [1022:145c]
IOMMU Group 19 40:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 19 40:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 19 43:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]
IOMMU Group 19 43:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 20 41:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
IOMMU Group 20 41:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
IOMMU Group 2 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 3 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 4 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 5 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 6 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 6 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 6 09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]
IOMMU Group 6 09:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1456]
IOMMU Group 6 09:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB3 Host Controller [1022:145c]
IOMMU Group 7 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 7 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 7 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]
IOMMU Group 7 0a:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 8 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 8 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 9 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1460]
IOMMU Group 9 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1461]
IOMMU Group 9 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1462]
IOMMU Group 9 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1463]
IOMMU Group 9 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1464]
IOMMU Group 9 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1465]
IOMMU Group 9 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1466]
IOMMU Group 9 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1467]

Are these any good?

My 660 Ti will be replaced by a 1030 in a few weeks, as I don’t need all the power of the 660 Ti within Linux.

wendell · September 10, 2017, 2:20pm

It’s decent. Vga is isolated. The nics are not not are some other peripherals but it should be fine. Thing to try is a bunch of nvme devices to see if they group with the pcie slots I’ll try it on the fatality when I get a sec

dr_ew · September 10, 2017, 5:45pm

confirmed, its writing /etc/default/grub contents correctly

Yeah, if im reading the output correctly, it does. Interestingly enough, you can see the omit statement at the beginning too. So i’ve tried omitting via dracut (on its own), blacklisting via /etc/default/grub (on its own), and both simultaneously.

[root@localhost ~]# lsinitrd |grep nvidia

<-- Large Output Summary Below (click the carrot to descend)

Arguments: drwxr-xr-x 12 root root drwxr-xr-x 4 root root drwxr-xr-x 2 root root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root drwxr-xr-x 2 root root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root drwxr-xr-x 4 root root drwxr-xr-x 2 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root drwxr-xr-x 2 root root lrwxrwxrwx 1 root root -rw-r–r-- 1 root lrwxrwxrwx 1 root root -rw-r–r-- 1 root lrwxrwxrwx 1 root root -rw-r–r-- 1 root lrwxrwxrwx 1 root root -rw-r–r-- 1 root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root drwxr-xr-x 4 root root drwxr-xr-x 2 root root lrwxrwxrwx 1 root root -rw-r–r-- 1 root -rw-r–r-- 1 root drwxr-xr-x 2 root root lrwxrwxrwx 1 root root -rw-r–r-- 1 root lrwxrwxrwx 1 root root -rw-r–r-- 1 root lrwxrwxrwx 1 root root -rw-r–r-- 1 root lrwxrwxrwx 1 root root -rw-r–r-- 1 root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root drwxr-xr-x 5 root root drwxr-xr-x 2 root root -rw-r–r-- 1 root -rw-r–r-- 1 root drwxr-xr-x 2 root root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root lrwxrwxrwx 1 root root -rw-r–r-- 1 root drwxr-xr-x 2 root root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root drwxr-xr-x 4 root root drwxr-xr-x 2 root root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root drwxr-xr-x 2 root root lrwxrwxrwx 1 root root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root lrwxrwxrwx 1 root root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 2 root -rw-r–r-- 1 root -rw-r–r-- 2 root -rw-r–r-- 1 root drwxr-xr-x 6 root root large output -f --omit-drivers ‘nvidia’ --kver '4.12.9-300.fc26.x86_64’
0 Aug 11 08:19 usr/lib/firmware/nvidia
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm200
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm200/acr
root 832 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/acr/bl.bin
root 10144 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/acr/ucode_load.bin
root 1440 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/acr/ucode_unload.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm200/gr
root 576 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/fecs_bl.bin
root 1968 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/fecs_data.bin
root 16271 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/fecs_inst.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/fecs_sig.bin
root 576 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/gpccs_bl.bin
root 2056 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/gpccs_data.bin
root 9768 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/gpccs_inst.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/gpccs_sig.bin
root 7616 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/sw_bundle_init.bin
root 5592 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/sw_ctx.bin
root 10800 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/sw_method_init.bin
root 1440 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/sw_nonctx.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm204
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/acr
59 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/acr/bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/acr/bl.bin
67 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/acr/ucode_load.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/acr/ucode_load.bin
69 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/acr/ucode_unload.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/acr/ucode_unload.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/gr
63 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/gr/fecs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/fecs_bl.bin
root 1968 Jun 22 08:55 usr/lib/firmware/nvidia/gm204/gr/fecs_data.bin
65 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/gr/fecs_inst.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/fecs_inst.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gm204/gr/fecs_sig.bin
64 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/gr/gpccs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/gpccs_bl.bin
root 2056 Jun 22 08:55 usr/lib/firmware/nvidia/gm204/gr/gpccs_data.bin
66 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/gr/gpccs_inst.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/gpccs_inst.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gm204/gr/gpccs_sig.bin
70 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/gr/sw_bundle_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/sw_bundle_init.bin
62 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/gr/sw_ctx.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/sw_ctx.bin
70 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/gr/sw_method_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/sw_method_init.bin
65 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/gr/sw_nonctx.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/sw_nonctx.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm206
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/acr
59 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/acr/bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/acr/bl.bin
root 10144 Jun 22 08:55 usr/lib/firmware/nvidia/gm206/acr/ucode_load.bin
root 1440 Jun 22 08:55 usr/lib/firmware/nvidia/gm206/acr/ucode_unload.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/gr
63 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/gr/fecs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/fecs_bl.bin
root 1968 Jun 22 08:55 usr/lib/firmware/nvidia/gm206/gr/fecs_data.bin
65 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/gr/fecs_inst.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/fecs_inst.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gm206/gr/fecs_sig.bin
64 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/gr/gpccs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/gpccs_bl.bin
root 2056 Jun 22 08:55 usr/lib/firmware/nvidia/gm206/gr/gpccs_data.bin
66 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/gr/gpccs_inst.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/gpccs_inst.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gm206/gr/gpccs_sig.bin
70 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/gr/sw_bundle_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/sw_bundle_init.bin
62 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/gr/sw_ctx.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/sw_ctx.bin
70 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/gr/sw_method_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/sw_method_init.bin
65 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/gr/sw_nonctx.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/sw_nonctx.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm20b
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm20b/acr
root 832 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/acr/bl.bin
root 18592 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/acr/ucode_load.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm20b/gr
root 576 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/gr/fecs_bl.bin
root 1964 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/gr/fecs_data.bin
root 17021 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/gr/fecs_inst.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/gr/fecs_sig.bin
root 2068 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/gr/gpccs_data.bin
root 9964 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/gr/gpccs_inst.bin
root 7616 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/gr/sw_bundle_init.bin
root 5448 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/gr/sw_ctx.bin
70 Aug 11 08:19 usr/lib/firmware/nvidia/gm20b/gr/sw_method_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/sw_method_init.bin
root 1432 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/gr/sw_nonctx.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm20b/pmu
root 652 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/pmu/desc.bin
root 47872 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/pmu/image.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/pmu/sig.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gp100
0 Aug 11 08:19 usr/lib/firmware/nvidia/gp100/acr
root 832 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/acr/bl.bin
root 9632 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/acr/ucode_load.bin
root 1440 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/acr/ucode_unload.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gp100/gr
63 Aug 11 08:19 usr/lib/firmware/nvidia/gp100/gr/fecs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/fecs_bl.bin
root 2028 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/fecs_data.bin
root 20955 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/fecs_inst.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/fecs_sig.bin
64 Aug 11 08:19 usr/lib/firmware/nvidia/gp100/gr/gpccs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/gpccs_bl.bin
root 2080 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/gpccs_data.bin
root 12458 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/gpccs_inst.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/gpccs_sig.bin
root 0 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/sw_bundle_init.bin
root 6240 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/sw_ctx.bin
root 0 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/sw_method_init.bin
root 2248 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/sw_nonctx.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gp102alhost ~]# lsinitrd |grep nvidia
below (click the carrot to descend)

drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp102/acr
-rw-r–r-- 1 root root 1280 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/acr/bl.bin
-rw-r–r-- 1 root root 17152 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/acr/ucode_load.bin
-rw-r–r-- 1 root root 3328 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/acr/ucode_unload.bin
-rw-r–r-- 1 root root 1280 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/acr/unload_bl.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp102/gr
lrwxrwxrwx 1 root root 63 Aug 11 08:19 usr/lib/firmware/nvidia/gp102/gr/fecs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/fecs_bl.bin
-rw-r–r-- 1 root root 2256 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/fecs_data.bin
-rw-r–r-- 1 root root 20927 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/fecs_inst.bin
-rw-r–r-- 1 root root 192 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/fecs_sig.bin
lrwxrwxrwx 1 root root 64 Aug 11 08:19 usr/lib/firmware/nvidia/gp102/gr/gpccs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/gpccs_bl.bin
-rw-r–r-- 1 root root 1832 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/gpccs_data.bin
-rw-r–r-- 2 root root 0 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/gpccs_inst.bin
-rw-r–r-- 1 root root 192 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/gpccs_sig.bin
-rw-r–r-- 1 root root 7680 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/sw_bundle_init.bin
-rw-r–r-- 1 root root 6216 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/sw_ctx.bin
-rw-r–r-- 1 root root 12288 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/sw_method_init.bin
-rw-r–r-- 1 root root 2496 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/sw_nonctx.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp102/nvdec
-rw-r–r-- 1 root root 3328 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/nvdec/scrubber.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp102/sec2
-rw-r–r-- 1 root root 656 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/sec2/desc.bin
-rw-r–r-- 1 root root 99072 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/sec2/image.bin
-rw-r–r-- 1 root root 192 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/sec2/sig.bin
drwxr-xr-x 6 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp104
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/acr
lrwxrwxrwx 1 root root 59 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/acr/bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/bl.bin
lrwxrwxrwx 1 root root 67 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/acr/ucode_load.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/ucode_load.bin
lrwxrwxrwx 1 root root 69 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/acr/ucode_unload.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/ucode_unload.bin
lrwxrwxrwx 1 root root 66 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/acr/unload_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/unload_bl.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/gr
lrwxrwxrwx 1 root root 63 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/gr/fecs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/fecs_bl.bin
-rw-r–r-- 1 root root 2576 Jun 22 08:55 usr/lib/firmware/nvidia/gp104/gr/fecs_data.bin
-rw-r–r-- 1 root root 22760 Jun 22 08:55 usr/lib/firmware/nvidia/gp104/gr/fecs_inst.bin
-rw-r–r-- 1 root root 192 Jun 22 08:55 usr/lib/firmware/nvidia/gp104/gr/fecs_sig.bin
lrwxrwxrwx 1 root root 64 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/gr/gpccs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/gpccs_bl.bin
-rw-r–r-- 1 root root 1832 Jun 22 08:55 usr/lib/firmware/nvidia/gp104/gr/gpccs_data.bin
-rw-r–r-- 2 root root 13307 Jun 22 08:55 usr/lib/firmware/nvidia/gp104/gr/gpccs_inst.bin
-rw-r–r-- 1 root root 192 Jun 22 08:55 usr/lib/firmware/nvidia/gp104/gr/gpccs_sig.bin
lrwxrwxrwx 1 root root 70 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/gr/sw_bundle_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_bundle_init.bin
lrwxrwxrwx 1 root root 62 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/gr/sw_ctx.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_ctx.bin
lrwxrwxrwx 1 root root 70 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/gr/sw_method_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_method_init.bin
lrwxrwxrwx 1 root root 65 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/gr/sw_nonctx.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_nonctx.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/nvdec
lrwxrwxrwx 1 root root 67 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/nvdec/scrubber.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/nvdec/scrubber.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/sec2
lrwxrwxrwx 1 root root 62 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/sec2/desc.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/sec2/desc.bin
lrwxrwxrwx 1 root root 63 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/sec2/image.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/sec2/image.bin
lrwxrwxrwx 1 root root 61 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/sec2/sig.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/sec2/sig.bin
drwxr-xr-x 6 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp106
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/acr
lrwxrwxrwx 1 root root 59 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/acr/bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/bl.bin
lrwxrwxrwx 1 root root 67 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/acr/ucode_load.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/ucode_load.bin
lrwxrwxrwx 1 root root 69 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/acr/ucode_unload.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/ucode_unload.bin
lrwxrwxrwx 1 root root 66 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/acr/unload_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/unload_bl.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/gr
lrwxrwxrwx 1 root root 63 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/gr/fecs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/fecs_bl.bin
-rw-r–r-- 1 root root 2256 Jun 22 08:55 usr/lib/firmware/nvidia/gp106/gr/fecs_data.bin
lrwxrwxrwx 1 root root 65 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/gr/fecs_inst.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/fecs_inst.bin
-rw-r–r-- 1 root root 192 Jun 22 08:55 usr/lib/firmware/nvidia/gp106/gr/fecs_sig.bin
lrwxrwxrwx 1 root root 64 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/gr/gpccs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/gpccs_bl.bin
-rw-r–r-- 1 root root 1832 Jun 22 08:55 usr/lib/firmware/nvidia/gp106/gr/gpccs_data.bin
lrwxrwxrwx 1 root root 66 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/gr/gpccs_inst.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/gpccs_inst.bin
-rw-r–r-- 1 root root 192 Jun 22 08:55 usr/lib/firmware/nvidia/gp106/gr/gpccs_sig.bin
lrwxrwxrwx 1 root root 70 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/gr/sw_bundle_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_bundle_init.bin
lrwxrwxrwx 1 root root 62 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/gr/sw_ctx.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_ctx.bin
lrwxrwxrwx 1 root root 70 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/gr/sw_method_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_method_init.bin
lrwxrwxrwx 1 root root 65 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/gr/sw_nonctx.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_nonctx.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/nvdec
lrwxrwxrwx 1 root root 67 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/nvdec/scrubber.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/nvdec/scrubber.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/sec2
lrwxrwxrwx 1 root root 62 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/sec2/desc.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/sec2/desc.bin
lrwxrwxrwx 1 root root 63 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/sec2/image.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/sec2/image.bin
lrwxrwxrwx 1 root root 61 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/sec2/sig.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/sec2/sig.bin
drwxr-xr-x 6 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp107
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/acr
lrwxrwxrwx 1 root root 59 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/acr/bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/bl.bin
lrwxrwxrwx 1 root root 67 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/acr/ucode_load.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/ucode_load.bin
lrwxrwxrwx 1 root root 69 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/acr/ucode_unload.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/ucode_unload.bin
lrwxrwxrwx 1 root root 66 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/acr/unload_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/unload_bl.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/gr
-rw-r–r-- 1 root root 576 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/fecs_bl.bin
-rw-r–r-- 1 root root 2756 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/fecs_data.bin
-rw-r–r-- 1 root root 22879 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/fecs_inst.bin
-rw-r–r-- 1 root root 192 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/fecs_sig.bin
-rw-r–r-- 1 root root 576 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/gpccs_bl.bin
-rw-r–r-- 1 root root 2100 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/gpccs_data.bin
-rw-r–r-- 1 root root 12587 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/gpccs_inst.bin
-rw-r–r-- 1 root root 192 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/gpccs_sig.bin
lrwxrwxrwx 1 root root 70 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/gr/sw_bundle_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_bundle_init.bin
-rw-r–r-- 1 root root 6000 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/sw_ctx.bin
lrwxrwxrwx 1 root root 70 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/gr/sw_method_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_method_init.bin
-rw-r–r-- 1 root root 2496 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/sw_nonctx.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/nvdec
lrwxrwxrwx 1 root root 67 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/nvdec/scrubber.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/nvdec/scrubber.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/sec2
lrwxrwxrwx 1 root root 62 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/sec2/desc.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/sec2/desc.bin
lrwxrwxrwx 1 root root 63 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/sec2/image.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/sec2/image.bin
lrwxrwxrwx 1 root root 61 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/sec2/sig.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/sec2/sig.bin
drwxr-xr-x 5 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp10b
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp10b/acr
-rw-r–r-- 1 root root 1280 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/acr/bl.bin
-rw-r–r-- 1 root root 19456 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/acr/ucode_load.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp10b/gr
-rw-r–r-- 1 root root 576 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/fecs_bl.bin
-rw-r–r-- 1 root root 3668 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/fecs_data.bin
-rw-r–r-- 1 root root 22306 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/fecs_inst.bin
-rw-r–r-- 1 root root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/fecs_sig.bin
-rw-r–r-- 1 root root 576 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/gpccs_bl.bin
-rw-r–r-- 1 root root 2340 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/gpccs_data.bin
-rw-r–r-- 1 root root 12921 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/gpccs_inst.bin
-rw-r–r-- 1 root root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/gpccs_sig.bin
-rw-r–r-- 2 root root 7664 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/sw_bundle_init.bin
-rw-r–r-- 1 root root 6036 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/sw_ctx.bin
-rw-r–r-- 2 root root 11928 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/sw_method_init.bin
-rw-r–r-- 1 root root 2744 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/sw_nonctx.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp10b/pmu
-rw-r–r-- 1 root root 652 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/pmu/desc.bin
-rw-r–r-- 1 root root 49408 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/pmu/image.bin
-rw-r–r-- 1 root root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/pmu/sig.bin

Are there multiple drivers i need to block?

In case you still want to see vfio.conf, and grub line:

/etc/default/grub: GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap nomodeset rhgb quiet rd.driver.blacklist=nouveau iommu=1 amd_iommu=on rd.driver.pre=vfio.pci rd.driver.blacklist=nvidia"

/etc/modprobe.d/vfio.conf: options vfio-pci ids=10de:1401,10de:0fba

pci-ids:
21:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401] (rev a1)
21:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fba] (rev a1)

UPDATE: This one guy is suggesting that you need to use pci-stub to prevent the graphics driver from grabbing the gpu. Ill try this tonight (if i have time) and will report back. https://superuser.com/questions/1039315/qemu-with-gpu-pass-through-wont-start

wendell · September 10, 2017, 10:37pm

Yeah that can work too. You can also add the PCi IDs to the kernel command line and try that as well ?

wendell · September 10, 2017, 10:38pm

vfio-pci ? Not dot?

yverner · September 11, 2017, 1:27am

Wendell, thank you for the terrific guide. However, I am running into a few issues - perhaps someone here may be able to lend a hand.

So, I’m trying to passthrough a Fury X on an Asus Prime x370-pro. I’ve managed to successfully configure and apply the vfio driver, which I have been able to verify with lscpi -nk which outputs:

29:00.0 0300: 1002:7300 (rev c8)
        Subsystem: 1002:0b36
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu
29:00.1 0403: 1002:aae8
        Subsystem: 1002:aae8
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

My issue is that I do not receive any output from the VM that this gpu is passed through to. Am I missing something simple in the way in which I need to configure the virtual machine? I have removed all of the virtual display hw as well as the display console. Any help would be appreciated, many thanks.

dr_ew · September 11, 2017, 1:50am

oh my god. that was it.

21:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401] (rev a1) (prog-if 00 [VGA controller])
Subsystem: eVga.com. Corp. Device [3842:3967]
Flags: fast devsel, IRQ 11
Memory at f4000000 (32-bit, non-prefetchable) [disabled] [size=16M]
Memory at e0000000 (64-bit, prefetchable) [disabled] [size=256M]
Memory at f0000000 (64-bit, prefetchable) [disabled] [size=32M]
I/O ports at d000 [disabled] [size=128]
Expansion ROM at f5000000 [disabled] [size=512K]
Capabilities:
Kernel driver in use: vfio-pci
Kernel modules: nouveau, nvidia_drm, nvidia

-> -> ->