Ryzen GPU Passthrough Setup Guide: Fedora 26 + Windows Gaming on Linux | Level One Techs

wendell · September 10, 2017, 3:03am

So do you see the drivers on lsinitrd ? If so the vfio conf isn’t being read in time. You can remove the Nvidia driver from your initial ramdisk or add the PCi IDs to the kernel line so your module config is right on the kernel line. Don’t forget to retina grub2 if you update the kernel command line

dr_ew · September 10, 2017, 3:10am

whats the syntax for that? I dont think i want to block the nvidia drivers because my gtx 1030 (primary / fedora gpu) wouldnt see them then.

[root@localhost ~]# lsinitrd | grep vfio
-rw-r–r-- 1 root root 41 Aug 11 08:19 etc/modprobe.d/vfio.conf
drwxr-xr-x 3 root root 0 Aug 11 08:19 usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/pci
-rw-r–r-- 1 root root 21116 Aug 11 08:19 usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/pci/vfio-pci.ko.xz
-rw-r–r-- 1 root root 9680 Aug 11 08:19 usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/vfio_iommu_type1.ko.xz
-rw-r–r-- 1 root root 10736 Aug 11 08:19 usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/vfio.ko.xz
-rw-r–r-- 1 root root 2884 Aug 11 08:19 usr/lib/modules/4.12.9-300.fc26.x86_64/kernel/drivers/vfio/vfio_virqfd.ko.xz

wendell · September 10, 2017, 3:11am

In afk for a bit but man dracut to remove Nvidia it’s fine because it’ll load later in boot

dr_ew · September 10, 2017, 3:39am

i tried 2 methods:
#1 omitting the driver using:
dracut -f --omit-drivers “nvidia” --kver `uname -r`
then checked the driver in use for with lspci -vnn, still saying nvidia

#2 blacklisting driver by
vim /etc/default/grub
adding: rd.driver.blacklist=nvidia
then checked the driver in use for with lspci -vnn, still saying nvidia

to recap: I cant get my 2nd GPU to use the vfio drivers, keeps loading up the nvidia drivers. i do see the vfio drivers in lsinitrd though…

thanks for your help. where can i donate to your channel? regardless of getting this working, im learning a ton about boot procedure (stuff i didnt learn during RHCSA training too :D)

wendell · September 10, 2017, 5:48am

Does lsinitrd have Nvidia still? If so that’s what we gotta fix. If it doesn’t something is up with your vfio config because there is no way vfio can’t not grab the device if Nvidia is not part of your initrd. ? Make sense?

I don’t actually know what in doing I just reason my way out of all these situations hence mixing up mentally lib and var earlier. Been doing lots of manual labor today so that always makes the old noodle a bit foggy.

Anyway, if Nvidia is not part of initrd what’s your vfio conf, the PCi IDs of your card and your grub command line?

Hit e to edit at boot and confirm your kernel line is changing when you change default and run grub2-mkconfig ?

Gixxa · September 10, 2017, 11:41am

Hi so I just setup my Threadripper build this is the iommu groups on an ASRock X399 Taichi:

IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 10 00:19.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1460]
IOMMU Group 10 00:19.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1461]
IOMMU Group 10 00:19.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1462]
IOMMU Group 10 00:19.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1463]
IOMMU Group 10 00:19.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1464]
IOMMU Group 10 00:19.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1465]
IOMMU Group 10 00:19.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1466]
IOMMU Group 10 00:19.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1467]
IOMMU Group 11 01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ba] (rev 02)
IOMMU Group 11 01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b6] (rev 02)
IOMMU Group 11 01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b1] (rev 02)
IOMMU Group 11 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 11 02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 11 02:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 11 02:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 11 02:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 11 04:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 11 05:00.0 Network controller [0280]: Intel Corporation Device [8086:24fb] (rev 10)
IOMMU Group 11 06:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 12 08:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 660 Ti] [10de:1183] (rev a1)
IOMMU Group 12 08:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
IOMMU Group 13 40:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 14 40:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 15 40:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 16 40:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 17 40:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 18 40:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 18 40:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 18 42:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]
IOMMU Group 18 42:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1456]
IOMMU Group 18 42:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB3 Host Controller [1022:145c]
IOMMU Group 19 40:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 19 40:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 19 43:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]
IOMMU Group 19 43:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 20 41:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
IOMMU Group 20 41:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
IOMMU Group 2 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 3 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 4 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 5 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 6 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 6 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 6 09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]
IOMMU Group 6 09:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1456]
IOMMU Group 6 09:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB3 Host Controller [1022:145c]
IOMMU Group 7 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 7 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 7 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]
IOMMU Group 7 0a:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 8 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 8 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 9 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1460]
IOMMU Group 9 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1461]
IOMMU Group 9 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1462]
IOMMU Group 9 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1463]
IOMMU Group 9 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1464]
IOMMU Group 9 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1465]
IOMMU Group 9 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1466]
IOMMU Group 9 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1467]

Are these any good?

My 660 Ti will be replaced by a 1030 in a few weeks, as I don’t need all the power of the 660 Ti within Linux.

wendell · September 10, 2017, 2:20pm

It’s decent. Vga is isolated. The nics are not not are some other peripherals but it should be fine. Thing to try is a bunch of nvme devices to see if they group with the pcie slots I’ll try it on the fatality when I get a sec

dr_ew · September 10, 2017, 5:45pm

confirmed, its writing /etc/default/grub contents correctly

Yeah, if im reading the output correctly, it does. Interestingly enough, you can see the omit statement at the beginning too. So i’ve tried omitting via dracut (on its own), blacklisting via /etc/default/grub (on its own), and both simultaneously.

[root@localhost ~]# lsinitrd |grep nvidia

<-- Large Output Summary Below (click the carrot to descend)

Arguments: drwxr-xr-x 12 root root drwxr-xr-x 4 root root drwxr-xr-x 2 root root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root drwxr-xr-x 2 root root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root drwxr-xr-x 4 root root drwxr-xr-x 2 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root drwxr-xr-x 2 root root lrwxrwxrwx 1 root root -rw-r–r-- 1 root lrwxrwxrwx 1 root root -rw-r–r-- 1 root lrwxrwxrwx 1 root root -rw-r–r-- 1 root lrwxrwxrwx 1 root root -rw-r–r-- 1 root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root drwxr-xr-x 4 root root drwxr-xr-x 2 root root lrwxrwxrwx 1 root root -rw-r–r-- 1 root -rw-r–r-- 1 root drwxr-xr-x 2 root root lrwxrwxrwx 1 root root -rw-r–r-- 1 root lrwxrwxrwx 1 root root -rw-r–r-- 1 root lrwxrwxrwx 1 root root -rw-r–r-- 1 root lrwxrwxrwx 1 root root -rw-r–r-- 1 root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root drwxr-xr-x 5 root root drwxr-xr-x 2 root root -rw-r–r-- 1 root -rw-r–r-- 1 root drwxr-xr-x 2 root root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root lrwxrwxrwx 1 root root -rw-r–r-- 1 root drwxr-xr-x 2 root root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root drwxr-xr-x 4 root root drwxr-xr-x 2 root root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root drwxr-xr-x 2 root root lrwxrwxrwx 1 root root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root lrwxrwxrwx 1 root root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 2 root -rw-r–r-- 1 root -rw-r–r-- 2 root -rw-r–r-- 1 root drwxr-xr-x 6 root root large output -f --omit-drivers ‘nvidia’ --kver '4.12.9-300.fc26.x86_64’
0 Aug 11 08:19 usr/lib/firmware/nvidia
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm200
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm200/acr
root 832 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/acr/bl.bin
root 10144 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/acr/ucode_load.bin
root 1440 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/acr/ucode_unload.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm200/gr
root 576 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/fecs_bl.bin
root 1968 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/fecs_data.bin
root 16271 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/fecs_inst.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/fecs_sig.bin
root 576 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/gpccs_bl.bin
root 2056 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/gpccs_data.bin
root 9768 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/gpccs_inst.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/gpccs_sig.bin
root 7616 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/sw_bundle_init.bin
root 5592 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/sw_ctx.bin
root 10800 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/sw_method_init.bin
root 1440 Jun 22 08:55 usr/lib/firmware/nvidia/gm200/gr/sw_nonctx.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm204
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/acr
59 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/acr/bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/acr/bl.bin
67 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/acr/ucode_load.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/acr/ucode_load.bin
69 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/acr/ucode_unload.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/acr/ucode_unload.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/gr
63 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/gr/fecs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/fecs_bl.bin
root 1968 Jun 22 08:55 usr/lib/firmware/nvidia/gm204/gr/fecs_data.bin
65 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/gr/fecs_inst.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/fecs_inst.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gm204/gr/fecs_sig.bin
64 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/gr/gpccs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/gpccs_bl.bin
root 2056 Jun 22 08:55 usr/lib/firmware/nvidia/gm204/gr/gpccs_data.bin
66 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/gr/gpccs_inst.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/gpccs_inst.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gm204/gr/gpccs_sig.bin
70 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/gr/sw_bundle_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/sw_bundle_init.bin
62 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/gr/sw_ctx.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/sw_ctx.bin
70 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/gr/sw_method_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/sw_method_init.bin
65 Aug 11 08:19 usr/lib/firmware/nvidia/gm204/gr/sw_nonctx.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/sw_nonctx.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm206
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/acr
59 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/acr/bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/acr/bl.bin
root 10144 Jun 22 08:55 usr/lib/firmware/nvidia/gm206/acr/ucode_load.bin
root 1440 Jun 22 08:55 usr/lib/firmware/nvidia/gm206/acr/ucode_unload.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/gr
63 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/gr/fecs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/fecs_bl.bin
root 1968 Jun 22 08:55 usr/lib/firmware/nvidia/gm206/gr/fecs_data.bin
65 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/gr/fecs_inst.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/fecs_inst.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gm206/gr/fecs_sig.bin
64 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/gr/gpccs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/gpccs_bl.bin
root 2056 Jun 22 08:55 usr/lib/firmware/nvidia/gm206/gr/gpccs_data.bin
66 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/gr/gpccs_inst.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/gpccs_inst.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gm206/gr/gpccs_sig.bin
70 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/gr/sw_bundle_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/sw_bundle_init.bin
62 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/gr/sw_ctx.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/sw_ctx.bin
70 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/gr/sw_method_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/sw_method_init.bin
65 Aug 11 08:19 usr/lib/firmware/nvidia/gm206/gr/sw_nonctx.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/sw_nonctx.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm20b
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm20b/acr
root 832 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/acr/bl.bin
root 18592 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/acr/ucode_load.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm20b/gr
root 576 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/gr/fecs_bl.bin
root 1964 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/gr/fecs_data.bin
root 17021 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/gr/fecs_inst.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/gr/fecs_sig.bin
root 2068 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/gr/gpccs_data.bin
root 9964 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/gr/gpccs_inst.bin
root 7616 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/gr/sw_bundle_init.bin
root 5448 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/gr/sw_ctx.bin
70 Aug 11 08:19 usr/lib/firmware/nvidia/gm20b/gr/sw_method_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/sw_method_init.bin
root 1432 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/gr/sw_nonctx.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gm20b/pmu
root 652 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/pmu/desc.bin
root 47872 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/pmu/image.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gm20b/pmu/sig.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gp100
0 Aug 11 08:19 usr/lib/firmware/nvidia/gp100/acr
root 832 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/acr/bl.bin
root 9632 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/acr/ucode_load.bin
root 1440 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/acr/ucode_unload.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gp100/gr
63 Aug 11 08:19 usr/lib/firmware/nvidia/gp100/gr/fecs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/fecs_bl.bin
root 2028 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/fecs_data.bin
root 20955 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/fecs_inst.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/fecs_sig.bin
64 Aug 11 08:19 usr/lib/firmware/nvidia/gp100/gr/gpccs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/gpccs_bl.bin
root 2080 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/gpccs_data.bin
root 12458 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/gpccs_inst.bin
root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/gpccs_sig.bin
root 0 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/sw_bundle_init.bin
root 6240 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/sw_ctx.bin
root 0 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/sw_method_init.bin
root 2248 Jun 22 08:55 usr/lib/firmware/nvidia/gp100/gr/sw_nonctx.bin
0 Aug 11 08:19 usr/lib/firmware/nvidia/gp102alhost ~]# lsinitrd |grep nvidia
below (click the carrot to descend)

drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp102/acr
-rw-r–r-- 1 root root 1280 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/acr/bl.bin
-rw-r–r-- 1 root root 17152 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/acr/ucode_load.bin
-rw-r–r-- 1 root root 3328 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/acr/ucode_unload.bin
-rw-r–r-- 1 root root 1280 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/acr/unload_bl.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp102/gr
lrwxrwxrwx 1 root root 63 Aug 11 08:19 usr/lib/firmware/nvidia/gp102/gr/fecs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/fecs_bl.bin
-rw-r–r-- 1 root root 2256 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/fecs_data.bin
-rw-r–r-- 1 root root 20927 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/fecs_inst.bin
-rw-r–r-- 1 root root 192 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/fecs_sig.bin
lrwxrwxrwx 1 root root 64 Aug 11 08:19 usr/lib/firmware/nvidia/gp102/gr/gpccs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/gpccs_bl.bin
-rw-r–r-- 1 root root 1832 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/gpccs_data.bin
-rw-r–r-- 2 root root 0 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/gpccs_inst.bin
-rw-r–r-- 1 root root 192 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/gpccs_sig.bin
-rw-r–r-- 1 root root 7680 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/sw_bundle_init.bin
-rw-r–r-- 1 root root 6216 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/sw_ctx.bin
-rw-r–r-- 1 root root 12288 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/sw_method_init.bin
-rw-r–r-- 1 root root 2496 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/gr/sw_nonctx.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp102/nvdec
-rw-r–r-- 1 root root 3328 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/nvdec/scrubber.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp102/sec2
-rw-r–r-- 1 root root 656 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/sec2/desc.bin
-rw-r–r-- 1 root root 99072 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/sec2/image.bin
-rw-r–r-- 1 root root 192 Jun 22 08:55 usr/lib/firmware/nvidia/gp102/sec2/sig.bin
drwxr-xr-x 6 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp104
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/acr
lrwxrwxrwx 1 root root 59 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/acr/bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/bl.bin
lrwxrwxrwx 1 root root 67 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/acr/ucode_load.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/ucode_load.bin
lrwxrwxrwx 1 root root 69 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/acr/ucode_unload.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/ucode_unload.bin
lrwxrwxrwx 1 root root 66 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/acr/unload_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/unload_bl.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/gr
lrwxrwxrwx 1 root root 63 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/gr/fecs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/fecs_bl.bin
-rw-r–r-- 1 root root 2576 Jun 22 08:55 usr/lib/firmware/nvidia/gp104/gr/fecs_data.bin
-rw-r–r-- 1 root root 22760 Jun 22 08:55 usr/lib/firmware/nvidia/gp104/gr/fecs_inst.bin
-rw-r–r-- 1 root root 192 Jun 22 08:55 usr/lib/firmware/nvidia/gp104/gr/fecs_sig.bin
lrwxrwxrwx 1 root root 64 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/gr/gpccs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/gpccs_bl.bin
-rw-r–r-- 1 root root 1832 Jun 22 08:55 usr/lib/firmware/nvidia/gp104/gr/gpccs_data.bin
-rw-r–r-- 2 root root 13307 Jun 22 08:55 usr/lib/firmware/nvidia/gp104/gr/gpccs_inst.bin
-rw-r–r-- 1 root root 192 Jun 22 08:55 usr/lib/firmware/nvidia/gp104/gr/gpccs_sig.bin
lrwxrwxrwx 1 root root 70 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/gr/sw_bundle_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_bundle_init.bin
lrwxrwxrwx 1 root root 62 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/gr/sw_ctx.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_ctx.bin
lrwxrwxrwx 1 root root 70 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/gr/sw_method_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_method_init.bin
lrwxrwxrwx 1 root root 65 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/gr/sw_nonctx.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_nonctx.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/nvdec
lrwxrwxrwx 1 root root 67 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/nvdec/scrubber.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/nvdec/scrubber.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/sec2
lrwxrwxrwx 1 root root 62 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/sec2/desc.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/sec2/desc.bin
lrwxrwxrwx 1 root root 63 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/sec2/image.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/sec2/image.bin
lrwxrwxrwx 1 root root 61 Aug 11 08:19 usr/lib/firmware/nvidia/gp104/sec2/sig.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/sec2/sig.bin
drwxr-xr-x 6 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp106
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/acr
lrwxrwxrwx 1 root root 59 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/acr/bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/bl.bin
lrwxrwxrwx 1 root root 67 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/acr/ucode_load.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/ucode_load.bin
lrwxrwxrwx 1 root root 69 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/acr/ucode_unload.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/ucode_unload.bin
lrwxrwxrwx 1 root root 66 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/acr/unload_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/unload_bl.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/gr
lrwxrwxrwx 1 root root 63 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/gr/fecs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/fecs_bl.bin
-rw-r–r-- 1 root root 2256 Jun 22 08:55 usr/lib/firmware/nvidia/gp106/gr/fecs_data.bin
lrwxrwxrwx 1 root root 65 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/gr/fecs_inst.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/fecs_inst.bin
-rw-r–r-- 1 root root 192 Jun 22 08:55 usr/lib/firmware/nvidia/gp106/gr/fecs_sig.bin
lrwxrwxrwx 1 root root 64 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/gr/gpccs_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gm200/gr/gpccs_bl.bin
-rw-r–r-- 1 root root 1832 Jun 22 08:55 usr/lib/firmware/nvidia/gp106/gr/gpccs_data.bin
lrwxrwxrwx 1 root root 66 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/gr/gpccs_inst.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/gpccs_inst.bin
-rw-r–r-- 1 root root 192 Jun 22 08:55 usr/lib/firmware/nvidia/gp106/gr/gpccs_sig.bin
lrwxrwxrwx 1 root root 70 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/gr/sw_bundle_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_bundle_init.bin
lrwxrwxrwx 1 root root 62 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/gr/sw_ctx.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_ctx.bin
lrwxrwxrwx 1 root root 70 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/gr/sw_method_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_method_init.bin
lrwxrwxrwx 1 root root 65 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/gr/sw_nonctx.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_nonctx.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/nvdec
lrwxrwxrwx 1 root root 67 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/nvdec/scrubber.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/nvdec/scrubber.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/sec2
lrwxrwxrwx 1 root root 62 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/sec2/desc.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/sec2/desc.bin
lrwxrwxrwx 1 root root 63 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/sec2/image.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/sec2/image.bin
lrwxrwxrwx 1 root root 61 Aug 11 08:19 usr/lib/firmware/nvidia/gp106/sec2/sig.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/sec2/sig.bin
drwxr-xr-x 6 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp107
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/acr
lrwxrwxrwx 1 root root 59 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/acr/bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/bl.bin
lrwxrwxrwx 1 root root 67 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/acr/ucode_load.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/ucode_load.bin
lrwxrwxrwx 1 root root 69 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/acr/ucode_unload.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/ucode_unload.bin
lrwxrwxrwx 1 root root 66 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/acr/unload_bl.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/acr/unload_bl.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/gr
-rw-r–r-- 1 root root 576 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/fecs_bl.bin
-rw-r–r-- 1 root root 2756 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/fecs_data.bin
-rw-r–r-- 1 root root 22879 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/fecs_inst.bin
-rw-r–r-- 1 root root 192 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/fecs_sig.bin
-rw-r–r-- 1 root root 576 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/gpccs_bl.bin
-rw-r–r-- 1 root root 2100 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/gpccs_data.bin
-rw-r–r-- 1 root root 12587 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/gpccs_inst.bin
-rw-r–r-- 1 root root 192 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/gpccs_sig.bin
lrwxrwxrwx 1 root root 70 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/gr/sw_bundle_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_bundle_init.bin
-rw-r–r-- 1 root root 6000 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/sw_ctx.bin
lrwxrwxrwx 1 root root 70 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/gr/sw_method_init.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/gr/sw_method_init.bin
-rw-r–r-- 1 root root 2496 Jun 22 08:55 usr/lib/firmware/nvidia/gp107/gr/sw_nonctx.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/nvdec
lrwxrwxrwx 1 root root 67 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/nvdec/scrubber.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/nvdec/scrubber.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/sec2
lrwxrwxrwx 1 root root 62 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/sec2/desc.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/sec2/desc.bin
lrwxrwxrwx 1 root root 63 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/sec2/image.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/sec2/image.bin
lrwxrwxrwx 1 root root 61 Aug 11 08:19 usr/lib/firmware/nvidia/gp107/sec2/sig.bin -> …/…/…/…/…/…//usr/lib/firmware/nvidia/gp102/sec2/sig.bin
drwxr-xr-x 5 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp10b
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp10b/acr
-rw-r–r-- 1 root root 1280 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/acr/bl.bin
-rw-r–r-- 1 root root 19456 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/acr/ucode_load.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp10b/gr
-rw-r–r-- 1 root root 576 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/fecs_bl.bin
-rw-r–r-- 1 root root 3668 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/fecs_data.bin
-rw-r–r-- 1 root root 22306 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/fecs_inst.bin
-rw-r–r-- 1 root root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/fecs_sig.bin
-rw-r–r-- 1 root root 576 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/gpccs_bl.bin
-rw-r–r-- 1 root root 2340 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/gpccs_data.bin
-rw-r–r-- 1 root root 12921 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/gpccs_inst.bin
-rw-r–r-- 1 root root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/gpccs_sig.bin
-rw-r–r-- 2 root root 7664 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/sw_bundle_init.bin
-rw-r–r-- 1 root root 6036 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/sw_ctx.bin
-rw-r–r-- 2 root root 11928 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/sw_method_init.bin
-rw-r–r-- 1 root root 2744 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/gr/sw_nonctx.bin
drwxr-xr-x 2 root root 0 Aug 11 08:19 usr/lib/firmware/nvidia/gp10b/pmu
-rw-r–r-- 1 root root 652 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/pmu/desc.bin
-rw-r–r-- 1 root root 49408 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/pmu/image.bin
-rw-r–r-- 1 root root 76 Jun 22 08:55 usr/lib/firmware/nvidia/gp10b/pmu/sig.bin

Are there multiple drivers i need to block?

In case you still want to see vfio.conf, and grub line:

/etc/default/grub: GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap nomodeset rhgb quiet rd.driver.blacklist=nouveau iommu=1 amd_iommu=on rd.driver.pre=vfio.pci rd.driver.blacklist=nvidia"

/etc/modprobe.d/vfio.conf: options vfio-pci ids=10de:1401,10de:0fba

pci-ids:
21:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401] (rev a1)
21:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fba] (rev a1)

UPDATE: This one guy is suggesting that you need to use pci-stub to prevent the graphics driver from grabbing the gpu. Ill try this tonight (if i have time) and will report back. https://superuser.com/questions/1039315/qemu-with-gpu-pass-through-wont-start

wendell · September 10, 2017, 10:37pm

Yeah that can work too. You can also add the PCi IDs to the kernel command line and try that as well ?

wendell · September 10, 2017, 10:38pm

vfio-pci ? Not dot?

yverner · September 11, 2017, 1:27am

Wendell, thank you for the terrific guide. However, I am running into a few issues - perhaps someone here may be able to lend a hand.

So, I’m trying to passthrough a Fury X on an Asus Prime x370-pro. I’ve managed to successfully configure and apply the vfio driver, which I have been able to verify with lscpi -nk which outputs:

29:00.0 0300: 1002:7300 (rev c8)
        Subsystem: 1002:0b36
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu
29:00.1 0403: 1002:aae8
        Subsystem: 1002:aae8
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

My issue is that I do not receive any output from the VM that this gpu is passed through to. Am I missing something simple in the way in which I need to configure the virtual machine? I have removed all of the virtual display hw as well as the display console. Any help would be appreciated, many thanks.

dr_ew · September 11, 2017, 1:50am

oh my god. that was it.

21:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401] (rev a1) (prog-if 00 [VGA controller])
Subsystem: eVga.com. Corp. Device [3842:3967]
Flags: fast devsel, IRQ 11
Memory at f4000000 (32-bit, non-prefetchable) [disabled] [size=16M]
Memory at e0000000 (64-bit, prefetchable) [disabled] [size=256M]
Memory at f0000000 (64-bit, prefetchable) [disabled] [size=32M]
I/O ports at d000 [disabled] [size=128]
Expansion ROM at f5000000 [disabled] [size=512K]
Capabilities:
Kernel driver in use: vfio-pci
Kernel modules: nouveau, nvidia_drm, nvidia

-> -> ->

wendell · September 11, 2017, 2:22am

This is secondary CPU? Shut the machine down and from. Fresh power on is this the case? Fury does t support reinit so you can only init it one time per cold boot

yverner · September 11, 2017, 3:10am

Yes this is the secondary gpu. Still no luck after fully powering down back. That’s too bad about the fury, do you know if it will ever support it?

Wild guess, but could the non-support of reinit be responsible for why the vm isn’t producing any output? Could something cause it to initialize despite lspci stating that the kernel is using the vfio driver?

wendell · September 11, 2017, 3:37am

If the display stays dead/black no bios screen you might try a downloaded bios from tech power up and use that instead.

khaudio · September 11, 2017, 3:41am

Last time I had this issue, a new vbios solved it.

yverner · September 11, 2017, 2:39pm

Success! That was it. I bought my fury x used - turns out the previous owner had loaded a custom bios onto it it which was causing some kind of issue with the passthrough.

dr_ew · September 12, 2017, 12:25am

Has anyone seen this error before? I’m trying to attach the gpu / sound pci device to my VM through virt-manager. Only recommendation I can find online is to manually overide VCS but isnt that only for bad groupings? My groupings look fine to me (see below)

Actions taken:
remove: spice display
remove: sound
change: firmware = uefi
add pci: device: 0000:21:00:0: "GTX 960"
add pci: device (immediately below gpu): 0000:21:00:1 "NVIDIA Corp"
begin installation

Error
"2017-09-12T00:13:29.807418Z qemu-system-x86_64: -device vfio-pci,host=21:00.0,id=hostdev0,bus=pci.0,addr=0x6: vfio error: 0000:21:00.0: group 10 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver."

Full KVM Error Output

Unable to complete install: ‘internal error: process exited while connecting to monitor: 2017-09-12T00:13:29.765582Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/2 (label charserial0)
2017-09-12T00:13:29.807418Z qemu-system-x86_64: -device vfio-pci,host=21:00.0,id=hostdev0,bus=pci.0,addr=0x6: vfio error: 0000:21:00.0: group 10 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.’

Traceback (most recent call last):
File “/usr/share/virt-manager/virtManager/asyncjob.py”, line 89, in cb_wrapper
callback(asyncjob, *args, **kwargs)
File “/usr/share/virt-manager/virtManager/create.py”, line 2545, in _do_async_install
guest.start_install(meter=meter)
File “/usr/share/virt-manager/virtinst/guest.py”, line 497, in start_install
doboot, transient)
File “/usr/share/virt-manager/virtinst/guest.py”, line 433, in _create_guest
domain = self.conn.createXML(install_xml or final_xml, 0)
File “/usr/lib64/python2.7/site-packages/libvirt.py”, line 3567, in createXML
if ret is None:raise libvirtError(‘virDomainCreateXML() failed’, conn=self)
libvirtError: internal error: process exited while connecting to monitor: 2017-09-12T00:13:29.765582Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/2 (label charserial0)
2017-09-12T00:13:29.807418Z qemu-system-x86_64: -device vfio-pci,host=21:00.0,id=hostdev0,bus=pci.0,addr=0x6: vfio error: 0000:21:00.0: group 10 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

They are definitely bound to VFIO. Proof:

21:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401] (rev a1) (prog-if 00 [VGA controller])
Kernel driver in use: vfio-pci
Kernel modules: nouveau, nvidia_drm, nvidia

21:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fba] (rev a1)
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel

List of all groups

[root@localhost Documents]# lspci -nn
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1450]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Device [1022:1451]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1460]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1461]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1462]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1463]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1464]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1465]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1466]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1467]
03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB 3.1 XHCI Controller [1022:43bb] (rev 02)
03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b7] (rev 02)
03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b2] (rev 02)
1d:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
1d:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
1d:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
1e:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
1f:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 04)
21:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401] (rev a1)
21:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fba] (rev a1)
22:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP108 [GeForce GT 1030] [10de:1d01] (rev a1)
22:00.1 Audio device [0403]: NVIDIA Corporation GP108 High Definition Audio Controller [10de:0fb8] (rev a1)
23:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]
23:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1456]
23:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB3 Host Controller [1022:145c]
24:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]
24:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
24:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Device [1022:1457]

wendell · September 12, 2017, 12:35am

Can you run and paste the output of ls-iommu shell script?
Lspci -nn is just the PCi device IDs not iommu groups

dr_ew · September 12, 2017, 12:42am

Oh dang, i thought the outputs were the same… it does look like its shared… I only have 2 PCIe 16 slots too… what are the options? only VCS override? Looks like my host GPU is on its own group, can i assign VFIO to that one and pass it through? or does primary slot have to be used for host?

IOMMU Group 10 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB 3.1 XHCI Controller [1022:43bb] (rev 02)
IOMMU Group 10 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b7] (rev 02)
IOMMU Group 10 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b2] (rev 02)
IOMMU Group 10 1d:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 10 1d:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 10 1d:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 10 1e:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU Group 10 1f:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 04)
IOMMU Group 10 21:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401] (rev a1)
IOMMU Group 10 21:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fba] (rev a1)

Full IOMMU Groups

[root@localhost Documents]# ./iommu.sh
IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 10 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB 3.1 XHCI Controller [1022:43bb] (rev 02)
IOMMU Group 10 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b7] (rev 02)
IOMMU Group 10 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b2] (rev 02)
IOMMU Group 10 1d:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 10 1d:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 10 1d:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 10 1e:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU Group 10 1f:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 04)
IOMMU Group 10 21:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401] (rev a1)
IOMMU Group 10 21:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fba] (rev a1)
IOMMU Group 11 22:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP108 [GeForce GT 1030] [10de:1d01] (rev a1)
IOMMU Group 11 22:00.1 Audio device [0403]: NVIDIA Corporation GP108 High Definition Audio Controller [10de:0fb8] (rev a1)
IOMMU Group 1 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 2 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 3 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 4 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 5 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 6 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 6 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 6 23:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]
IOMMU Group 6 23:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1456]
IOMMU Group 6 23:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB3 Host Controller [1022:145c]
IOMMU Group 7 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 7 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 7 24:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]
IOMMU Group 7 24:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 7 24:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Device [1022:1457]
IOMMU Group 8 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 8 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 9 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1460]
IOMMU Group 9 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1461]
IOMMU Group 9 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1462]
IOMMU Group 9 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1463]
IOMMU Group 9 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1464]
IOMMU Group 9 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1465]
IOMMU Group 9 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1466]
IOMMU Group 9 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1467]