Return to Level1Techs.com

Fedora 30, X470 Taichi, Ryzen 7 2700x, BIOS 3.2, NVIDIA 760 passthrough not working

Hi everyone, I have tried several guides now and I am just not able to get a working NVIDIA GPU pass-through setup. Specifically, I am using an ASRock, X470 Taichi Ultimate and no matter what I try, I am not able to pass-through my EVGA NVIDIA Geforce GTX 760 to a windows VM. Every time I try, I either get some sort of 127 error or the machine starts but I get no video signal anywhere and I know that the cards work. I got to the point where I started getting kernel tainted messages as well so that cannot be good.

I am also sort of confused as to the following:

  1. should upgrade my BIOS to the latest version
  2. should I patch the kernel
  3. Should I be adding the PCI card before windows VM install or after

I have to be missing something because it was not this difficult to get things working on my intel based system.

Here is my build and some of the steps that I followed. Any guidance would be great.

Build
Motherboard:
ASRock, X470 Taichi Ultimate
BIOS Version: SMBIOS 3.2.1 present
Processor:
AMD Ryzen 7 2700X Eight-Core Processor
Disk:
Linux: Samsung Electronics NVMe SSD Controller SM961/PM961
Windows(7/10) VMSamsung SSD 860
GPUs:
Linux: PNY nVidia Quadro M2000
For VM Pass-through: EVGA nVidia GeForce GTX 760
Linux:
Fedora 30
Kernel: 5.2.9-200.fc30.x86_6
BIOS version: 3.2

vfio
[[email protected] ~]# dmesg | grep vfio
[ 0.000000] Command line: BOOT_IMAGE=(hd1,msdos2)/vmlinuz-5.2.9-200.fc30.x86_64 root=UUID=87faaa8b-9fad-488a-b180-5d6c516d19c9 ro resume=UUID=5799d52a-c9b5-4325-b228-a650ceda9066 rhgb iommu=1 amd_iommu=on rd.driver.blacklist=nouveau rd.driver.pre=vfio-pci
[ 0.000000] Kernel command line: BOOT_IMAGE=(hd1,msdos2)/vmlinuz-5.2.9-200.fc30.x86_64 root=UUID=87faaa8b-9fad-488a-b180-5d6c516d19c9 ro resume=UUID=5799d52a-c9b5-4325-b228-a650ceda9066 rhgb iommu=1 amd_iommu=on rd.driver.blacklist=nouveau rd.driver.pre=vfio-pci
[ 3.668937] vfio-pci 0000:0f:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[ 3.682022] vfio_pci: add [10de:1187[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.695089] vfio_pci: add [10de:0e0a[ffffffff:ffffffff]] class 0x000000/00000000
[ 4.855427] vfio-pci 0000:0f:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none

nouveau is blacklisted
[email protected] modprobe.d]$ cat blacklist.conf
blacklist nouveau
options nouveau modeset=0

vfio.conf exists
[[email protected] modprobe.d]$ cat vfio.conf
options vfio-pci ids=10de:1187,10de:0e0a

ran dracut and grub2-mkconfig

the kernel command line arguments are set
[[email protected] modprobe.d]cat/etc/default/grubGRUBTIMEOUT=5GRUBDISTRIBUTOR="(sed ‘s, release .*$,g’ /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT=“console”
GRUB_CMDLINE_LINUX=“resume=UUID=5799d52a-c9b5-4325-b228-a650ceda9066 rhgb iommu=1 amd_iommu=on rd.driver.blacklist=nouveau rd.driver.pre=vfio-pci”
GRUB_DISABLE_RECOVERY=“true”
GRUB_ENABLE_BLSCFG=true

[[email protected] proc]# cat cmdline
BOOT_IMAGE=(hd1,msdos2)/vmlinuz-5.2.9-200.fc30.x86_64 root=UUID=87faaa8b-9fad-488a-b180-5d6c516d19c9 ro resume=UUID=5799d52a-c9b5-4325-b228-a650ceda9066 rhgb iommu=1 amd_iommu=on rd.driver.blacklist=nouveau rd.driver.pre=vfio-pci

iommu groups look right
[u[email protected] ~]$ ./iommu.sh | grep “IOMMU Group 1[6,7]”
IOMMU Group 16 0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206GL [Quadro M2000] [10de:1430] (rev a1)
IOMMU Group 16 0e:00.1 Audio device [0403]: NVIDIA Corporation GM206 High Definition Audio Controller [10de:0fba] (rev a1)
IOMMU Group 17 0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 760] [10de:1187] (rev a1)
IOMMU Group 17 0f:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)

looks like passthrough works
[[email protected] ~]# lspci -knn | grep -e NVIDIA -A 4
0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206GL [Quadro M2000] [10de:1430] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1190]
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia_drm, nvidia
0e:00.1 Audio device [0403]: NVIDIA Corporation GM206 High Definition Audio Controller [10de:0fba] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1190]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 760] [10de:1187] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:2766]
Kernel driver in use: vfio-pci
Kernel modules: nouveau, nvidia_drm, nvidia
0f:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:2766]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
10:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a

For Ryzen if you’re using anything other than BIOS F22 (I think, don’t quote me on the exact version), you need to patch your kernel to avoid the “Unknown Header type” error.
It’s actually simple to do.
I’m not sure about the exact process on Fedora, but on Debian and Arch you just download the kernel’s source, download a patch file and compile them.