Hi everyone, I have tried several guides now and I am just not able to get a working NVIDIA GPU pass-through setup. Specifically, I am using an ASRock, X470 Taichi Ultimate and no matter what I try, I am not able to pass-through my EVGA NVIDIA Geforce GTX 760 to a windows VM. Every time I try, I either get some sort of 127 error or the machine starts but I get no video signal anywhere and I know that the cards work. I got to the point where I started getting kernel tainted messages as well so that cannot be good.
I am also sort of confused as to the following:
- should upgrade my BIOS to the latest version
- should I patch the kernel
- Should I be adding the PCI card before windows VM install or after
I have to be missing something because it was not this difficult to get things working on my intel based system.
Here is my build and some of the steps that I followed. Any guidance would be great.
Build
Motherboard:
ASRock, X470 Taichi Ultimate
BIOS Version: SMBIOS 3.2.1 present
Processor:
AMD Ryzen 7 2700X Eight-Core Processor
Disk:
Linux: Samsung Electronics NVMe SSD Controller SM961/PM961
Windows(7/10) VMSamsung SSD 860
GPUs:
Linux: PNY nVidia Quadro M2000
For VM Pass-through: EVGA nVidia GeForce GTX 760
Linux:
Fedora 30
Kernel: 5.2.9-200.fc30.x86_6
BIOS version: 3.2
vfio
[root@localhost ~]# dmesg | grep vfio
[ 0.000000] Command line: BOOT_IMAGE=(hd1,msdos2)/vmlinuz-5.2.9-200.fc30.x86_64 root=UUID=87faaa8b-9fad-488a-b180-5d6c516d19c9 ro resume=UUID=5799d52a-c9b5-4325-b228-a650ceda9066 rhgb iommu=1 amd_iommu=on rd.driver.blacklist=nouveau rd.driver.pre=vfio-pci
[ 0.000000] Kernel command line: BOOT_IMAGE=(hd1,msdos2)/vmlinuz-5.2.9-200.fc30.x86_64 root=UUID=87faaa8b-9fad-488a-b180-5d6c516d19c9 ro resume=UUID=5799d52a-c9b5-4325-b228-a650ceda9066 rhgb iommu=1 amd_iommu=on rd.driver.blacklist=nouveau rd.driver.pre=vfio-pci
[ 3.668937] vfio-pci 0000:0f:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[ 3.682022] vfio_pci: add [10de:1187[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.695089] vfio_pci: add [10de:0e0a[ffffffff:ffffffff]] class 0x000000/00000000
[ 4.855427] vfio-pci 0000:0f:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
nouveau is blacklisted
user0@localhost modprobe.d]$ cat blacklist.conf
blacklist nouveau
options nouveau modeset=0
vfio.conf exists
[user0@localhost modprobe.d]$ cat vfio.conf
options vfio-pci ids=10de:1187,10de:0e0a
ran dracut and grub2-mkconfig
the kernel command line arguments are set
[user0@localhost modprobe.d]cat/etc/default/grubGRUBTIMEOUT=5GRUBDISTRIBUTOR="(sed ‘s, release .*$,g’ /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT=“console”
GRUB_CMDLINE_LINUX=“resume=UUID=5799d52a-c9b5-4325-b228-a650ceda9066 rhgb iommu=1 amd_iommu=on rd.driver.blacklist=nouveau rd.driver.pre=vfio-pci”
GRUB_DISABLE_RECOVERY=“true”
GRUB_ENABLE_BLSCFG=true
[root@localhost proc]# cat cmdline
BOOT_IMAGE=(hd1,msdos2)/vmlinuz-5.2.9-200.fc30.x86_64 root=UUID=87faaa8b-9fad-488a-b180-5d6c516d19c9 ro resume=UUID=5799d52a-c9b5-4325-b228-a650ceda9066 rhgb iommu=1 amd_iommu=on rd.driver.blacklist=nouveau rd.driver.pre=vfio-pci
iommu groups look right
[user0@localhost ~]$ ./iommu.sh | grep “IOMMU Group 1[6,7]”
IOMMU Group 16 0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206GL [Quadro M2000] [10de:1430] (rev a1)
IOMMU Group 16 0e:00.1 Audio device [0403]: NVIDIA Corporation GM206 High Definition Audio Controller [10de:0fba] (rev a1)
IOMMU Group 17 0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 760] [10de:1187] (rev a1)
IOMMU Group 17 0f:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
looks like passthrough works
[root@localhost ~]# lspci -knn | grep -e NVIDIA -A 4
0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206GL [Quadro M2000] [10de:1430] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1190]
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia_drm, nvidia
0e:00.1 Audio device [0403]: NVIDIA Corporation GM206 High Definition Audio Controller [10de:0fba] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1190]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 760] [10de:1187] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:2766]
Kernel driver in use: vfio-pci
Kernel modules: nouveau, nvidia_drm, nvidia
0f:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:2766]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
10:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a