Ryzen GPU Passthrough Setup Guide: Fedora 26 + Windows Gaming on Linux | Level One Techs

Hi y’all,
after a windows update as well as a fedora 28 update I’ve been stuck in a bootloop at the vm startscreen (the tianocore thing).

Has anyone experience with that ?

Here’s my kvm.xml:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>Win11</name>
  <uuid>ea9bef44-d48e-4f72-a3cb-3d171e9184c2</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>8</vcpu>
  <iothreads>4</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='5'/>
    <vcpupin vcpu='6' cpuset='6'/>
    <vcpupin vcpu='7' cpuset='7'/>
    <iothreadpin iothread='1' cpuset='0-1'/>
    <iothreadpin iothread='2' cpuset='2-3'/>
    <iothreadpin iothread='3' cpuset='4-5'/>
    <iothreadpin iothread='4' cpuset='6-7'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.10'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/Win11_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='allow'>qemu64</model>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:4e:11:4b'/>
      <source network='default'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x0d' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='1'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='host,hv_time,kvm=off,hv_vendor_id=null'/>
  </qemu:commandline>
</domain>

EDIT: reinstalling did the trick … but only for now. Any ideas what to do about upcomming updates ?

regards Warui

Hi,

I just found something interesting: I was passing though a 4 port nic but only 1 port would show up in the vm. When I went looking for what could cause this behavior I found the answer in the .XML file for the VM:

The 1 card was split in to 4 diffrent ‘slots’ as in the exaple below:

And putting it in the same ‘slot’ fixed the issue:

Quick question, So when you Passthrough a dedicated GPU, you run monitors to the host GPU right? and the VM just utilizes the passthrough GPU processing for the VM.

Now I have a specific setup because I have many monitors for example if I have 5 Monitors plugged into my host GPU I cannot plug my 6th monitor into the passthrough GPU for windows since it prob wont display anything right?

Trying to figure this out, as I might have to buy a diff card with enough ports for host if this is the case, since I utilize a shit cheap card for the 6th monitor atm.

Watch some of our vids. When you pass through the GPU the VM treats the GPU like a real GPU so the outputs work only for the vm, but the outputs on the VM GPU do work.

1 Like

Right, Just was thinking out loud i suppose. rewatched the video kinda figured it out.

Just thinking of a good cheap alternative workstation card with x6 ports instead of using 3 GPU’s, 2 for host and 1 for vm dedication.

Matrox?

Yeah, I got an MST hub but linux hates it won’t recognize in Ubuntu just more hassle than its worth, could also daisy chain monitors but my monitors are not dp native. So I’ll prob just end up getting a used amd w600 or vision tek 7780 x6 mdp. Plus active adapters.

So, was setting up a windows VM so I could passthrough a GPU and get that sweet sweet gaming on windows on linux, but for me the sound didn’t “Just Work”

I installed Fedora just for this project, and didn’t know about SeLinux, which kept squashing every attempt the valliant little VMM Q/L was making to give me the sounds I was craving…

Well, now I know better!
I guess a little knowledge is a dangerous thing?

1 Like

Finally i got it working. My Setup:

AMD Ryzen 1700
Asrock Killer SLI X370 Board
Nvme Slot 1 : my old bare metal Windows 10 install
Nvme Slot 2: Fedora 30
PCIex16 1: Nvidia GT 710 for Host @ Nvidia driver
PCIex16 2: Vega 64 for Guest

libvirt config:

<domain type='kvm'>
  <name>win10-2</name>
  <uuid>a878a984-30b9-4f0a-a92d-93129e38c0b7</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>16775168</memory>
  <currentMemory unit='KiB'>16775168</currentMemory>
  <vcpu placement='static'>8</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-3.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10-2_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <vmport state='off'/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
    <topology sockets='1' cores='8' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/usr/share/virtio-win/virtio-win-0.1.141.iso'/>
      <target dev='sdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sdb'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sdc'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sda'/>
      <target dev='vdc' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:1d:bb:ab'/>
      <source network='br0'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
	<model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <sound model='ich9'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
	<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
	<address domain='0x0000' bus='0x0f' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
	<address domain='0x0000' bus='0x0f' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
	<vendor id='0x17ef'/>
	<product id='0x6047'/>
      </source>
      <address type='usb' bus='0' port='4'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
	<vendor id='0x1038'/>
	<product id='0x1300'/>
      </source>
      <address type='usb' bus='0' port='5'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

vfio.conf

options vfio-pci ids=1002:687f,1002:aaf8

iommu

IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 10 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 10 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 11 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
IOMMU Group 11 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
IOMMU Group 11 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
IOMMU Group 11 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
IOMMU Group 11 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
IOMMU Group 11 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
IOMMU Group 11 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
IOMMU Group 11 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
IOMMU Group 12 01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 [144d:a802] (rev 01)
IOMMU Group 13 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset USB 3.1 xHCI Controller [1022:43b9] (rev 02)
IOMMU Group 13 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset SATA Controller [1022:43b5] (rev 02)
IOMMU Group 13 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset PCIe Upstream Port [1022:43b0] (rev 02)
IOMMU Group 13 04:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 13 04:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 13 04:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 13 04:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 13 04:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 13 04:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 13 04:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 13 06:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 13 09:00.0 Non-Volatile memory controller [0108]: Silicon Motion, Inc. Device [126f:2260] (rev 03)
IOMMU Group 13 0a:00.0 Audio device [0403]: Creative Labs Sound Core3D [Sound Blaster Recon3D / Z-Series] [1102:0012] (rev 01)
IOMMU Group 14 0c:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
IOMMU Group 14 0c:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
IOMMU Group 15 0d:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1470] (rev c1)
IOMMU Group 16 0e:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1471]
IOMMU Group 17 0f:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XL/XT [Radeon RX Vega 56/64] [1002:687f] (rev c1)
IOMMU Group 18 0f:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 HDMI Audio [Radeon Vega 56/64] [1002:aaf8]
IOMMU Group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 2 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 6 00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 7 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 8 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 8 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 8 11:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
IOMMU Group 8 11:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
IOMMU Group 8 11:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller [1022:145c]
IOMMU Group 9 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 9 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 9 12:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
IOMMU Group 9 12:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)

Unfotunately when i try the CPU host-passthrough it gives me a hanging system with no graphics at all.

Have anyone experienced the same issue?

Tnx in advance,
Stefan

1 Like

You’re using a Vega card. It will have the reset issue so you need to FULLY cold boot to ensure it launches. It’s likely not an issue with host-passthrough but rather your Vega card.

With a GT 710, Nouveau might be better vs the proprietary Nvidia driver because since Fedora 30 is a rolling release, if the proprietary driver is incompatible with the new kernel, you’re screwed until it’s fixed. This problem doesn’t exist on Nouveau and for the GT 710, Nouveau makes more sense.

Regular Fedora is not a rolling release. Package updates go through the bugtracker for testing before they are published. And inside a release cycle there’s typically only minor updates to packages. The kernel is also being extensively tested, especially with drivers.

Fedora Rawhide on the other hand is rolling release or rather a daily build.

From an Nvidia and Blackmagic perspective, where new kernels and GCC versions simply break the proprietary drivers, I consider that a rolling release in terms of maintenance.

If you run AMD with purely UVC USB 3.0 capture cards, no issues.

Hi! anyone tried the pci passthrough on an asus b350f mobo using fedora 30 with two similar graphics cards?

I don’t have one, but b350 is pretty hit and miss. The acs groups on the b chipset is not that great, so most of them require the acs patch to get it working. That requires a custom-built kernel, which isn’t too hard to do, but it is a pain.

1 Like

Hi fellas,

I am using an ASRock, X470 Taichi Ultimate and no matter what I try, I am not able to pass-through my EVGA NVIDIA Geforce GTX 760 to a windows VM. Everytime I try, I either get some sort of 127 error or the machine starts but I get no video signal anywhere and I know that the cards work. In addition I am forced to restart my host but only after adding the NVIDIA to the VM and attempting to run the VM, What am I missing?

Build
Motherboard:
ASRock, X470 Taichi Ultimate
BIOS Version: SMBIOS 3.2.1 present
Processor:
AMD Ryzen 7 2700X Eight-Core Processor
Disk:
Linux: Samsung Electronics NVMe SSD Controller SM961/PM961
Windows(7/10) VMSamsung SSD 860
GPUs:
Linux: PNY nVidia Quadro M2000
For VM Pass-through: EVGA nVidia GeForce GTX 760
Linux:
Fedora 30
Kernel: 5.2.13-200.fc30.x86_64

vfio

[root@localhost ~]# dmesg | grep vfio

[ 0.000000] Command line: BOOT_IMAGE=(hd1,msdos2)/vmlinuz-5.2.13-200.fc30.x86_64 root=UUID=97f15595-efee-46ab-bd7f-97662444e4f8 ro resume=UUID=62022175-903a-47b1-92bb-989c8c02d2c3 iommu=1 amd_iommu=on rd.driver.pre=vfio-pci rhgb quiet rd.driver.blacklist=nouveau
[ 0.000000] Kernel command line: BOOT_IMAGE=(hd1,msdos2)/vmlinuz-5.2.13-200.fc30.x86_64 root=UUID=97f15595-efee-46ab-bd7f-97662444e4f8 ro resume=UUID=62022175-903a-47b1-92bb-989c8c02d2c3 iommu=1 amd_iommu=on rd.driver.pre=vfio-pci rhgb quiet rd.driver.blacklist=nouveau
[ 3.718855] vfio-pci 0000:0f:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[ 3.730025] vfio_pci: add [10de:1187[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.742089] vfio_pci: add [10de:0e0a[ffffffff:ffffffff]] class 0x000000/00000000
[ 4.633599] vfio-pci 0000:0f:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[ 96.372194] vfio-pci 0000:0f:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[ 96.384941] vfio-pci 0000:0f:00.1: enabling device (0000 -> 0002)
[ 97.604049] vfio-pci 0000:0f:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 97.608062] vfio-pci 0000:0f:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 98.754677] vfio-pci 0000:0f:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 98.754900] vfio-pci 0000:0f:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 98.772831] vfio-pci 0000:0f:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 98.773063] vfio-pci 0000:0f:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 98.787762] vfio-pci 0000:0f:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 98.787998] vfio-pci 0000:0f:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 98.812714] vfio-pci 0000:0f:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 98.814835] vfio-pci 0000:0f:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff
[ 98.814959] vfio-pci 0000:0f:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 98.825643] vfio-pci 0000:0f:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 98.825825] vfio-pci 0000:0f:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 98.825860] vfio-pci 0000:0f:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 98.825901] vfio-pci 0000:0f:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 98.826047] vfio-pci 0000:0f:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 98.826082] vfio-pci 0000:0f:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 98.826207] vfio-pci 0000:0f:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 98.826241] vfio-pci 0000:0f:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 98.844549] vfio-pci 0000:0f:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 98.845177] vfio-pci 0000:0f:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 98.858289] vfio-pci 0000:0f:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 98.858670] vfio-pci 0000:0f:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 98.858707] vfio-pci 0000:0f:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 98.859028] vfio-pci 0000:0f:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 98.881584] vfio-pci 0000:0f:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 98.882178] vfio-pci 0000:0f:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 161.618983] vfio-pci 0000:0f:00.1: Refused to change power state, currently in D3

nouveau is blacklisted

[user0@localhost ~]$ cat /etc/default/grub

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="(sed 's, release .*,g’ /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT=“console”
GRUB_CMDLINE_LINUX=“resume=UUID=62022175-903a-47b1-92bb-989c8c02d2c3 iommu=1 amd_iommu=on rd.driver.pre=vfio-pci rhgb quiet rd.driver.blacklist=nouveau”
GRUB_DISABLE_RECOVERY=“true”
GRUB_ENABLE_BLSCFG=true

[user0@localhost ~]$ cat /etc/modprobe.d/blacklist.conf 

blacklist nouveau
options nouveau modeset=0

vfio.conf exists

[user0@localhost ~]$ cat /etc/modprobe.d/vfio.conf 

options vfio-pci ids=10de:1187,10de:0e0a

ran dracut and grub2-mkconfig

the kernel command line arguments are set

<code>[user0@localhost ~]$ cat /etc/default/grub

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="(sed 's, release .*,g’ /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT=“console”
GRUB_CMDLINE_LINUX=“resume=UUID=62022175-903a-47b1-92bb-989c8c02d2c3 iommu=1 amd_iommu=on rd.driver.pre=vfio-pci rhgb quiet rd.driver.blacklist=nouveau”
GRUB_DISABLE_RECOVERY=“true”
GRUB_ENABLE_BLSCFG=true

[root@localhost proc]# cat cmdline

BOOT_IMAGE=(hd1,msdos2)/vmlinuz-5.2.9-200.fc30.x86_64 root=UUID=87faaa8b-9fad-488a-b180-5d6c516d19c9 ro resume=UUID=5799d52a-c9b5-4325-b228-a650ceda9066 rhgb iommu=1 amd_iommu=on rd.driver.blacklist=nouveau rd.driver.pre=vfio-pci

iommu groups look right

[user0@localhost ~]$ ./iommu.sh | grep “IOMMU Group 1[6,7]”

IOMMU Group 16 0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206GL [Quadro M2000] [10de:1430] (rev a1)
IOMMU Group 16 0e:00.1 Audio device [0403]: NVIDIA Corporation GM206 High Definition Audio Controller [10de:0fba] (rev a1)
IOMMU Group 17 0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 760] [10de:1187] (rev a1)
IOMMU Group 17 0f:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)

looks like passthrough works

[user0@localhost ~]$ lspci -knn | grep -e NVIDIA -A 4

0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206GL [Quadro M2000] [10de:1430] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1190]
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia_drm, nvidia
0e:00.1 Audio device [0403]: NVIDIA Corporation GM206 High Definition Audio Controller [10de:0fba] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1190]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 760] [10de:1187] (rev ff)
Kernel driver in use: vfio-pci
Kernel modules: nouveau, nvidia_drm, nvidia
0f:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev ff)
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
10:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]

Hi @wendell,

can you please help me out. I am not able to get gpu passthrough working on Fedora 30 running on a x470 ASRock Taichi Ultimate. Went through a number of sources but still stuck:(

we had another user reporting the 127 error. I am investigating.

Gotcha,

I’m a newbie but I’m glad to help. I can do anything with this install as I am figuring things out. So please let me know if you would like me to try something. My build is listed in the above comment and below. Million thanks.

Motherboard:
ASRock, X470 Taichi Ultimate
BIOS Version: SMBIOS 3.2.1 present
Processor:
AMD Ryzen 7 2700X Eight-Core Processor
Disk:
Linux: Samsung Electronics NVMe SSD Controller SM961/PM961
Windows(7/10) VMSamsung SSD 860
GPUs:
Linux: PNY nVidia Quadro M2000
For VM Pass-through: EVGA nVidia GeForce GTX 760
Linux:
Fedora 30
Kernel: 5.2.13-200.fc30.x86_64

Hi @wendell - i made the pledge to move everything to linux before i leave quarantine, so i am consuming way to much of your content right now - with all the news since 2017, i was wondering whether there are any updates to this guide?

Also a question to everyone - i will probably be running a vpn server, a nas, a windows vm and the host, as well as 2-3 development vms in the end - would it make sense to move to proxmox for the host?

So that guide is still fairly accurate. Most people don’t succeed the first time off a guide alone, so my recommendation is to follow a guide and make a thread when you run into issues. There’s a fair few of us who are more than happy to help.

2 Likes