Ryzen GPU Passthrough Setup Guide: Fedora 26 + Windows Gaming on Linux | Level One Techs

Chop91 · August 16, 2017, 9:18pm

I am trying to set up GPU passthrough according to this guide the last 5 days, but it is still not working.

System:
Asus Prime x370-Pro (Bios 0807 with AGESA update 1.0.0.6a)
Ryzen 7 1700
GPU 1: AMD RX 460 (also tried AMD 6970 and AMD 5870) in Slot 1
GPU 2: AMD 6450 in Slot 3

I activated SVM and IOMMU in the UEFI, then I have installed Fedora 26 Workstation.
Next I did the following:

sudo dnf update
sudo dnf install @virtualization
sudo dnf install nano

sudo nano /etc/default/grub
added “iommu=1 amd_iommu=on rd.driver.pre=vfio-pci” to GRUB_CMDLINE_LINUX

sudo nano /etc/modprobe.d/kvm.conf
checked if kvm_amd nested=1 and kvm_intel nested=1 are not activated

sudo nano /etc/modprobe.d/vfio.conf
set the ids (got them by using the bash script) of the GPU to passthrough (GPU 1)
options vfio-pci ids=1002:67ef,1002:aae0

sudo dracut -f --kver uname -r
sudo grub2-mkconfig -o /etc/grub2-efi.cfg

There is no error for the two commands above, so I restarted as the guide says, but I am not able to boot again. I get the following two errors:
[0.211157] ACPI Error: Needed [Integer/String/Buffer], found [Region] ffff91e37e8e73a8 (20170119/exresop-424)
[0.211163] ACPI Exception: AE_AML_OPERAND_TYPE, Could not execute arguments for [IOB2] (Region) (20170119/nsinit-412)

I can boot with acpi=off as boot option, but if I check the driver of GPU 1 by using lspci -vnn it still says radeon or amdgpu and not vfio-pci.

I have no idea what I am doing wrong.

wendell · August 17, 2017, 1:33am

Try updating/creating a vfio.conf in /etc/dracut.conf.d with add_drivers+=“vfio vfio_iommu_type1 vfio_pci” and THEN running dracut -f --kver

tkoham · August 17, 2017, 1:39am

@wendell: Have you tried to bypass the vbios security processor by specifying a romfile=?

I’d be interested to see if that works, as a patched kernel that does the same for linux seems to work a treat.

How funny would it be if the only way to get bios modding to work on windows was running it in a linux hypervisor?

wendell · August 17, 2017, 1:53am

Not for that purpose but for single GPU machines (serial terminal) the vbios file works great to reinit the primary GPU.

tkoham · August 17, 2017, 2:03am

I’d be interested to see if a modded bios would work. I know it works for unlocking 560’s without flashing, but I don’t have a vega GPU on hand to test with.

I appreciate the lengths you guys go to. Good content.

Chop91 · August 17, 2017, 10:04am

I tried that before but its still the same result for me.
I did another kernel update and I am now using 4.12.5 and the only thing that changes seems to be the error code and date.

[0.205289] ACPI Error: Needed [Integer/String/Buffer], found [Region] ffff98bb6e16e000 (20170303/exresop-424)
[0.205296] ACPI Exception: AE_AML_OPERAND_TYPE, Could not execute arguments for [IOB2] (Region) (20170303/nsinit-412)

I also tried everything I mentioned in my post before, but without doing the update, since I read somewhere that a newer kernel could be a problem.

I guess if everything works Fedora should boot on the monitor of my host gpu and not using the guest gpu?

Chop91 · August 20, 2017, 6:47pm

OK, I still have the same error but now I have “vfio-pci” as driver for my passthrough GPU.
I changed the slots of both GPUs to get this result.

Now I am running the following configuration:
PCIx16_1: HD6450 - 8 Lanes
PCIx16_2: RX460 (passthrough) - 8 Lanes
PCIx16_3: empty

I read somewhere that the following configuration should also work (tried that all the time):
PCIx16_1: RX460 (passthrough) - 16 Lanes
PCIx16_2: empty
PCIx16_3: HD6450 - 2 or 4 Lanes

so one can use all 16 lanes for the passthrough GPU. Unfortunately this does not seem to work for me with the Asus Prime x370-Pro. Is this configuration working with the mentioned AsRock or Gigabyte board from the guide?

foppe · August 20, 2017, 6:52pm

I don’t have an answer for you, but I can tell you that unless you have a 1080Ti, you won’t come close to saturating 8 lanes, so worrying about this isn’t really necessary.

mihawk90 · August 20, 2017, 7:07pm

As I understand the GPU in the first slot is always initialised first, so it is always the host GPU. There is supposedly a way around this if you have a card that supports being re-initialised, because then you can uninitialise it after the boot and use it for passthrough that way.

Chop91 · August 20, 2017, 7:28pm

At the moment I am trying the following:

But if I deactivate CSM I get “The VGA card is not supported by UEFI driver”. So at the moment I search for a BIOS update to try this.

I plan to buy a Vega 64 or 56 if there are some affordable custom models and as far as I understand my motherboard manual, if I use slot 1 and 2 it runs PCIE 2.0 @ 8 lanes each and slot 1 alone PCIE 3.0 @ 16 lanes.

EDIT:
Successfully updated the BIOS according to this thread:

Turning off CSM works, but still no success to use the 3rd slot as primary GPU.

mihawk90 · August 20, 2017, 7:43pm

PCIe 3.0 x8, but yes. It’s how it works on all boards (including the B350 boards that support CF configurations).

Chop91 · August 20, 2017, 8:17pm

I think you are right (GPUZ shows the same result).
I am just a little bit confused about the motherboard manual.

“PCIE 3.0/2.0 x16_1 slot” and “PCIE 3.0/2.0 x16_2 slot”, so I expected for some reason they use PCIE 2.0 if you use slot 1 and 2.

mihawk90 · August 21, 2017, 1:13am

No it just means they re backward compatible with 2.0 devices, just weird then that it doesn’t state 1.0 devices… kind of inconsistent I guess.

axavio · August 23, 2017, 6:20am

More troubles here. Wendell makes it look so easy! )

It appears to me that my vfio module isn’t getting loaded. How should it show up in lsmod?

I have Ryzen 7 1700X on a X370 Taichi with a 1070 and an RX 550. Does it matter which card is in which slot?

Also it looks like vfio isn’t getting hold of my 1070:

From ls -nnk:

0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:5173]
Kernel driver in use: nouveau
Kernel modules: nouveau
0e:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:5173]
Kernel driver in use: snd_hda_intel

My dracut.conf:
[dave@localhost ~]$ cat /etc/dracut.conf.d/vfio.conf
add_drivers+=“vfio vfio_iommu_type1 vfio_pci”

I have the 10de:10f0 and 10de:1b81 in my vfio.conf:
[dave@localhost ~]$ cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:1b81,10de:10f0

Any pointers would be greatly appreciated.

munchiesmadness · August 23, 2017, 10:51am

Hey Chop91 I have similar hardware and get very similar errors on fedora.

I choose not to worry about that.

what I would like is help getting past this error 43 issue with nvidia cards.

here is my basic vm win10-xml

I would love it if someone who managed to get nvidia cards working in windows 10 on a fedora host, could reply with a copy of their xml for comparison.

my last vm attempted to use a copy of my cards rom as per this thread on stack overflow THIS SOLOUTION , However virt-manager threw up a ton of “feature not supported” errors and failed to boot.

so
Ideas?

did you manage to get your nvidia card working?

Chop91 · August 23, 2017, 11:20am

I had to run my host GPU in the primary PCIEx16 slot and my guest GPU in the secondary PCIEx16 slot.
Maybe this works for you too.

Chop91 · August 23, 2017, 11:26am

The guide says you should add the following lines to the .xml file to solve the error 43:

#if you have an nvidia card add: 
<kvm>
      <hidden state='on'/>
</kvm>

munchiesmadness · August 23, 2017, 11:31am

yeah nah. needs a little more than that. tried failed. thats why i was asking for other examples. ta ^_^.

from arch linux wiki but not with current drivers, and this one

Chop91 · August 23, 2017, 11:37am

I managed to run win10 as guest, but in the Cinebench I have just arround 32 fps.
Running win10 bare metal I get arround 75 fps.

If I add “kvm_amd npt=0” to the kvm.conf I get arround 50 fps in the benchmark but in games performance seems to be worse. (As far as I read the GPU performance with ryzen is not the best).
Anyone here with better performance?

Also I have a strange problem…
I also installed a xubuntu 17.04 guest with GPU passthrough.
Everything seems to work fine, but if I shutdown the guest, it freezes and also my host,
so I have to force off the host, after that my whole host system seems to be broken.
It shows me the emergency console and before the following error (additionally to the 2 ACPI errors):

[ 8.934994] radeon 0000:01:00.0: Invalid PCI ROM header signature: expecting
0xaa55, got 0xffff

EDIT: I configured it like my win10 guest.
I tried exactly the same configuration of the xubuntu guest with and without GPU passthrough.
The first case freezes and the latter one works fine.
Any ideas?

My current GPU passthrough config for xubuntu 17.04 (not sure if hyperv is a windows only setting):

<domain type='kvm'>
  <name>ubuntu17.04</name>
  <uuid>f3cc0bd9-e7fc-4277-85e9-cdd3f218bbe4</uuid>
  <memory unit='KiB'>14336000</memory>
  <currentMemory unit='KiB'>14336000</currentMemory>
  <vcpu placement='static'>14</vcpu>
  <iothreads>7</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='5'/>
    <vcpupin vcpu='6' cpuset='6'/>
    <vcpupin vcpu='7' cpuset='7'/>
    <vcpupin vcpu='8' cpuset='8'/>
    <vcpupin vcpu='9' cpuset='9'/>
    <vcpupin vcpu='10' cpuset='10'/>
    <vcpupin vcpu='11' cpuset='11'/>
    <vcpupin vcpu='12' cpuset='12'/>
    <vcpupin vcpu='13' cpuset='13'/>
    <iothreadpin iothread='1' cpuset='0-1'/>
    <iothreadpin iothread='2' cpuset='2-3'/>
    <iothreadpin iothread='3' cpuset='4-5'/>
    <iothreadpin iothread='4' cpuset='6-7'/>
    <iothreadpin iothread='5' cpuset='8-9'/>
    <iothreadpin iothread='6' cpuset='10-11'/>
    <iothreadpin iothread='7' cpuset='12-13'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-2.9'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/ubuntu17.04_VARS.fd</nvram>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
    </hyperv>
    <vmport state='off'/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='7' threads='2'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/var/lib/libvirt/images/xubuntu.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='nec-xhci'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='3'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xc'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0xd'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0xe'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:01:95:7a'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/>
    </sound>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046a'/>
        <product id='0x010d'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc01e'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x29' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x29' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
</domain>

axavio · August 23, 2017, 4:02pm

Thanks. I put the guest GPU in the lower slot. No change, but at least I am ready once I get past this “vfio not loading” business.