Single AMD GPU passthrough guide

Software & Operating Systems VFIO
1

Hello everyone,

Skip this if you are only interested in the guide
Ever since my first successful vfio experience, I knew I will never want to go back to just running one standalone os if that os ain’t Linux.

Since I had used single gpu passthrough in the past I, I thought I knew how to do it and boy oh boy was I wrong.
Using an nvidia in the past, and one of the tutorials available online this one being my favorite I managed to get some great performance out of my new(back then) r7 3700x rig.

After giving that card back and buying a rx 580 after a disastrous experience with a gigabyte 5700xt(and it’s beautiful driver issues), I wanted to replicate the process.

But like many, I failed, but I was consistent and with a few semi-dead semi-seg-faulted sessions I managed to successfully get my RX 580 to behave like virtualization champ .

The tutorial part
As a starting point I used this outstanding guide and I adapted my stuff to work with AMD cards.
While I only had my xfx rx 580 which I know that suffers from the reset bug, I think that others should be able to use my guide(ofc, change the gpu pcie address, and adapt to your desktop enviornment)
Please keep in mind that this is my first guide, by no means do I believe I did something outstanding in the tutorial itself, and because of that any advice or pointer would be greatly appreciated towards creating a better guide for people that are not as versed in vfio, or people that just don’t have the time to poke around.
amd-single-gpu-passthrough

In it’s current state, the reset works, I can use my debian with kde normally, and when I start my macOS vm, kde is being shut down, amdgpu module is unloaded and the sound counterpart as well.
I also use the cpu governor to performance and hugepages allocation hook scripts.
When I stop my vm, kde is back up and runs like a champ.

Currently, I mostly do my sysadmin work on my macOS Big Sur vm, this iteration of macOS added a few virtio drivers, and using OpenCore, I also got filevault working and I pass my own Intune checks which is hilarious.

Hope you guys will find this useful
Also big thank you to Wendel and gnif for posting youtube videos and making this possible.

2 Likes
2

Hi. Maybe you can help me with your learnings from your experiments. I want to setup a single GPU passthrough on my Arch system with my Sapphire Radeon 550.

IOMMU Group 16:
	09:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Lexa PRO [Radeon 540/540X/550/550X / RX 540X/550/550X] [1002:699f] (rev c7)
	09:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] [1002:aae0]

I followed a lot of tutorials and I assume I created the hooks and qemu config correctly. When I boot up the screen goes black and the monitor goes to standby. No signal.

I checked the logs and it seems that I cannot unload the amdgpu

sudo modprobe -r amdgpu
modprobe: FATAL: Module amdgpu is in use.

There are other modules using the amdgpu. But I’m not able to unload these modules as well.

Any ideas?
@examen1996

1 Like
3

Hey @raabe @examen1996 , any solution to this issue yet? I’m running into the same problem and can’t for the life of me figure it out. Any help is appreciated :slight_smile: Thanks!

4

Thanks for sharing your guide, I am trying to do the same thing here but I have not assign any uefi bios to VM.

Currently win10 is reporting atimpeg.sys error on bluescreen.(once I have amd driver installed, same thing happen on win7)

blus
blus928×670 11 KB

do you reckon thats casue by GPU not inited by VM?

@wendell I know you may be hated fiddling into single amd gpu passthrough topic, I am just not so good with linux and kvm, has been once successfully passed ATI5970 into a win7 vm runing on i5 4690k with PCI_stub following a youtube video.

this time I am working with the hardware list below
cpu: amd 2700x
RAM:16g ram
MB: B450M-A PRO MAX (MS-7C52) (bios 7C52v3D) should be latest already
GPU: AMD HD7970

BaseOS: Linux Mint 20.2 Uma"
kvm: QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.21)
virsh:6.0.0

this is current driver status on the system

29:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [100)
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT2 [Radeon HD 7970 GHz Edition] [1002:3000]
Flags: fast devsel, IRQ 65
Memory at e0000000 (64-bit, prefetchable) [size=256M]
Memory at fcf00000 (64-bit, non-prefetchable) [size=256K]
I/O ports at e000 [size=256]
Expansion ROM at 000c0000 [disabled] [size=128K]
Capabilities:
Kernel driver in use: vfio-pci
Kernel modules: radeon, amdgpu

29:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti HDMI Audio [Radeon HD 7870 XT / 7950/7970] [1002:aaa0]
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti HDMI Audio [Radeon HD 7870 XT / 7950/7970] [1002:aaa0]
Flags: bus master, fast devsel, latency 0, IRQ 5
Memory at fcf60000 (64-bit, non-prefetchable) [size=16K]
Capabilities:
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel

I start my vm with virsh in cli over ssh, please help me on this (been working on it for 2 days now) I will attach my vm xml under as well


<domain type='kvm' id='10'>
  <name>win</name>
  <uuid>c7a5fdbd-cdaf-9455-926a-d65c16db1809</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <iothreads>4</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='5'/>
    <vcpupin vcpu='6' cpuset='6'/>
    <vcpupin vcpu='7' cpuset='7'/>
    <iothreadpin iothread='1' cpuset='0-1'/>
    <iothreadpin iothread='2' cpuset='2-3'/>
    <iothreadpin iothread='3' cpuset='4-5'/>
    <iothreadpin iothread='4' cpuset='6-7'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hap state='on'/>
    <privnet/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vpindex state='on'/>
      <synic state='on'/>
      <reset state='on'/>
      <vendor_id state='on' value='123'/>
      <frequencies state='on'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='4' threads='2'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/home/rainman/new200/win10.img' index='2'/>
      <backingStore/>
      <target dev='hda' bus='sata'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu'/>
      <target dev='hdc' bus='sata'/>
      <readonly/>
      <alias name='sata1-0-0'/>
      <address type='drive' controller='1' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='qemu-xhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='sata' index='1'>
      <alias name='sata1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x01' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:88:e8:84'/>
      <source bridge='nm-bridge'/>
      <target dev='vnet0'/>
      <model type='rtl8139'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x29' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>libvirt-c7a5fdbd-cdaf-9455-926a-d65c16db1809</label>
    <imagelabel>libvirt-c7a5fdbd-cdaf-9455-926a-d65c16db1809</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+64055:+108</label>
    <imagelabel>+64055:+108</imagelabel>
  </seclabel>
</domain>


Sorry about messy formatting, please feel free to ask if any more information is missing.

Thank you