Return to Level1Techs.com

GPU passthrough with matching gpus (Arch)

I’m having some issues with gpu passthrough with matching Vega 64 cards on Arch. I have attempted to follow the instructions in the wiki but a few steps are unclear to me. Specifically, when using the scripts to isolate the gpu, what should be in MODULES=(...)?

If i don’t include (... vfio_pci vfio vfio_iommu_type1 vfio_virqfd...) I get a modprobe error at boot.

Regardless, when I run lspci -nnk the card shows Kernel modules: amdgpu

If I run modprode -i vfio-pci and then lspci -nnk I will see Kernel driver in use: vfio-pci only if i do not inlude the vfio modules in mkinitcpio.conf.

I cannot seem to find the combination that works or properly isolates the gpu at boot.

Okay so I have had a chance to play with this. I went back and followed the wiki again. It looks like the scripts stop the amdgpu driver from loading for the second card but does not load the vfio-pci driver.

30:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XL/XT [Radeon RX Vega 56/64] [1002:687f] (rev c1)
    Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] RX Vega64 [1002:0b36]
    Kernel driver in use: amdgpu
    Kernel modules: amdgpu
30:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 HDMI Audio [Radeon Vega 56/64] [1002:aaf8]
    Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 HDMI Audio [Radeon Vega 56/64] [1002:aaf8]
    Kernel driver in use: snd_hda_intel
    Kernel modules: snd_hda_intel
33:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XL/XT [Radeon RX Vega 56/64] [1002:687f] (rev c1)
    Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] RX Vega64 [1002:0b36]
    Kernel modules: amdgpu
33:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 HDMI Audio [Radeon Vega 56/64] [1002:aaf8]
    Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 HDMI Audio [Radeon Vega 56/64] [1002:aaf8]
    Kernel modules: snd_hda_intel

Once I try to boot the VM virt-manager doesn’t give me any errors but I get a black screen. I looks like this also loads the vfio-pci drivers

33:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XL/XT [Radeon RX Vega 56/64] [1002:687f] (rev c1)
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] RX Vega64 [1002:0b36]
	Kernel driver in use: vfio-pci
	Kernel modules: amdgpu
33:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 HDMI Audio [Radeon Vega 56/64] [1002:aaf8]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 HDMI Audio [Radeon Vega 56/64] [1002:aaf8]
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

If you disable ROM BAR I get a little farther, I get a boot screen but then the VM freezes and 1 cpu gets pegged to 100.

Thanks for any input!