Need Help w/ X570 Taichi GPU Passthrough [Fedora 30]

I just upgraded to the new Ryzen 7 3700X w/ the X570 Taichi. I’m very familiar with GPU passthrough, however I’m having some trouble with IOMMU groups with this board on the latest BIOS (Version 1.80 w/ AGESA 1.0.0.3 ABB). Both IOMMU and AMD Virtualization are enabled in my BIOS. Is there any workarounds for Ryzen with bad IOMMU groups? I’m also currently running a Quadro K2000 in my top PCI-E slot and a RX Vega 56 in the second PCI-E slot which seem to both be placed in IOMMU group 0. My Windows 10 VM is already configured and installed. I also plan to try Looking Glass so I can avoid using 2 inputs/monitors for my host and VM.

Kernel: 5.2.8-200.fc30.x86_64

My IOMMU groups are as follows:
/sys/kernel/iommu_groups/7/devices/0000:00:14.3 /sys/kernel/iommu_groups/7/devices/0000:00:14.0 /sys/kernel/iommu_groups/5/devices/0000:00:07.0 /sys/kernel/iommu_groups/5/devices/0000:11:00.0 /sys/kernel/iommu_groups/5/devices/0000:00:07.1 /sys/kernel/iommu_groups/3/devices/0000:00:04.0 /sys/kernel/iommu_groups/1/devices/0000:00:02.0 /sys/kernel/iommu_groups/8/devices/0000:00:18.3 /sys/kernel/iommu_groups/8/devices/0000:00:18.1 /sys/kernel/iommu_groups/8/devices/0000:00:18.6 /sys/kernel/iommu_groups/8/devices/0000:00:18.4 /sys/kernel/iommu_groups/8/devices/0000:00:18.2 /sys/kernel/iommu_groups/8/devices/0000:00:18.0 /sys/kernel/iommu_groups/8/devices/0000:00:18.7 /sys/kernel/iommu_groups/8/devices/0000:00:18.5 /sys/kernel/iommu_groups/6/devices/0000:00:08.0 /sys/kernel/iommu_groups/6/devices/0000:14:00.0 /sys/kernel/iommu_groups/6/devices/0000:12:00.4 /sys/kernel/iommu_groups/6/devices/0000:13:00.0 /sys/kernel/iommu_groups/6/devices/0000:00:08.3 /sys/kernel/iommu_groups/6/devices/0000:12:00.0 /sys/kernel/iommu_groups/6/devices/0000:00:08.1 /sys/kernel/iommu_groups/6/devices/0000:12:00.3 /sys/kernel/iommu_groups/6/devices/0000:12:00.1 /sys/kernel/iommu_groups/6/devices/0000:00:08.2 /sys/kernel/iommu_groups/4/devices/0000:00:05.0 /sys/kernel/iommu_groups/2/devices/0000:00:03.1 /sys/kernel/iommu_groups/2/devices/0000:10:00.1 /sys/kernel/iommu_groups/2/devices/0000:0d:00.1 /sys/kernel/iommu_groups/2/devices/0000:00:03.2 /sys/kernel/iommu_groups/2/devices/0000:0f:00.0 /sys/kernel/iommu_groups/2/devices/0000:00:03.0 /sys/kernel/iommu_groups/2/devices/0000:0e:00.0 **/sys/kernel/iommu_groups/2/devices/0000:10:00.0** **/sys/kernel/iommu_groups/2/devices/0000:0d:00.0** /sys/kernel/iommu_groups/0/devices/0000:0c:00.0 /sys/kernel/iommu_groups/0/devices/0000:00:01.2 /sys/kernel/iommu_groups/0/devices/0000:03:03.0 /sys/kernel/iommu_groups/0/devices/0000:02:00.0 /sys/kernel/iommu_groups/0/devices/0000:00:01.0 /sys/kernel/iommu_groups/0/devices/0000:05:05.0 /sys/kernel/iommu_groups/0/devices/0000:0b:00.0 /sys/kernel/iommu_groups/0/devices/0000:08:00.0 /sys/kernel/iommu_groups/0/devices/0000:03:09.0 /sys/kernel/iommu_groups/0/devices/0000:01:00.0 /sys/kernel/iommu_groups/0/devices/0000:0a:00.0 /sys/kernel/iommu_groups/0/devices/0000:05:01.0 /sys/kernel/iommu_groups/0/devices/0000:03:0a.0 /sys/kernel/iommu_groups/0/devices/0000:06:00.0 /sys/kernel/iommu_groups/0/devices/0000:05:07.0 /sys/kernel/iommu_groups/0/devices/0000:03:08.0 /sys/kernel/iommu_groups/0/devices/0000:00:01.1 /sys/kernel/iommu_groups/0/devices/0000:0a:00.3 /sys/kernel/iommu_groups/0/devices/0000:05:03.0 /sys/kernel/iommu_groups/0/devices/0000:0a:00.1 /sys/kernel/iommu_groups/0/devices/0000:04:00.0

lspci output:
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480] 00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU [1022:1481] 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] 00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] 00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] 00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61) 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51) 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440] 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441] 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442] 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443] 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444] 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445] 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446] 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447] 01:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation Device [1987:5016] (rev 01) 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57ad] 03:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a3] 03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4] 03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4] 03:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4] 04:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184] 05:01.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184] 05:03.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184] 05:05.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184] 05:07.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184] 06:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a) 08:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03) 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485] 0a:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] 0a:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] 0b:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51) 0c:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51) 0d:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [Quadro K2000] [10de:0ffe] (rev a1) 0d:00.1 Audio device [0403]: NVIDIA Corporation GK107 HDMI Audio Controller [10de:0e1b] (rev a1) 0e:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1470] (rev c1) 0f:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1471] 10:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XL/XT [Radeon RX Vega 56/64] [1002:687f] (rev c1) 10:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 HDMI Audio [Radeon Vega 56/64] [1002:aaf8] 11:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a] 12:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485] 12:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486] 12:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] 12:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487] 13:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51) 14:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)

grub boot parameters:
GRUB_CMDLINE_LINUX="resume=/dev/mapper/fedora-swap rd.lvm.lv=fedora/root rd.luks.uuid=luks-edacbfc8-4166-47e6-9e2d-02412a63c828 rd.lvm.lv=fedora/swap rhgb amdgpu.ppfeaturemask=0xffffffff nopti nospectre_v2 nospec_store_bypass_disable iommu=1 amd_iommu=on rd.driver.pre=vfio-pci vfio_pci.ids=10de:094c,10de:094c pcie_acs_override=downstream quiet"

/etc/modprobe.d/vfio.conf
install vfio-pci /sbin/vfio-pci-override.sh

/etc/dracut.conf.d/vfio.conf
add_drivers+="vfio vfio_iommu_type1 vfio_pci vfio_virqfd" install_items+="/sbin/vfio-pci-override.sh /usr/bin/dirname"

I am currently going off the guide here: https://qubitrenegade.com/virtualization/kvm/vfio/2019/07/17/VFIO-Fedora-Notes.html

Much appriciated,

  • Aaron

Hi there,

It looks like all lines going directly to CPU are in the same group :frowning:
I am not sure about AMD, but you might try forcing the split to 2x8x for the GPU slots.

PS: you should use a pipe to sort the groups, so we do not have match like retirees do in Florida :slight_smile: