Return to Level1Techs.com

Threadripper 3970X GPU Passthrough Issues

I just got a 3970X with a Gigabyte TRX40 Aorus Xtreme motherboard and have been having problems getting GPU passthrough working well and I’m not really sure what to try next.

System Config

  • AMD Threadripper 3970X
  • Gigabyte TRX40 Aorus Xtreme
  • 64GB RAM
  • Nvidia 2080 Ti - Host GPU
  • MSI AMD 5700 - Guest GPU
  • Arch Linux (5.5.3)

There are multiple problems I’m running into, but the primary problem I have is when attempting to boot my macOS VM. My VM was set up by following the macOS-Simple-KVM guide on Github and works fine when using the virtualized graphics. When I attempt to pass my GPU through though, the guest hangs during the boot process:

I found a post on here by someone else that had the same problem, and the solution was to pass through a PCIE upstream and downstream device as well. I tried it, but I got the exact same result. I thought it might be related to GPU kexts, so I added lilu and WhateverGreen to my EFI partition, along with agdpmod=pikera to my boot args, which didn’t change the result either way.

I am able to pass the GPU through to Windows. The VM see’s the GPU and installs drivers, though it usually bluescreens after ~10 minutes of sitting idle due to a ‘SYSTEM_SERVICE_EXCEPTION’ where gdrv2.sys failed. This happens when I only passthrough my NVME drive and use virtualized graphics. I’m not sure if that’s relevant to this macOS issue I’m having.

After I shut down one of my VMs that had the GPU attached, I get the infamous “127 error” due to the reset bug. I applied the Navi reset patch to my kernel, and that error goes away, though booting up the VM again still results in an unresponsive GPU. I have to sleep/reboot the host in order to get the GPU to work in a VM again.

I have an old RX470 I attempted to get working, but had no luck there either. I couldn’t get any video out of it passed through to either Windows or macOS. I recently read a post somewhere about someone else having difficulties getting the RX 580 working with his new Threadripper, but I’m unable to find that now.

I’d appreciate any ideas or suggestions about how I can resolve this, particularly for my macOS VM. For Windows I plan on passing through an Nvidia card.

GRUB Command Line

GRUB_CMDLINE_LINUX_DEFAULT="loglevel=3 quiet module_blacklist=amdgpu video=efifb:off amd_iommu=on iommu=pt vfio-pci.ids=1002:731f,1002:ab38"

Full output of lspci -nnk

00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU [1022:1481]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU [1022:1481]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
        Kernel driver in use: pcieport
00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
        Kernel driver in use: pcieport
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
        Kernel driver in use: pcieport
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
        Kernel driver in use: pcieport
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
        Subsystem: Gigabyte Technology Co., Ltd FCH SMBus Controller [1458:5001]
        Kernel modules: i2c_piix4, sp5100_tco
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
        Subsystem: Gigabyte Technology Co., Ltd FCH LPC Bridge [1458:5001]
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0 [1022:1490]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1 [1022:1491]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2 [1022:1492]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3 [1022:1493]
        Kernel driver in use: k10temp
        Kernel modules: k10temp
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4 [1022:1494]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5 [1022:1495]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6 [1022:1496]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7 [1022:1497]
01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804]
        Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a801]
        Kernel driver in use: nvme
02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804]
        Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a801]
        Kernel driver in use: nvme
03:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
04:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
04:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c]
        Subsystem: Gigabyte Technology Co., Ltd Starship USB 3.0 Host Controller [1458:5007]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
20:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
20:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU [1022:1481]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU [1022:1481]
20:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
20:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
20:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
20:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
        Kernel driver in use: pcieport
20:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
20:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
20:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
20:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
        Kernel driver in use: pcieport
20:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
20:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
        Kernel driver in use: pcieport
21:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti Rev. A] [10de:1e07] (rev a1)
        Subsystem: ZOTAC International (MCO) Ltd. TU102 [GeForce RTX 2080 Ti Rev. A] [19da:6513]
        Kernel driver in use: nvidia
        Kernel modules: nouveau, nvidia_drm, nvidia
21:00.1 Audio device [0403]: NVIDIA Corporation TU102 High Definition Audio Controller [10de:10f7] (rev a1)
        Subsystem: ZOTAC International (MCO) Ltd. TU102 High Definition Audio Controller [19da:6513]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
21:00.2 USB controller [0c03]: NVIDIA Corporation TU102 USB 3.1 Host Controller [10de:1ad6] (rev a1)
        Subsystem: ZOTAC International (MCO) Ltd. TU102 USB 3.1 Host Controller [19da:6513]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
21:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU102 USB Type-C UCSI Controller [10de:1ad7] (rev a1)
        Subsystem: ZOTAC International (MCO) Ltd. TU102 USB Type-C UCSI Controller [19da:6513]
        Kernel driver in use: nvidia-gpu
        Kernel modules: i2c_nvidia_gpu
22:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
23:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
23:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
        Kernel driver in use: ccp
        Kernel modules: ccp
23:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c]
        Subsystem: Gigabyte Technology Co., Ltd Starship USB 3.0 Host Controller [1458:5007]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
23:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:d102]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
40:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
40:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU [1022:1481]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU [1022:1481]
40:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
40:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
        Kernel driver in use: pcieport
40:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
40:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
40:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
        Kernel driver in use: pcieport
40:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
40:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
40:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
40:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
        Kernel driver in use: pcieport
40:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
40:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
        Kernel driver in use: pcieport
41:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad]
        Kernel driver in use: pcieport
42:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
        Kernel driver in use: pcieport
42:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
        Kernel driver in use: pcieport
42:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
        Kernel driver in use: pcieport
42:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
        Kernel driver in use: pcieport
42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
        Kernel driver in use: pcieport
42:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
        Kernel driver in use: pcieport
42:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
        Kernel driver in use: pcieport
43:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10G X550T [8086:1563] (rev 01)
        Subsystem: Gigabyte Technology Co., Ltd Ethernet Controller 10G X550T [1458:e000]
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe
43:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10G X550T [8086:1563] (rev 01)
        Subsystem: Gigabyte Technology Co., Ltd Ethernet Controller 10G X550T [1458:e000]
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe
45:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller [1b21:2142]
        Subsystem: Gigabyte Technology Co., Ltd ASM2142 USB 3.1 Host Controller [1458:5007]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
46:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
        DeviceName: Onboard LAN RTL8111
        Subsystem: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:1060]
        Kernel driver in use: ahci
        Kernel modules: ahci
47:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a)
        Subsystem: Intel Corporation Wi-Fi 6 AX200 [8086:0084]
        Kernel driver in use: iwlwifi
        Kernel modules: iwlwifi
48:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
48:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:1486]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
48:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:148c]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
49:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
        DeviceName: X570/590 SATA0
        Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901]
        Kernel driver in use: ahci
        Kernel modules: ahci
4a:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
        DeviceName: X570/590 SATA1
        Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901]
        Kernel driver in use: ahci
        Kernel modules: ahci
4b:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c4)
        Kernel driver in use: pcieport
4c:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
        Kernel driver in use: pcieport
4d:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] [1002:731f] (rev c4)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] [1462:3811]
        Kernel modules: amdgpu
4d:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio [1002:ab38]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio [1002:ab38]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
4e:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
4f:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
60:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
60:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU [1022:1481]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU [1022:1481]
60:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
60:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
60:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
60:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
60:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
60:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
60:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
        Kernel driver in use: pcieport
60:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
60:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
        Kernel driver in use: pcieport
61:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
62:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]

dmesg | grep -i iommu

[    4.317668] iommu: Default domain type: Passthrough (set via kernel command line)
[    4.419727] pci 0000:60:00.2: AMD-Vi: IOMMU performance counters supported
[    4.419760] pci 0000:40:00.2: AMD-Vi: IOMMU performance counters supported
[    4.419780] pci 0000:20:00.2: AMD-Vi: IOMMU performance counters supported
[    4.419798] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    4.419844] pci 0000:00:01.0: Adding to iommu group 0
[    4.419869] pci 0000:00:01.1: Adding to iommu group 1
[    4.419895] pci 0000:00:01.2: Adding to iommu group 2
[    4.419916] pci 0000:00:02.0: Adding to iommu group 3
[    4.419938] pci 0000:00:03.0: Adding to iommu group 4
[    4.419959] pci 0000:00:04.0: Adding to iommu group 5
[    4.419980] pci 0000:00:05.0: Adding to iommu group 6
[    4.420004] pci 0000:00:07.0: Adding to iommu group 7
[    4.420028] pci 0000:00:07.1: Adding to iommu group 8
[    4.420053] pci 0000:00:08.0: Adding to iommu group 9
[    4.420076] pci 0000:00:08.1: Adding to iommu group 10
[    4.420102] pci 0000:00:14.0: Adding to iommu group 11
[    4.420119] pci 0000:00:14.3: Adding to iommu group 11
[    4.420180] pci 0000:00:18.0: Adding to iommu group 12
[    4.420195] pci 0000:00:18.1: Adding to iommu group 12
[    4.420210] pci 0000:00:18.2: Adding to iommu group 12
[    4.420226] pci 0000:00:18.3: Adding to iommu group 12
[    4.420242] pci 0000:00:18.4: Adding to iommu group 12
[    4.420259] pci 0000:00:18.5: Adding to iommu group 12
[    4.420273] pci 0000:00:18.6: Adding to iommu group 12
[    4.420288] pci 0000:00:18.7: Adding to iommu group 12
[    4.420313] pci 0000:01:00.0: Adding to iommu group 13
[    4.420338] pci 0000:02:00.0: Adding to iommu group 14
[    4.420360] pci 0000:03:00.0: Adding to iommu group 15
[    4.420386] pci 0000:04:00.0: Adding to iommu group 16
[    4.420410] pci 0000:04:00.3: Adding to iommu group 17
[    4.420431] pci 0000:20:01.0: Adding to iommu group 18
[    4.420452] pci 0000:20:02.0: Adding to iommu group 19
[    4.420477] pci 0000:20:03.0: Adding to iommu group 20
[    4.420506] pci 0000:20:03.1: Adding to iommu group 21
[    4.420528] pci 0000:20:04.0: Adding to iommu group 22
[    4.420550] pci 0000:20:05.0: Adding to iommu group 23
[    4.420573] pci 0000:20:07.0: Adding to iommu group 24
[    4.420595] pci 0000:20:07.1: Adding to iommu group 25
[    4.420619] pci 0000:20:08.0: Adding to iommu group 26
[    4.420640] pci 0000:20:08.1: Adding to iommu group 27
[    4.420699] pci 0000:21:00.0: Adding to iommu group 28
[    4.420729] pci 0000:21:00.1: Adding to iommu group 28
[    4.420758] pci 0000:21:00.2: Adding to iommu group 28
[    4.420788] pci 0000:21:00.3: Adding to iommu group 28
[    4.420809] pci 0000:22:00.0: Adding to iommu group 29
[    4.420834] pci 0000:23:00.0: Adding to iommu group 30
[    4.420858] pci 0000:23:00.1: Adding to iommu group 31
[    4.420881] pci 0000:23:00.3: Adding to iommu group 32
[    4.420904] pci 0000:23:00.4: Adding to iommu group 33
[    4.420929] pci 0000:40:01.0: Adding to iommu group 34
[    4.420952] pci 0000:40:01.1: Adding to iommu group 35
[    4.420975] pci 0000:40:02.0: Adding to iommu group 36
[    4.421000] pci 0000:40:03.0: Adding to iommu group 37
[    4.421023] pci 0000:40:03.1: Adding to iommu group 38
[    4.421044] pci 0000:40:04.0: Adding to iommu group 39
[    4.421065] pci 0000:40:05.0: Adding to iommu group 40
[    4.421089] pci 0000:40:07.0: Adding to iommu group 41
[    4.421109] pci 0000:40:07.1: Adding to iommu group 42
[    4.421133] pci 0000:40:08.0: Adding to iommu group 43
[    4.421156] pci 0000:40:08.1: Adding to iommu group 44
[    4.421182] pci 0000:41:00.0: Adding to iommu group 45
[    4.421257] pci 0000:42:02.0: Adding to iommu group 46
[    4.421332] pci 0000:42:03.0: Adding to iommu group 47
[    4.421407] pci 0000:42:04.0: Adding to iommu group 48
[    4.421482] pci 0000:42:05.0: Adding to iommu group 49
[    4.421519] pci 0000:42:08.0: Adding to iommu group 50
[    4.421555] pci 0000:42:09.0: Adding to iommu group 51
[    4.421592] pci 0000:42:0a.0: Adding to iommu group 52
[    4.421674] pci 0000:43:00.0: Adding to iommu group 53
[    4.421753] pci 0000:43:00.1: Adding to iommu group 54
[    4.421814] pci 0000:45:00.0: Adding to iommu group 55
[    4.421883] pci 0000:46:00.0: Adding to iommu group 56
[    4.421944] pci 0000:47:00.0: Adding to iommu group 57
[    4.421965] pci 0000:48:00.0: Adding to iommu group 50
[    4.421991] pci 0000:48:00.1: Adding to iommu group 50
[    4.422006] pci 0000:48:00.3: Adding to iommu group 50
[    4.422028] pci 0000:49:00.0: Adding to iommu group 51
[    4.422049] pci 0000:4a:00.0: Adding to iommu group 52
[    4.422074] pci 0000:4b:00.0: Adding to iommu group 58
[    4.422098] pci 0000:4c:00.0: Adding to iommu group 59
[    4.422156] pci 0000:4d:00.0: Adding to iommu group 60
[    4.422188] pci 0000:4d:00.1: Adding to iommu group 61
[    4.422210] pci 0000:4e:00.0: Adding to iommu group 62
[    4.422233] pci 0000:4f:00.0: Adding to iommu group 63
[    4.422255] pci 0000:60:01.0: Adding to iommu group 64
[    4.422279] pci 0000:60:02.0: Adding to iommu group 65
[    4.422301] pci 0000:60:03.0: Adding to iommu group 66
[    4.422322] pci 0000:60:04.0: Adding to iommu group 67
[    4.422343] pci 0000:60:05.0: Adding to iommu group 68
[    4.422367] pci 0000:60:07.0: Adding to iommu group 69
[    4.422387] pci 0000:60:07.1: Adding to iommu group 70
[    4.422411] pci 0000:60:08.0: Adding to iommu group 71
[    4.422432] pci 0000:60:08.1: Adding to iommu group 72
[    4.422453] pci 0000:61:00.0: Adding to iommu group 73
[    4.422476] pci 0000:62:00.0: Adding to iommu group 74
[    4.422621] pci 0000:60:00.2: AMD-Vi: Found IOMMU cap 0x40
[    4.422624] pci 0000:40:00.2: AMD-Vi: Found IOMMU cap 0x40
[    4.422626] pci 0000:20:00.2: AMD-Vi: Found IOMMU cap 0x40
[    4.422628] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    4.426446] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    4.426478] perf/amd_iommu: Detected AMD IOMMU #1 (2 banks, 4 counters/bank).
[    4.426510] perf/amd_iommu: Detected AMD IOMMU #2 (2 banks, 4 counters/bank).
[    4.426540] perf/amd_iommu: Detected AMD IOMMU #3 (2 banks, 4 counters/bank).
[    4.441228] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <[email protected]>

QEMU Command Line

I’ve tried using virt-manager as well, it didn’t change the result.

qemu-system-x86_64 \
    -enable-kvm \
    -m 8G \
    -machine q35,accel=kvm \
    -smp cores=4,threads=2,sockets=2 \
    -cpu Penryn,vendor=GenuineIntel,kvm=on,+sse3,+sse4.2,+aes,+xsave,+avx,+xsaveopt,+xsavec,+xgetbv1,+avx2,+bmi2,+smep,+bmi1,+fma,+movbe,+invtsc \
    -device isa-applesmc,osk="$OSK" \
    -smbios type=2 \
    -drive if=pflash,format=raw,readonly,file="$OVMF/OVMF_CODE.fd" \
    -drive if=pflash,format=raw,file="$OVMF/OVMF_VARS-1024x768.fd" \
    -vga none \
    -device ich9-intel-hda -device hda-output \
    -usb -device usb-kbd -device usb-mouse \
    -netdev user,id=net0 \
    -device e1000-82545em,netdev=net0,id=net0,mac=52:54:00:c9:18:27 \
    -device ich9-ahci,id=sata \
    -drive id=SystemDisk,if=none,file=MyDisk.qcow2 \
    -device ide-hd,bus=sata.4,drive=SystemDisk \
    -device pcie-root-port,bus=pcie.0,multifunction=on,port=1,chassis=1,id=port.1 \
    -device ioh3420,id=root_port1,chassis=1,slot=2,bus=pcie.0 \
    -device x3130-upstream,id=upstream_port1,bus=root_port1     \
    -device xio3130-downstream,id=downstream_port1,chassis=11,slot=21,bus=upstream_port1 \
    -device vfio-pci,host=4d:00.0,bus=downstream_port1,multifunction=on \
    -device vfio-pci,host=4d:00.1,bus=downstream_port1 \

Have you made any progress on this? I was going to revisit and see if I could replicate your setup since I have the same parts.

1 Like

I did a similar setup with a threadripper 3960x and the gigabyte trx aorus xtreme motherboard. I used an ACS patched kernel and used the grub option pcie_acs_override=downstream I don’t know if this will make a difference because I intentionally used a compatible GPU (geforce 700 series).

I have really exactly the same hardware. Did you make it? Any news?