ASRock B550M Pro 4 IOMMU groups - am I toasted?

Software & Operating Systems VFIO
1

After pulling some hardware from collecting dust I discovered a rather unpleasant surprise with my mb: IOMMU groups are brain-dead.

Hardware:

  • AMD Ryzen 5600X (microcode: A20F10/A201016)
  • ASRock B550M Pro4 (BIOS: P2.30)
  • Currently no NVMe but planned

IOMMU groups

Show groups dump 
IOMMU Group 0:
	00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 1:
	00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 2:
	00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 3:
	00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 4:
	00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 5:
	00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 6:
	00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 7:
	00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8:
	00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 9:
	00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 10:
	00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 11:
	00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
	00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 12:
	00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
	00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
	00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
	00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
	00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
	00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
	00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
	00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU Group 13:
	01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ee]
	01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43eb]
	01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43e9]
	02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
	02:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
	02:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
	03:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9235 PCIe 2.0 x2 4-port SATA 6 Gb/s Controller [1b4b:9235] (rev 11)
	04:00.0 Ethernet controller [0200]: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper applications) [8086:105e] (rev 06)
	04:00.1 Ethernet controller [0200]: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper applications) [8086:105e] (rev 06)
	05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU Group 14:
	06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107GL [Quadro P400] [10de:1cb3] (rev a1)
	06:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
IOMMU Group 15:
	07:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 16:
	08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 17:
	08:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU Group 18:
	08:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

Group 14 contains just the GPU connected to the CPU directly (top x16 slot), which is good. What’s terrible is Group 13.

Group 13 contains:

  • built-in USB 2.0 controller
  • built-in SATA controller
  • built-in ethernet
  • external SATA controller in slot 3 (bottom x16 slot)
  • external Intel ethernet in slot 2 (middle x1 slot)

The export above was done under Proxmox with amd_iommu=on iommu=pt. I’m planning to replace the SATA controller with a proper LSI and the Intel dual-gigabit ethernet with a proper SFP cards.

BIOS Configuration

  • Advanced => PCI Configuration
    • Above 4G Decoding: Enable
    • Re-Size BAR Support: Disabled
    • SR=IOV Support: Disabled
  • Advanced => AMD PBS
    • PCIe/GFX Lanes configuration: x16
    • PCIe x16 Bus Interface: Gen3
  • Advanced => AMD CBS => NBIO Common Options
    • IOMMU: Enable
    • DMA Protection: Enable
    • DMAr Support: Auto
    • ACS Enable: Enable
    • PCIe ARI Support: Enable
    • PCIe ARI Enumeration: Enable
    • PCIe AER Cap: Enable
    • SRIS: Enable

All this creates some problems:

  1. I cannot pass the GPU to a VM doing HVEC transcoding
  2. I cannot pass the HBA to a different VM to use ZFS
  3. The ethernet adapter will probably be passed to the same VM as HBA but I cannot detach the built-in realtek one (mgmt interface)

Am I SOL? :frowning:

3

add pcie_acs_override=downstream,multifunction to grub command and you are good to go

here’s mine after adding it (same b550m pro4)

dump

Group 0: [1022:1482] 00:01.0 Host bridge Starship/Matisse PCIe Dummy Host Bridge
Group 1: [1022:1483] [R] 00:01.1 PCI bridge Starship/Matisse GPP Bridge
Group 2: [1022:1483] [R] 00:01.2 PCI bridge Starship/Matisse GPP Bridge
Group 3: [1022:1482] 00:02.0 Host bridge Starship/Matisse PCIe Dummy Host Bridge
Group 4: [1022:1482] 00:03.0 Host bridge Starship/Matisse PCIe Dummy Host Bridge
Group 5: [1022:1483] [R] 00:03.1 PCI bridge Starship/Matisse GPP Bridge
Group 6: [1022:1482] 00:04.0 Host bridge Starship/Matisse PCIe Dummy Host Bridge
Group 7: [1022:1482] 00:05.0 Host bridge Starship/Matisse PCIe Dummy Host Bridge
Group 8: [1022:1482] 00:07.0 Host bridge Starship/Matisse PCIe Dummy Host Bridge
Group 9: [1022:1484] [R] 00:07.1 PCI bridge Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
Group 10: [1022:1482] 00:08.0 Host bridge Starship/Matisse PCIe Dummy Host Bridge
Group 11: [1022:1484] [R] 00:08.1 PCI bridge Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
Group 12: [1022:790b] 00:14.0 SMBus FCH SMBus Controller
[1022:790e] 00:14.3 ISA bridge FCH LPC Bridge
Group 13: [1022:1440] 00:18.0 Host bridge Matisse Device 24: Function 0
[1022:1441] 00:18.1 Host bridge Matisse Device 24: Function 1
[1022:1442] 00:18.2 Host bridge Matisse Device 24: Function 2
[1022:1443] 00:18.3 Host bridge Matisse Device 24: Function 3
[1022:1444] 00:18.4 Host bridge Matisse Device 24: Function 4
[1022:1445] 00:18.5 Host bridge Matisse Device 24: Function 5
[1022:1446] 00:18.6 Host bridge Matisse Device 24: Function 6
[1022:1447] 00:18.7 Host bridge Matisse Device 24: Function 7
Group 14: [144d:a80a] [R] 01:00.0 Non-Volatile memory controller NVMe SSD Controller PM9A1/980PRO
Group 15: [1022:43ee] [R] 02:00.0 USB controller Device 43ee
USB: [26ce:01a2] Bus 001 Device 003 ASRock LED Controller
USB: [05e3:0610] Bus 001 Device 002 Genesys Logic, Inc. Hub
USB: [1d6b:0002] Bus 001 Device 001 Linux Foundation 2.0 root hub
USB: [1d6b:0003] Bus 002 Device 001 Linux Foundation 3.0 root hub
Group 16: [1022:43eb] 02:00.1 SATA controller Device 43eb
Group 17: [1022:43e9] 02:00.2 PCI bridge Device 43e9
Group 18: [1022:43ea] [R] 03:00.0 PCI bridge Device 43ea
Group 19: [1022:43ea] 03:08.0 PCI bridge Device 43ea
Group 20: [10de:0a65] [R] 04:00.0 VGA compatible controller GT218 [GeForce 210]
Group 21: [10de:0be3] 04:00.1 Audio device High Definition Audio Controller
Group 22: [10ec:8168] [R] 05:00.0 Ethernet controller RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
Group 23: [15b3:1007] [R] 06:00.0 Network controller MT27520 Family [ConnectX-3 Pro]
Group 24: [1022:148a] [R] 07:00.0 Non-Essential Instrumentation [1300] Starship/Matisse PCIe Dummy Function
Group 25: [1022:1485] [R] 08:00.0 Non-Essential Instrumentation [1300] Starship/Matisse Reserved SPP
Group 26: [1022:1486] [R] 08:00.1 Encryption controller Starship/Matisse Cryptographic Coprocessor PSPCPP
Group 27: [1022:149c] [R] 08:00.3 USB controller Matisse USB 3.0 Host Controller
USB: [2109:8888] Bus 003 Device 005 VIA Labs, Inc. USB Billboard Device
USB: [2109:2817] Bus 003 Device 004 VIA Labs, Inc. USB2.0 Hub
USB: [046d:c31c] Bus 003 Device 003 Logitech, Inc. Keyboard K120
USB: [174c:2074] Bus 003 Device 002 ASMedia Technology Inc. ASM1074 High-Speed hub
USB: [1d6b:0002] Bus 003 Device 001 Linux Foundation 2.0 root hub
USB: [174c:3074] Bus 004 Device 002 ASMedia Technology Inc. ASM1074 SuperSpeed hub
USB: [1d6b:0003] Bus 004 Device 001 Linux Foundation 3.0 root hub
Group 28: [1022:1487] 08:00.4 Audio device Starship/Matisse HD Audio Controller
Group 29: [15b3:1004] [R] 06:00.1 Network controller MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
Group 30: [15b3:1004] [R] 06:00.2 Network controller MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
Group 31: [15b3:1004] [R] 06:00.3 Network controller MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
Group 32: [15b3:1004] [R] 06:00.4 Network controller MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
Group 33: [15b3:1004] [R] 06:00.5 Network controller MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
Group 34: [15b3:1004] [R] 06:00.6 Network controller MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
Group 35: [15b3:1004] [R] 06:00.7 Network controller MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
Group 36: [15b3:1004] 06:01.0 Network controller MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
Group 37: [15b3:1004] 06:01.1 Network controller MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
Group 38: [15b3:1004] 06:01.2 Network controller MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
Group 39: [15b3:1004] 06:01.3 Network controller MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
Group 40: [15b3:1004] 06:01.4 Network controller MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
Group 41: [15b3:1004] 06:01.5 Network controller MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
Group 42: [15b3:1004] 06:01.6 Network controller MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
Group 43: [15b3:1004] 06:01.7 Network controller MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
Group 44: [15b3:1004] 06:02.0 Network controller MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

1 Like
4

Welcome to the forum @kiler129 @SuperHacker !

With the given configuration, I would just use Proxmox built-in ZFS. The ethernet adapter, you are better off making a bridge in Proxmox and passing a virtual ethernet to the VM. It’s not a big overhead for eth nics. Or you could install the virtio ethernet driver and use the virtio ethernet adapter in proxmox.

Why would you not be able to pass the GPU for HEVC? The GPU seems to be in a proper separated group, so it would be all you need. Besides, I recall one should be able to pass a USB device to the host, instead of the entire USB controller, but I could be wrong. I used to pass a USB with a HDD enclosure to a VM on a NUC running Proxmox without any IOMMU finagling.