AM5 IOMMU VFIO - Best Motherboard

It’s time to upgrade and looks like it’s the best time to do so.

I have anUnraid server that I use for plex and a couple of VM’s, but I will be using a windows 11 VM as my daily driver.

I’m looking at the new AMD 7000 series and I don’t know what motherboard is going to give me the best IOMMU separation, so passing through my GPU, USBs, storage, and possibly motherboard sound to my Windows 1 VM.

If anyone has tested their AM5 chipset motherboard, please send me your IOMMU screenshots. At least some advice on which brand to go with.

Also, if you have a great recommendation on a AM4 chipset motherboard that will have great IOMMU separation, please send that as well.

Another question:

Are the IOMMU grouping defined by purely the chipset?

Too early to know for B650. Also not sure if the B650/B650E difference may play into that.

@Patrick_Layton Asrock X570 Taichi motherboard has the best virtualization features in their BIOS. While I didn’t mess with graphic card pass-through due to the lack of having two graphic cards, using VirtualBox, I was able to create a few virtual machines inside a virtual machine. If I hadn’t ascendingly destroyed the computer’s power supply, I would still be using that system instead of my current one.

1 Like

IOMMU groups on the Asus B650M TUF Gaming Wifi mATX (from Arch Linux script):

### ASUS B650M-PLUS TUF Gaming WiFi AMD AM5 microATX Motherboard
 
### archlinux iommu script output
 
IOMMU Group 0:
	00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 1:
	00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 2:
	00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 3:
	00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 4:
	00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 5:
	00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 6:
	00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 7:
	00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
IOMMU Group 8:
	00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
IOMMU Group 9:
	00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
	00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 10:
	00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e0]
	00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e1]
	00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e2]
	00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e3]
	00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e4]
	00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e5]
	00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e6]
	00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e7]
IOMMU Group 11:
	01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU Group 12:
	02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f4] (rev 01)
IOMMU Group 13:
	03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
IOMMU Group 14:
	03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
IOMMU Group 15:
	03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
IOMMU Group 16:
	03:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
	07:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
IOMMU Group 17:
	03:0b.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
	08:00.0 Network controller [0280]: MEDIATEK Corp. MT7921 802.11ax PCI Express Wireless Network Adapter [14c3:7961]
IOMMU Group 18:
	03:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
	09:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f7] (rev 01)
IOMMU Group 19:
	03:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
	0a:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f6] (rev 01)
IOMMU Group 20:
	0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:164e] (rev c3)
IOMMU Group 21:
	0b:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1640]
IOMMU Group 22:
	0b:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP [1022:1649]
IOMMU Group 23:
	0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b6]
IOMMU Group 24:
	0b:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b7]
IOMMU Group 25:
	0b:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller [1022:15e3]
IOMMU Group 26:
	0c:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b8]
 
2 Likes

are you sure you don’t have the acs patch enabled in your kernel?

From what I can tell (I dont have many b650 mobos yet) the apu video/audio should be in one group, and chipset usb hubs should be grouped together as well.

3 Likes

If you’re looking for one to get from ASRock, the PG Riptide mATX is what I’m dying to know the IOMMU to. If it’s good, I’m gonna get it and wait for the Phoenix APU AM5 chips to start my homelab.

The Asus ProArt X670E-CREATOR WIFI is the MB I’m looking at for an upcoming build as it has the connectivity I’m looking for as far as PCIe slot layout, M.2s and USB4(which may or may not be Thunderbolt). So if anyone gets one of these and can dump the IOMMU groups that it would be much appreciated.

@wendell

Vanilla Ubuntu 22.04 LTS install. Only updated and rebooted. 100% sure it’s the stock Ubuntu kernel.

Really wish I had a DDR5 ECC UDIMM to test with. I’m not willing to pay Dell $350 for a 16GB stick, ugh.

# uname -r; md5sum /boot/initrd.img-5.15.0-48-generic ; cat /etc/os-release
5.15.0-48-generic
3e00ed9148431d4b56c3e4d6a277b3bd /boot/initrd.img-5.15.0-48-generic

PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

# lspci -tvkP
-[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Device 14d8
           +-00.2  Advanced Micro Devices, Inc. [AMD] Device 14d9
           +-01.0  Advanced Micro Devices, Inc. [AMD] Device 14da
           +-01.2-[01]----00.0  Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
           +-02.0  Advanced Micro Devices, Inc. [AMD] Device 14da
           +-02.1-[02-0a]----00.0-[03-0a]--+-00.0-[04]--
           |                               +-08.0-[05]--
           |                               +-09.0-[06]--
           |                               +-0a.0-[07]----00.0  Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller
           |                               +-0b.0-[08]----00.0  MEDIATEK Corp. MT7921 802.11ax PCI Express Wireless Network Adapter
           |                               +-0c.0-[09]----00.0  Advanced Micro Devices, Inc. [AMD] Device 43f7
           |                               \-0d.0-[0a]----00.0  Advanced Micro Devices, Inc. [AMD] Device 43f6
           +-03.0  Advanced Micro Devices, Inc. [AMD] Device 14da
           +-04.0  Advanced Micro Devices, Inc. [AMD] Device 14da
           +-08.0  Advanced Micro Devices, Inc. [AMD] Device 14da
           +-08.1-[0b]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Device 164e
           |            +-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Device 1640
           |            +-00.2  Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP
           |            +-00.3  Advanced Micro Devices, Inc. [AMD] Device 15b6
           |            +-00.4  Advanced Micro Devices, Inc. [AMD] Device 15b7
           |            \-00.6  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller
           +-08.3-[0c]----00.0  Advanced Micro Devices, Inc. [AMD] Device 15b8
           +-14.0  Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
           +-14.3  Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
           +-18.0  Advanced Micro Devices, Inc. [AMD] Device 14e0
           +-18.1  Advanced Micro Devices, Inc. [AMD] Device 14e1
           +-18.2  Advanced Micro Devices, Inc. [AMD] Device 14e2
           +-18.3  Advanced Micro Devices, Inc. [AMD] Device 14e3
           +-18.4  Advanced Micro Devices, Inc. [AMD] Device 14e4
           +-18.5  Advanced Micro Devices, Inc. [AMD] Device 14e5
           +-18.6  Advanced Micro Devices, Inc. [AMD] Device 14e6
           \-18.7  Advanced Micro Devices, Inc. [AMD] Device 14e7



1 Like

Here you go.

IOMMU Group 0:
        00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 1:
        00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 10:
        00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
IOMMU Group 11:
        00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
        00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 12:
        00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e0]
        00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e1]
        00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e2]
        00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e3]
        00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e4]
        00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e5]
        00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e6]
        00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e7]
IOMMU Group 13:
        01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c1)
IOMMU Group 14:
        02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
IOMMU Group 15:
        03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M] [1002:73df] (rev c1)
IOMMU Group 16:
        03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
IOMMU Group 17:
        04:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO [144d:a80a]
IOMMU Group 18:
        05:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f4] (rev 01)
IOMMU Group 19:
        06:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
IOMMU Group 2:
        00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 20:
        06:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        08:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f4] (rev 01)
        09:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        09:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        09:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        09:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        09:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        09:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        09:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        0a:00.0 Network controller [0280]: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter [14c3:0616]
        0b:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)
        0c:00.0 Ethernet controller [0200]: Aquantia Corp. AQC113CS NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] [1d6a:94c0] (rev 03)
        0e:00.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        0f:00.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        0f:01.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        0f:02.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        0f:03.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        10:00.0 USB controller [0c03]: Intel Corporation Thunderbolt 4 NHI [Maple Ridge 4C 2020] [8086:1137]
        3c:00.0 USB controller [0c03]: Intel Corporation Thunderbolt 4 USB Controller [Maple Ridge 4C 2020] [8086:1138]
        69:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f7] (rev 01)
IOMMU Group 21:
        06:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        6a:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f7] (rev 01)
IOMMU Group 22:
        06:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        6b:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f6] (rev 01)
IOMMU Group 23:
        6c:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation E12 NVMe Controller [1987:5012] (rev 01)
IOMMU Group 24:
        6d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:14de] (rev c1)
IOMMU Group 25:
        6d:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP [1022:1649]
IOMMU Group 26:
        6d:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b6]
IOMMU Group 27:
        6d:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b7]
IOMMU Group 28:
        6d:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller [1022:15e3]
IOMMU Group 29:
        6e:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b8]
IOMMU Group 3:
        00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 4:
        00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 5:
        00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 6:
        00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 7:
        00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 8:
        00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 9:
        00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
5 Likes

So that IOMMU dump is from the Asus ProArt X670E-CREATOR WIFI? Thanks!

Yeah that’s from the X670E-Creator. Finally managed to get one last Friday when they finally came in stock in the U.K. As you can see too it is thunderbolt.

What haben’s if you use a Thunderbolt to PCIe enclosure, do these cards get a own group or are they lumped together with other devices? - this wouldn‘t be very useful.

I’d imagine it stays in the same group, everything in group 20 I am guessing is chipset connected not entirely sure. An external graphics card or a need to pass through anything connected to the thunderbolt 4 ports is beyond my own use case and not something I’m looking to do.

After thinking of it more both will 100% be in group 20 still as they are also Display Port outs to be used with the integrated graphics on the 7000Series or if you use the Display Port input feature on the X670E-Creator.

It’s just a general question. I‘ve read that passed through PCIe devices on the thunderbolt port are available as separate devices while the host takes care about the thunderbolt controller. I just wanted to make sure before I blow 500 bucks on this that the groups are sorted.

That looks promising, thanks Blissing. Have you had any success passing through any pci cards besides the gpu? Looking to pass through a pci usb card and wondering if the small pci slot on the bottom of the board is in its own iommu group.

Not a card specifically but have passed a controller on the motherboard before. Passing a usb card shouldn’t be difficult as long as the IOMMU grouping is fine.