Choosing VFIO AM5 MB 2025

Hardware Hub Motherboards
, , , ,
1

I am having troubles choosing motherboard and GPU for my build. I want a Linux first experience and I want VFIO and Looking Glass to work. My plan is to pass the dGPU to a Windows VM and use the iGPU for the host. If possible detach the dGPU from the VM and using it in the host. I am also curious about running a headless host and pass the iGPU to a Linux VM. Flexibility would be nice. Initially I will only get one dGPU. I may expand it to two in the future if I feel I have to, to make things work better.

I am planing to get a 3840 x 2160 monitor that can be switched to FHD mode when gaming. I’d like to do gaming in Linux when possible and in a Windows VM when necessary. Also will do CAD work in Windows VM.

I’d like to stay with AMD GPU if possible. I like that their driver is open source. I’ve been fighting Nvidia drivers in the past.

I understand that I need iGPU, dGPU1 and dGPU2 to be in different IOMMU-groups and that this is not something MB manufacturers list in the specs, making it hard to choose a board. I also need an IOMMU toggle in the bios. Should I be mailing the manufacturer, asking them for the groups?

Choosing an AM5 Chipset seems hard to me as well. I don’t wanna get something that impedes my goal of having a nice Linux and FVIO/Looking glass experience.

What exactly am I looking for when it comes to the PCU-e lanes? I see that it is common for one PCI-e slot to be wired directly to the CPU but extra slots might go to the chipset. Does it matter if a second dGPU goes through the chipset? I don’t understand what I need here and what pitfalls there are.

So I need help choosing something appropriate.

I have been eyeing the following parts:

CPU:

9950x3d

GPU:

RX 9060 TX
RX 9070

Motherboard:

ASRock X870E Nova WIFI
ASRock X870 Nova WiFi
MSI MAG X870E Tomahawk WIFI

I am aware that there is something called the reset bug for AMD GPUs But I thought it might be worth living with having to reboot once in a while.

2

It is possible to use the GPUs on the host while they are not used by a VM, but it can be tricky depending on the desktop environment / compositor. For instance, the compositor may not support attaching or detaching, and may instead require a restart of the session.
At least I saw one positive report today: https://old.reddit.com/r/VFIO/comments/1ohll1v/9070_xt_passthrough_working_with_one_small_issue/

There are some guides for iGPU passthrough (on Ryzen 7000/9000), I haven’t tried myself though.

The IOMMU groups often are fine, though some of the chipset ports can be problematic at times (and it’s hard to know in advance). From what I’ve heard, the behaviour is quite similar across AM5 boards but may depend on the AGESA version. At least in the early days, single-chipset models (B6…, B8…, X870) were said to be less problematic on the chipset ports, while the dual-chipset ones (X670, X670E, X870E) occasionally had (or have?) some more chipset devices thrown into the same group.

Motherboard selection: If you want a second dGPU, make sure you get a board that gives you enough PCIe lanes on an additional port (it has to be x16 mechanically) so you don’t bottleneck the GPU too much.
In your selection, only the MSI X870 Tomahawk has PCIe 4 x4 on an x16 port (it appears to be the one at the bottom, connected via the chipset). Depending on your workload, that may be enough already.
There are a few boards with x8/x8 bifurcation (i.e. the ability to split the main x16 CPU port into two x8 ports on demand), such as:

  • ASUS ProArt B650-Creator (PCIe 4)
  • MSI MPG X670E Carbon WIFI (PCIe 5)
  • ASRock X870(E) Taichi variants (PCIe 5)
  • ASUS ProArt X870E-Creator WiFi (PCIe 5 (like the X670E model)
  • Gigabyte X870 AORUS (Tachyon|Master X3D) ICE (PCIe 5)

I’m not sure whether that is worth it, as most of those boards are on the more expensive side. Putting the RX 9060 (XT) on a PCIe 4.0 x4 chipset port may be good enough already, should be comparable to OCuLink on some mini PCs / handhelds (or maybe a tad slower since the entire chipset is connected by 4.0 x4).

2 Likes
3

@Precific Thanks a bunch for your reply! It was very helpful.

ASRock X870 Nova WiFi does have a PCIe 4.0 x4 through chipset.

Some more questions if you have time:

I haven’t observed any IOMMU groupings where two dGPUs were connected. On motherboards that bifurcate a 16× PCIe slot into two 8× slots, do both GPUs typically reside in the same IOMMU group? This post seems to suggest that. ASRock X870E Taichi Lite for example.

When a PCIe 5.0 x16 slot isn’t bifurcated, using all 16 lanes for one GPU seems excessive (correct me if I am wrong). I like that the CPU’s PCIe lanes can be used for other things, like M.2 storage.

Gigabyte X870E Aorus Elite WIFI7
The block diagram and description for this board are confusing. It seems the CPU PCIe 5.0 bus carries three M.2 slots and one GPU slot, with lanes allocated as GPU x8, M2B x4, and M2C x4. There’s also M2A at x4 on the same bus. The manual’s note of PCIe 5.0 x4/x2 for the M.2 slot only increase my confusion.

ASRock X870 Nova WiFi & MSI MAG X870E Tomahawk WIFI
On these two boards, USB4 and the M.2 slot share a PCIe 5.0 x4 link. By disabling USB4 to grant the M.2 slot all four lanes, will the USB4 ports still provide DisplayPort output when disabled?

4

That is at best outdated information… On AM5 these x16/x8 bifurcated slots are CPU-connected so they should always behave the same independent of motherboard implementation. On my AM5 system these are always in separate groups, no matter if I have 1 GPU and 1 NIC or 2 GPUs. The issues pop up on X670(E)/X870E, and as far as I’ve seen only on the stuff connected to the second daisy-chained chipset. These are my groups currently on a ProArt X670E with 2 GPUs in the top slots and a NIC in the last chipset slot. I have 3 SSDs, two of which in the CPU-slots, one in a chipset-slot. All of the SSDs and GPUs are isolated, but the NIC isn’t because it’s slot is fed through the second daisey chained chipset.

IOMMU Group 0:
        00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 1:
        00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
IOMMU Group 2:
        00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
IOMMU Group 3:
        00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
IOMMU Group 4:
        00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 5:
        00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
IOMMU Group 6:
        00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
IOMMU Group 7:
        00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 8:
        00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 9:
        00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 10:
        00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A] [1022:14dd]
IOMMU Group 11:
        00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A] [1022:14dd]
IOMMU Group 12:
        00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
        00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 13:
        00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 0 [1022:14e0]
        00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 1 [1022:14e1]
        00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 2 [1022:14e2]
        00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 3 [1022:14e3]
        00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 4 [1022:14e4]
        00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 5 [1022:14e5]
        00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 6 [1022:14e6]
        00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 7 [1022:14e7]
IOMMU Group 14:
        01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] [10de:2504] (rev a1)
        01:00.1 Audio device [0403]: NVIDIA Corporation GA106 High Definition Audio Controller [10de:228e] (rev a1)
IOMMU Group 15:
        02:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. KC3000/FURY Renegade NVMe SSD [E18] [2646:5013] (rev 01)
IOMMU Group 16:
        03:00.0 PCI bridge [0604]: Intel Corporation Device [8086:e2ff] (rev 01)
IOMMU Group 17:
        04:01.0 PCI bridge [0604]: Intel Corporation Device [8086:e2f0]
IOMMU Group 18:
        04:02.0 PCI bridge [0604]: Intel Corporation Device [8086:e2f1]
IOMMU Group 19:
        05:00.0 VGA compatible controller [0300]: Intel Corporation Battlemage G21 [Intel Graphics] [8086:e212]
IOMMU Group 20:
        06:00.0 Audio device [0403]: Intel Corporation Device [8086:e2f7]
IOMMU Group 21:
        07:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Upstream Port [1022:43f4] (rev 01)
IOMMU Group 22:
        08:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        09:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. NV2 NVMe SSD [SM2267XT] (DRAM-less) [2646:5017] (rev 03)
IOMMU Group 23:
        08:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        0a:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Upstream Port [1022:43f4] (rev 01)
        0b:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        0b:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        0b:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        0b:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        0b:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        0b:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        0b:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        0b:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        0b:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        0c:00.0 Network controller [0280]: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter [14c3:0616]
        0d:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)
        0e:00.0 Ethernet controller [0200]: Aquantia Corp. AQtion AQC113CS NBase-T/IEEE 802.3an Ethernet Controller [Antigua 10G] [1d6a:94c0] (rev 03)
        10:00.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        11:00.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        11:01.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        11:02.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        11:03.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        12:00.0 USB controller [0c03]: Intel Corporation Thunderbolt 4 NHI [Maple Ridge 4C 2020] [8086:1137]
        3e:00.0 USB controller [0c03]: Intel Corporation Thunderbolt 4 USB Controller [Maple Ridge 4C 2020] [8086:1138]
        6b:00.0 Ethernet controller [0200]: Mellanox Technologies MT27500 Family [ConnectX-3] [15b3:1003]
        6c:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset USB 3.2 Controller [1022:43f7] (rev 01)
        6d:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller [1022:43f6] (rev 01)
IOMMU Group 24:
        08:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        6e:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset USB 3.2 Controller [1022:43f7] (rev 01)
IOMMU Group 25:
        08:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
        6f:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller [1022:43f6] (rev 01)
IOMMU Group 26:
        70:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. NV2 NVMe SSD [SM2267XT] (DRAM-less) [2646:5017] (rev 03)
IOMMU Group 27:
        71:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raphael [1002:164e] (rev c1)
IOMMU Group 28:
        71:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Radeon High Definition Audio Controller [Rembrandt/Strix] [1002:1640]
IOMMU Group 29:
        71:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 19h PSP/CCP [1022:1649]
IOMMU Group 30:
        71:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 3.1 xHCI [1022:15b6]
IOMMU Group 31:
        71:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 3.1 xHCI [1022:15b7]
IOMMU Group 32:
        71:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h/1ah HD Audio Controller [1022:15e3]
IOMMU Group 33:
        72:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 2.0 xHCI [1022:15b8]

On lstopo you can see the ‘chaining’ more or less; best I can do since Asus doesn’t provide block diagrams:

image
image1237×927 66.1 KB

You see how PCI 01:00.0 (nvidia GPU), 02:00.0 (SSD 1), 05:00.0 (intel dGPU), 70:00.0 (SSD 2) and 71:00:0 (AMD iGPU) are all connected to the CPU directly and properly isolated. Everything on the first chipset (09:00.0 SSD; 6f:00.0 SATA controller) is also isolated. But everything on the second chipset is bunched together (networking, USB4, 4th NVMe, 3rd PCIe slot), at least on this board. Some boards apparently do isolate those, but here is where it gets tricky. But all AM5 groups I’ve seen so far have proper isolation for CPU-connected lanes and for everything on the first chipset (which is the only one on B650/B850/B870 non-E).

IMO people should look closer at the single Prom21 chipsets (cf. wikipedia) for VFIO on AM5. Such as the ProArt B650 (which seems discontinued), Taichi Creator X870, and Gigabyte Ai Top B850). The extra features on the top-line X670E/X870E seem more often than not to complicate VFIO…

1 Like