Budget VFIO build in NOV 2023 - How I found the perfect motherboard (for me)

Hardware Motherboards
, ,
1

Hello everyone!

Mainboard Requirements:

  • BIOS option to choose which PCIE slot is used at boot (Only on Gigabyte X570 boards?)
  • BIOS option to enable native ACS for for further group segregation
  • 2x PCIE 4.0 Slots ( 1x X16, 1x X4) (1 primary guest GPU / 1 secondary host GPU)
  • 6x SATA 6Gbit/s (4x also acceptable)
  • Onboard Audio with Center/Sub, Rear and Line Out connectivity (for 5.1 analog)
  • 2x M.2
  • Decent VRMS with at least 7 real phases and ~60 Amps Mosfets
  • Mem OC rating of >4800
  • $250 - $300

Virtualization:

  • Guest / Windows Gaming VM: 1x NVME, 1x GTX TITAN (PCIE slot 1 @ x16), 1x Soundblaster FX Soundcard (PCIE slot 3 @ x2) 1x USB Controller, 2x SATA HDD

  • Host / Arch: 1xNVME, 1x GTX 1060 (PCIE slot 2 @ x4), 2 Cores, 1x USB, 2x SATA, Onboard Audio

Conclusion:

I’ve been planning a 700$ budget VFIO build for a while, reading every single thread and recommendation about various boards and their IOMMU groups.

It seems that AM5 with B650 and X670 chipsets, at a mid-range level, don’t offer any options with decent IOMMU groups, and those who do are lacking in other areas. Most of them have a group with all of the chipset connectivity lumped together. I understand that this is due to Asmedia being the chipset supplier.

So after days of searching and comparing data, I’ve decided to use the GIGABYTE X570S AORUS Elite AX with AM4 socket and X570S chipset for this build.

It fulfills all my requirements, has reportedly got perfectly separate IOMMU groupings and costs < $300

One potential issue might be groups 12 - 14 on the X570 AORUS Elite (non AX)

Group 12:
	03:08.0 PCI bridge [0604]: Matisse PCIe GPP Bridge [57a4]
	05:00.0 Non-Essential Instrumentation [1300]: Starship/Matisse Reserved SPP [1485]
	05:00.1 USB controller [0c03]: Matisse USB 3.0 Host Controller [149c]
	05:00.3 USB controller [0c03]: Matisse USB 3.0 Host Controller [149c]
Group 13:
	03:09.0 PCI bridge [0604]: Matisse PCIe GPP Bridge [57a4]
	06:00.0 SATA controller [0106]: FCH SATA Controller [AHCI mode] [7901]
Group 14:
	03:0a.0 PCI bridge [0604]: Matisse PCIe GPP Bridge [57a4]
	07:00.0 SATA controller [0106]: FCH SATA Controller [AHCI mode] [7901]

However, I read that by enabling ACS in the BIOS all of those should break up nicely, so I’m hoping that’ll work. There are much newer BIOS versions out atm, so I guess it’ll be a bit of a gamble.

I’d love some feedback and recommendations in case there might even be a better option out there.

2

ASUS B650 Creator

IOMMU Group 0:
        00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 1:
        00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 2:
        00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 3:
        00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 4:
        00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 5:
        00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 6:
        00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 7:
        00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 8:
        00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 9:
        00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 10:
        00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
IOMMU Group 11:
        00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
IOMMU Group 12:
        00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
        00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 13:
        00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e0]
        00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e1]
        00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e2]
        00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e3]
        00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e4]
        00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e5]
        00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e6]
        00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e7]
IOMMU Group 14:
        01:00.0 Ethernet controller [0200]: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [15b3:1019]
IOMMU Group 15:
        01:00.1 Ethernet controller [0200]: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [15b3:1019]
IOMMU Group 16:
        02:00.0 Non-Volatile memory controller [0108]: Seagate Technology PLC FireCuda 530 SSD [1bb1:5018] (rev 01)
IOMMU Group 17:
        03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev 10)
IOMMU Group 18:
        04:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479] (rev 10)
IOMMU Group 19:
        05:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX] [1002:744c] (rev c8)
IOMMU Group 20:
        05:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 HDMI/DP Audio [1002:ab30]
IOMMU Group 21:
        05:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 USB [1002:7446]
IOMMU Group 22:
        05:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7444]
IOMMU Group 23:
        06:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f4] (rev 01)
IOMMU Group 24:
        07:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        08:00.0 Non-Volatile memory controller [0108]: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive [1cc1:8201] (rev 03)
IOMMU Group 25:
        07:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
IOMMU Group 26:
        07:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        0a:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU Group 27:
        07:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        0b:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
IOMMU Group 28:
        07:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        0c:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f7] (rev 01)
IOMMU Group 29:
        07:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        0d:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f6] (rev 01)
IOMMU Group 30:
        0e:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO [144d:a80a]
IOMMU Group 31:
        0f:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raphael [1002:164e] (rev c1)
IOMMU Group 32:
        0f:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]
IOMMU Group 33:
        0f:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP [1022:1649]
IOMMU Group 34:
        0f:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b6]
IOMMU Group 35:
        0f:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b7]
IOMMU Group 36:
        10:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b8]
IOMMU Group 37:
        01:00.2 Ethernet controller [0200]: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function] [15b3:101a]
IOMMU Group 38:
        01:00.3 Ethernet controller [0200]: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function] [15b3:101a]
IOMMU Group 39:
        01:00.4 Ethernet controller [0200]: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function] [15b3:101a]
IOMMU Group 40:
        01:00.5 Ethernet controller [0200]: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function] [15b3:101a]
IOMMU Group 41:
        01:00.6 Ethernet controller [0200]: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function] [15b3:101a]
IOMMU Group 42:
        01:00.7 Ethernet controller [0200]: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function] [15b3:101a]
IOMMU Group 43:
        01:01.0 Ethernet controller [0200]: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function] [15b3:101a]
IOMMU Group 44:
        01:01.1 Ethernet controller [0200]: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function] [15b3:101a
1 Like
3

Thank you, that’s some beautiful groups.

The Creator is like $280 I believe, not bad.

Would you happen to have any knowledge of the MSI B650 Tomahawk’s groups?

4

Asrock B650 live mixer should work out too. It groups the huge amount of USB ports, so you most likley won’t need a dedicated USB PCIe card and you can still put 3 pcie cards in there, if you need to.

Here is the link to the OP, detailing the groups: