X370 Board recommendation for GPU pass-through

Windforce · March 27, 2018, 2:23am

Hi, I am in the process of building a Ryzen system and one of my main goals with this system is to finally have a successful KVM/ GPU passthrough setup. I manged to snag a 1700x on black friday sale and well here I am…building an upgrade system that isn’t really necessary right now…

Anyway, I need some assistance with the motherboard options as I honestly did not keep up to date with recent developments to the Ryzen architecture. I was looking at the Asrock Taichi based on Wendell’s passthrough videos but what other board options are there with good IOMMU listings? I did a few searches for IOMMU listing for x370 boards but wasn’t really successful except for the Taichi.

Besides proper IOMMU grouping for successful GPU passthrough I want to be able to passthrough the onboard audio to the guest.
Are there any X370 boards that have the onboard audio in it’s own group? The IOMMU listings I found for the Taichi indicate that the audio is grouped together with some other stuff.

The reason I ask is that my last board (MSI 990FXA) had the onboard audio in it’s own group and I was able to pass that through to the VM. Of course I had no audio on the host when the VM was running but as soon as the VM was shut down the host regained control of the onboard audio which worked OK for me at the time.
Ultimately I want to have an add on sound card installed for the host and passthrough the onboard audio to the guest if possible. The plan is to have both of these cards connected to my logitech 5.1 via a mixer.

I noticed that there are some X370 boards on sale right now like the Gigabyte X370 boards and there is even and ROG board for US200 (https://www.amazon.com/ROG-Crosshair-VI-Hero-Motherboard/dp/B072LX7WLN/ref=sr_1_1?ie=UTF8&qid=1522111265&sr=8-1&keywords=ASUS+CROSSHAIR+VI+HERO&dpID=514i9%2BmLJsL&preST=SX300_QL70&dpSrc=srch). I wanted to know if any of these options or even the cheaper X370 boards could work for me?

HiGHRoLLeR · May 7, 2018, 8:58pm

I’m surprised you haven’t gotten any replies. I am looking for suggestions and leaning towards the Taichi board too

Windforce · May 13, 2018, 5:54pm

Yeh, I went with the Taichi after getting it on sale not too long after. The ROG board was sold out and the price went right back up after posting this

Anyway I actually started working on this build just yesterday but still haven’t worked on the passthrough yet.

HiGHRoLLeR · May 14, 2018, 2:20am

Ah, fantastic. Please update me on your progress! What host OS are you using? I’ll be using unraid

FurryJackman · May 14, 2018, 2:36am

The 2 suggested by Wendell in his Ryzen Passthrough video are great. The Gigabyte Gaming 5 has Dual NICs, but you will need a ACS patch to separate the NICs.

Windforce · May 15, 2018, 2:20am

I will probably run Ubuntu 18.04 for the final setup. Running Mint at the moment.

Upgraded ASrock BIOS from 3.20>3.3>4.7

IOMMU groupings with both GPUs (GT610 & R9 390) plugged in:

/sys/kernel/iommu_groups/17/devices/0000:2f:00.2
/sys/kernel/iommu_groups/7/devices/0000:00:07.0
/sys/kernel/iommu_groups/15/devices/0000:2e:00.0
/sys/kernel/iommu_groups/15/devices/0000:2e:00.1
/sys/kernel/iommu_groups/5/devices/0000:00:03.2
/sys/kernel/iommu_groups/13/devices/0000:03:00.1
/sys/kernel/iommu_groups/13/devices/0000:1d:00.0
/sys/kernel/iommu_groups/13/devices/0000:1d:03.0
/sys/kernel/iommu_groups/13/devices/0000:26:05.0
/sys/kernel/iommu_groups/13/devices/0000:03:00.2
/sys/kernel/iommu_groups/13/devices/0000:1d:02.0
/sys/kernel/iommu_groups/13/devices/0000:26:01.0
/sys/kernel/iommu_groups/13/devices/0000:21:00.0
/sys/kernel/iommu_groups/13/devices/0000:03:00.0
/sys/kernel/iommu_groups/13/devices/0000:27:00.0
/sys/kernel/iommu_groups/13/devices/0000:26:07.0
/sys/kernel/iommu_groups/13/devices/0000:20:00.0
/sys/kernel/iommu_groups/13/devices/0000:1d:04.0
/sys/kernel/iommu_groups/13/devices/0000:2a:00.0
/sys/kernel/iommu_groups/13/devices/0000:26:03.0
/sys/kernel/iommu_groups/3/devices/0000:00:03.0
/sys/kernel/iommu_groups/21/devices/0000:30:00.3
/sys/kernel/iommu_groups/11/devices/0000:00:14.0
/sys/kernel/iommu_groups/11/devices/0000:00:14.3
/sys/kernel/iommu_groups/1/devices/0000:00:01.3
/sys/kernel/iommu_groups/18/devices/0000:2f:00.3
/sys/kernel/iommu_groups/8/devices/0000:00:07.1
/sys/kernel/iommu_groups/16/devices/0000:2f:00.0
/sys/kernel/iommu_groups/6/devices/0000:00:04.0
/sys/kernel/iommu_groups/14/devices/0000:2d:00.0
/sys/kernel/iommu_groups/14/devices/0000:2d:00.1
/sys/kernel/iommu_groups/4/devices/0000:00:03.1
/sys/kernel/iommu_groups/12/devices/0000:00:18.6
/sys/kernel/iommu_groups/12/devices/0000:00:18.4
/sys/kernel/iommu_groups/12/devices/0000:00:18.2
/sys/kernel/iommu_groups/12/devices/0000:00:18.0
/sys/kernel/iommu_groups/12/devices/0000:00:18.7
/sys/kernel/iommu_groups/12/devices/0000:00:18.5
/sys/kernel/iommu_groups/12/devices/0000:00:18.3
/sys/kernel/iommu_groups/12/devices/0000:00:18.1
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/20/devices/0000:30:00.2
/sys/kernel/iommu_groups/10/devices/0000:00:08.1
/sys/kernel/iommu_groups/0/devices/0000:00:01.0
/sys/kernel/iommu_groups/19/devices/0000:30:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:08.0

lspci -nn output:

00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1450]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Device [1022:1451]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1460]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1461]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1462]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1463]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1464]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1465]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1466]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1467]
03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b9] (rev 02)
03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b5] (rev 02)
03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b0] (rev 02)
1d:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
1d:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
1d:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
1d:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
20:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
21:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
26:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
26:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
26:05.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
26:07.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
27:00.0 Network controller [0280]: Intel Corporation Device [8086:24fb] (rev 10)
2a:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
2d:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF119 [GeForce GT 610] [10de:104a] (rev a1)
2d:00.1 Audio device [0403]: NVIDIA Corporation GF119 HDMI Audio Controller [10de:0e08] (rev a1)
2e:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1] (rev 80)
2e:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [1002:aac8]
2f:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]
2f:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1456]
2f:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:145c]
30:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]
30:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
30:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Device [1022:1457]

My secondary GPU and onboard audio are in separate IOMMU groups so I am good with that. The groupings may change a bit when I add in my secondary PCIe Sound card. Still waiting on some additonal items like sound card & HDD before completing the final setup.

One little oddity I noticed after upgrading the Taichi to v4.7 BIOS is that the IOMMU option disappears from the BIOS under the north bridge chipset leaving only the SRIOV option. Strange but it works.