AMD Build for multi-GPU + GPU pass-through

Hardware Build a PC
1

Hi guys,

First post here, have been following the youtube videos for some time and have been very interested in the GPU-pass through series on Linux.

I’m an old-time Linux user (1996 through 2006 or so) but the past decade or so have been on Windows mostly for work and mostly Mac at home - except for a PC mostly purely for gaming and virtualisation.

My current build is a Haswell E3 Xeon on a H87 gigabyte board. If i’m not mistaken, i’d need a different motherboard (and thus CPU) to go down this GPU passthrough rabit hole and am currently agonising over Threadripper vs. Ryzen.

My primary home uses are gaming and VM stuff - want at least 32 GB of RAM (preferably 64 or more later) so threadripper is attractive.

I guess what I’m looking for are recommended AM4 and TR4 boards with sensible IOMMU groupings in order to build something capable of this with a minimum of fuss (understandably, there will be some!).

I currently have dual reference RX Vega 64s which will be migrated to the new build…

Current PC spec as follows: Xeon E3-1231v3, Gigabyte GA-H87-D3H, 32 GB RAM, Samsung Evo 850 500GB, 2TB SATA, Dual RX Vega 64

1 Like
2

Currently Asrock 's Tachi has the best IOMU separation, which makes GPU passthrough a whole lot easier to do. I myself am planning on getting its big brother which I have heard is basically the same motherboard with a few extra features that will be helpful with my use case. If you have the extra money meaning you are going to spend an extra $200 dollars for the Theadripper version of the same motherboard. Also I have heard setting up GPU passthrough on the Threadripper platform is a real pain in the butt, and I never heard of anyone being able to get it to work. Maybe that has changed.

1 Like
3

I believe Wendell has it working on the MSI gaming pro carbon under threadripper? Well at least according to his video series on the issue.

Thanks for your input in any case.

The other “issue” i guess i have at the moment is that currently the Ryzen platform is about to be updated in the next few months.

I’m one of those edge cases where realistically the big things threadripper will get me are the PCIe lanes and memory slots; 16 threads is probably plenty for me…

But then, if i’m going to go for a 1900x, the total BOM for a 1920X isn’t that much more… where does it stop… :smiley:

2 Likes
4

Well, I guess I missed that one, good to know AMD is will to work with the Linux community. I to would prefer a Threadripper instead of the Ryzen 7 1700 for the same reason as @thro, the increased PCIe lanes would come in handy, but among other things, I need to purchase, I will have to give Threadripper a pass unless Threadripper 2 becomes available sometime this year and prices drop quite a bit for Threadripper.

5

Currently Asrock 's Tachi has the best IOMU separation,

Build: Asrock X399 Taichi BIOS 2.0, 1950X (NH-U14S), 4x 8GB G.SKILL Flare 3200/CL14
GPUs: Asus Strix GTX 1080Ti OC, ATI HD5770 reference, Asus RX 550 2GB ,
Storage: NVMe Corsair Force MP500 120GB, Samusung 960 EVO 500GB, SSD: Samsung 830 120GB, some HDDs
Case: Corsair Obsidian 750D (+ 2x Noctua 140mm), PSU: Corsair RM1000x

IOMMU Groups

IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 10 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 10 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 11 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1460]
IOMMU Group 11 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1461]
IOMMU Group 11 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1462]
IOMMU Group 11 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1463]
IOMMU Group 11 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1464]
IOMMU Group 11 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1465]
IOMMU Group 11 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1466]
IOMMU Group 11 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1467]
IOMMU Group 12 00:19.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1460]
IOMMU Group 12 00:19.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1461]
IOMMU Group 12 00:19.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1462]
IOMMU Group 12 00:19.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1463]
IOMMU Group 12 00:19.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1464]
IOMMU Group 12 00:19.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1465]
IOMMU Group 12 00:19.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1466]
IOMMU Group 12 00:19.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1467]
IOMMU Group 13 01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ba] (rev 02)
IOMMU Group 13 01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b6] (rev 02)
IOMMU Group 13 01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b1] (rev 02)
IOMMU Group 13 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 13 02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 13 02:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 13 02:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 13 02:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 13 04:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 13 05:00.0 Network controller [0280]: Intel Corporation Device [8086:24fb] (rev 10)
IOMMU Group 13 06:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 14 08:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804]
IOMMU Group 15 09:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Polaris12 [1002:699f] (rev c7)
IOMMU Group 15 09:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aae0]
IOMMU Group 16 0a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 5770] [1002:68b8]
IOMMU Group 16 0a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series] [1002:aa58]
IOMMU Group 17 40:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 18 40:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 19 40:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 2 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 20 40:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 21 40:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 22 40:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 23 40:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 23 40:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 23 43:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]
IOMMU Group 23 43:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1456]
IOMMU Group 23 43:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB3 Host Controller [1022:145c]
IOMMU Group 24 40:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 24 40:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 24 44:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]
IOMMU Group 24 44:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 25 41:00.0 Non-Volatile memory controller [0108]: Device [1987:5007] (rev 01)
IOMMU Group 26 42:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] [10de:1b06] (rev a1)
IOMMU Group 26 42:00.1 Audio device [0403]: NVIDIA Corporation GP102 HDMI Audio Controller [10de:10ef] (rev a1)
IOMMU Group 3 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 4 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 5 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 6 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 7 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 8 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 8 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 8 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]
IOMMU Group 8 0b:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1456]
IOMMU Group 8 0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB3 Host Controller [1022:145c]
IOMMU Group 9 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 9 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 9 0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]
IOMMU Group 9 0c:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 9 0c:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Device [1022:1457]

2 Likes
6

@PetebLazar - to confirm you are doing IOMMU for a Windows VM without any hassles?

The grouping looks sane to me (though i am very new to this).

Any issues to report with the X399 taichi? I’m close to pulling the trigger on an ROG ZENITH purely based on the online support that seems to exist, but if the Taichi will do the job just fine it is a lot cheaper… and the addon stuff i’d be getting with the ZENITH is mostly stuff i don’t need. 10GbE is nice, but rest of my network isn’t, and i can add that later when it is presumably cheaper anyhow…

cheers