Best 2019 motherboard/other options for VT-D + Linux

**louhy** · July 15, 2019, 12:56pm

I was unable to find any recent threads or information on this, so I’m looking for current advice.

TLDR: Which motherboards should I be looking at for good VT-D support with Linux?

Planning to build a fairly high end Linux PC, but I’d like it to have VT-D support so I can boot a fast Linux VM or a Win10 VM (mostly for software testing, including 3D/OpenGL). I’ve done this a few times with older hardware, but it never quite worked as well as I’d have liked, so I’m hoping the latest generation stuff has improved support.

I want to make VT-D and GPU passthrough a top priority this time.

Here’s my starting point, although I’m somewhat flexible:

CPU: i7 9700K (would consider Ryzen but I’m uncertain about VT-D reliability)
1st GPU: NVidia 1080?
2nd GPU: 390X? Probably some AMD, since I think this helps significantly with GPU passthrough. (Please correct if I’m wrong or this has changed, I recall having to blacklist one card’s driver in the past.)
Motherboard: Undecided , this is what I mostly need help with. It seems like Z390 motherboards are popular with this CPU, but I’m unable to find any good information to confirm VT-D will work well. If there’s another chipset that is much more reliable for VT-D… what would I be missing out on by switching to (whatever that is)? I think I’d rather not switch to a Xeon processor…
Hard Drive: Not sure of brand but I definitely want a good M.2 drive for the main OS. I’m concerned about heat with these things, so I’ve been eyeing Gigabyte’s motherboards which have some “M.2 thermal guard” thing (not sure what that is yet).

Cost isn’t really an issue (I could upgrade or change GPUs if needed, etc.) but I don’t need absolute top of the line everything.

Using integrated graphics probably isn’t an option, partly because I want decent GPU performance on both the host bare-metal OS and the VM OS, but also because the monitor I use is near 4K and I anticipate problems (ex. HDMI 1.4).

Hoping to go with Linux Solus for the ‘bare metal’ host OS, and likely Mint for the guest. My typical workload is software development, but I do some gaming (Linux only). I’ve had GPU passthrough working before, but didn’t have luck with USB (of all things!), even with add-on cards.

How can I improve my odds of success this time around?

**SirTeddy** · July 15, 2019, 10:05am

VTd in intel and SVM in amd.Try x570 + ryzen 3000 series. Checkout my topic I didnt test all the mbs but ryzen impressed me a lot.I will say it is perfect for almost all situations.I had 6700k + z270 for kvm before.And now my [email protected] may have 210% performance compare to 6700k.

**louhy** · July 16, 2019, 12:50am

Thanks, clearly I have some research to do. Two NVidia GPUs… I wonder if that’s easier to do because I guess you did a headless setup. I hadn’t considered that option.

**SirTeddy** · July 16, 2019, 5:00am

Aha,It spent 5~10 min for linux install(copy files to disk, grub install,done.)

But the linux build is quite hard for noobs.I did it long time ago.Maybe two weeks include qemu & libvirt tests.

**louhy** · July 16, 2019, 5:40am

I’ve only ever done this with a Linux host, not too worried about that. I am kind of considering a 2950X now (GB’s X399 Aorus Pro is tempting), but I’m not sure I can get over the much slower single core performance vs a 9700K.

If I look at it compared to my current 6700K it’s a smaller gap on that metric, so maybe I can live with it.

This site https://passthroughpo.st/vfio-increments/ has me hesitating a bit with a z390 motherboard due to a kernel patch needed (for two discrete GPUs), and it doesn’t indicate that issue for X399.

I will likely need to read up on how these new AMD CPUs are doing on Linux. I’ve seen lots of “FUD” type stuff but not sure what the real story is.

I did find this awesome thread (and I think it says he used Solus too, so even more awesome), but I don’t think it has detailed instruction on what’s involved with that ACS override patch.

The Pragmatic Neckbeard 3: VFIO, IOMMU and PCIe Linux

Intro In this installation we are going to be discussing the technology behind PCI Passthrough to VMs. The concept of passthrough is relatively simple. You take a physical device and forward it’s memory registers to the VM. A simple idea, however, doesn’t make an implementation simple. There’s a lot that goes into passthrough, and a bunch of extremely talented people have put a lot of time into software to bring passthrough to the point where it’s a lot easier than it was before, but still not quite plug ‘n’ play. Conceptual Discussion Now, what goes into it exactly? To give a brief overview, we’ve got the hardware support for passthrough, the IOMMU or Input-Output Memory Management Unit, supported by both the motherboard and CPU. (more info on that here) The Linux driver, VFIO, is assigned to the device at boot, preventing the device from being initialized. This will help us when it comes to passing our GPU into the vm. If we’ve got the GPU bound to another driver, we won’t achieve successful passthrough, because you won’t be able to exclusively lock the GPU’s resources to the QEMU vm. Now, let’s talk about the PCIe bus. The machine I’m going to be using as a reference has an ASUS Z170-a and a 6700k. This gives me 16 PCIe lanes on the CPU to play with. Most GPU’s will be happy with 8 lanes, so we shouldn’t have bandwith issues here. I am going to be passing two devices to my VM: GPU and USB-3 controller. The passthrough GPU will be using an 8x connection on the PCH and the USB controller will be using a 4x connection on the CPU. This will allow the GPU I’m using for Linux output to be connected by an 8x connection as well. Handling PCIe passthrough isn’t 100% straightforward. There are limitations and rules about how the IOMMU sees a device and its IO Virtual Addresses (IOVA). Some devices will alias to the same IOVA space which makes the IOMMU unable to destinguish between the two devices. This becomes problematic when dealing with transaction…

**sgtawesomesauce** · July 16, 2019, 4:31pm

I’ve only ever had speed-bumps with debian based distros. Fedora, Arch, Gentoo, Suse, Solus were all easy going.

**gordonthree** · July 16, 2019, 6:40pm

A word of caution on Gigabyte’s x399 offerings. I have one the x399 Designare and my IOMMU groups are awful. Running the May 2019 Bios 12e. It seems everyone is all about passthrough these days, well, I’d pass on Gigabyte. Others claimed more success than I, so I wonder if something’s changed in the bios, or on the silicon?

**PaintChips** · July 16, 2019, 7:37pm

What might make you become “stuck” with Intel is if anything falls into legacy code(running something pre-Ryzen such as an old personal project) in a VM can be problematic or broken when used on a Ryzen CPU–from a developer point of view, it could be useful for bug testing

The best advice I’ll give is if you plan for reliability and stability, you’re really going to want ASUS or ASRock for a motherboard maker as they have had solid BIOS releases. (I’m not a fan of ASUS but they do push out updates to correct bugs/issues faster than others–ASRock as gotten better with updates too)

When it comes to GPUs just use whatever fits your budget, you could use a GeForce or a Quadro. Can’t say much about AMD GPUs, haven’t owned/used one for an extremely long time.

There are SSDs which aren’t as toasty such as SATA M.2 models, if you go the NVMe route you’ll just have to stick with a motherboard with an included thermal plate/heatsink or use a NVMe SSD which includes their own heatsink.

Normally I don’t like saying this, ever since Gigabyte jumped onto the RGB bandwagon via Aorus series they’ve turned into MSI
If you’ve ever used a Gigabyte board in a work environment their BIOS doesn’t get polished as quickly as other OEMs so your mileage varies, however some of their products have been rock solid and close to ASUS & ASRock.

**sgtawesomesauce** · July 16, 2019, 10:08pm

I’ll bite.

What could possibly make Ryzen not an acceptable option for running vms?

If you’re running bare metal, I’d get needing specific instructions, but putting a vm in the mix eliminates any concerns there.

**PaintChips** · July 16, 2019, 10:18pm

If the OP is running a current Windows and Linux in VMs there isn’t any risk of issues but I’ve seen some interesting examples of legacy OS or something which poked into something really old(ex: running Windows 8/10 32-bit to test older software or if you’re trying to migrate data). Even before Ryzen there were some interesting weirdness with Windows 8/10 32-bit on AMD CPUs, what I mean by weird is if a developer purposely coded for Intel CPUs.

**sgtawesomesauce** · July 16, 2019, 10:19pm

Right, but if you’re in a VM, setting the uarch properly means you won’t have this problem.

**louhy** · July 17, 2019, 1:14am

The passthroughpo.st site is mainly what prompted the x399 Gigabyte thing (they have an Aorus Pro listed). Of course I’m not really sure how much to trust that site either…

Does anyone have experience with ASRock Z390 Taichi or another Z390 ASRock board?

**SirTeddy** · July 17, 2019, 7:55am

emmm,I’ll checkout later.Really busy ,recently

**louhy** · July 20, 2019, 3:12am

Going to gamble (I guess) with these:

Core i7-9700K
AsRock Z390 Taichi LGA 1151 ATX

…and I’ll drop a spare R9 390 (Host) and 1080 (Guest) in for starters and see how that goes.

**louhy** · July 20, 2019, 11:08pm

Now I have to resist the urge to install a “temporary” cheap CPU cooler so I can start with OS setup, while I wait for a Noctua to arrive Monday… that being the only important thing left to get a hold of.

Kinda dumb to install a heat sink I know I’ll have to remove for sure in two days…

**MisteryAngel** · July 21, 2019, 2:03am

Should be fine i guess.
You could eventually also use the igpu for the host system.
If you run into any issues according the cards.