From what I understand all of the boards have this problem. However if you only need to pass through one gpu I think you’ll be fine with almost all of the boards now.
I’m trying to work out what board will let me do passthrough of 3 or 4 gpus. This is apparently where things start to get sketchy.
Ah, thank you very much for clarifying.
Yes, i’ll just be passing through a single GPU. My only reason for pass-through is non-linux compatible gaming, via a windows VM.
…and as I posted elsewhere, do check the Gigabyte X399’s UEFI releases. I haven’t seen anything since F3g listed on their site, and this was released when?
Ages ago. Since then Asus has posted at least 2-updates with comprehensive AGESA patches that tie with kernel 4.15 (and better memory support/stability). Not really sure what Gigabyte are thinking by not supporting stuff they are selling?!?
So you would recommend an Asus board then? I’ll check the ASRock website too for release dates. The fatal1ty board having a serial port is super useful for me.
Running 2x Zenith Extreme’s here and can’t really complain.
I sent an email to ASRock asking them about the IOMMU grouping on the Fatal1ty professional gaming. ASRock support went so far as to fully populate one of their boards with 3 nvme drives and 4 gpus and they sent me back the resulting IOMMU grouping.
It’s perfect, all of the drives are in their own group as well as the GPUs. There’s even some separation for the various USB controllers.
I’m going to go with this board. In part because of the serial port but also because this is a level of support I really didn’t expect to get.
I’ve asked ASRock support if I can share the information they gave me, if I can I’ll make a new topic with the IOMMU groupings for others to reference.
Quick note on the ASUS Zenith Extreme: You can get it to boot from a slot other than the first. However, I could not successfully pass a Vega 64 in the first slot through to a VM.
Vega still suffers the reset bug for me. I am on Kernel 4.16 built with the TR patch. Maybe you could get another card working? I cannot confirm though as I have not tried.
If I could do it all over again I’d honestly go with the following:
Pros: Potentially more cores, more PCIE lanes, more cache, hopefully better linux support, ability to get more standard ECC registered dimms, etc.
Cons: lower clock speeds, no wifi/bluetooth built into board, less usb slots, less aesthetic
I guess for me if I can get more cores the lower clock won’t matter that much. It’s mainly compile jobs so more threads work (almost) as well as higher frequencies. But the cost is a lot higher. I’m not so sure if this is really a viable option for this build. I’d have to get a 7401P to be in a somewhat similar ballpark (+200 EUR) which isn’t the end of the world but the only SP3 motherboard I can get with 4 x16 slots is a gigabyte one (Gigabyte MZ31-AR0) It’s also 200 euros more than the X399 Fatal1ty.
I’m not so sure this would really work out all that well for me. I could see if I can source a supermicro board from somewhere that does have 4 x16 slots though.
EDIT: The gigabyte motherboard’s physical pcie layout actually makes inserting 4 double-wide pcie x16 cards impossible 
Also I’m using an rx 550 for host, and have a Vega passed through without issue @SgtAwesomesauce @TMM2K18. I like having an AMD card for host for the solid open source drivers. I’ve also been a hobbyist alt coin miner for a while so, personally, I’m much more familiar with AMD than NVIDIA.
I have a sapphire rx 550. Its 1.5 slot size. If I did it again I’d probably grab a wx 2100 or 3100, simply for the true 1 slot form. True 1 slot form could allow for using the pcie x1 or x4 for a usb card without risers or watercooling.
I’m actually hoping to have no video card for the host at all but use a serial console only (and SSH) I won’t even be connecting any monitors to this thing at all. The pass through is only so that I can run integration tests on some GL and Vulkan programs across operating system, driver versions, and video card vendors. I should really only need to connect a monitor to that thing if things go really very wrong 
Maybe this board could suit your needs? I have little experience with server hardware so do not just take my word for it.
Edit: sorry just realized it only has 3 PCIE x16 slots. x8 gen 3 is difficult to bottle neck from what I’ve read however.
Yeah, but those slots are also physically x8 so a x16 card physically just won’t fit. I’m not so worried about the actual speeds. The physical layout of the board also makes it so that no more than 3 double wide cards could fit even if all of the slots were physically x16 
Like I said I would go for EPYC if I could do it again. THat said the Zenith comes with a bunch of extras and goodies. One cool thing I’ve never had before is temperature probes with their own mobo pins. Not positive, but pretty sure you can sync fans to these in UEFI.
Comes with all sorts of SLI cables, a fast pcie ethernet card, fast qualcomm and wilocity wifi cards (qualcomm works out of the box on recent linux distros like Fedora 27, Ubuntu 17.10+, etc. Doesn’t work on Centos 7 out of box). Also comes with a usb drive that can be used to back flash to original UEFI.
The onboard led message box thing is cool, eventually glitches out for uptime over 1 hour.
Memory can be tricky, but once it works, it works.
UEFI has almost too many options with little explanation of what any of them do.
Oh and dont expect to be able to change the LED color on the motherboard. ON or OFF in UEFI works. Even using the drivers provided, booting windows from bare metal, this doesn’t work for me. Always does rainbow unless you turn it off.
And expect a hell of a fight getting it to fit into the socket, as almost any threadripper review will tell you. I have the worse of the two (Foxcon i think?). It can be done eventually though. Don’t follow instructions, tighten them all bit by bit and press down very hard.
Edit: there is a CPU fan speed bug that is unfixed, you must have UEFI ignore cpu fan speed or be willing to have UEFI throw error every boot and sometimes reset all you settings because of this!
@gnusense thanks for the information! I’m probably going to go with the X399 Fatal1ty from ASRock though. They were very helpful with providing me technical information about the iommu groupings of that board. Plus I won’t have the pcie slots to spare to put the 10gig card that comes with the zenith extreeeeeme anywhere