Looks like you’ve put a fair bit of thought into this.
As far as using threadripper for passthrough, you might have problems with that. There were IOMMU issues with TR for a while, at least, that’s how I remember them. @Wendell or @bsodmike can probably give you more accurate information on the current state of TR passthrough.
On the hardware specs, I’ll recommend you change your boot GPU to an Nvidia unit. It will make it easier to bind the AMD cards to vfio. (just blacklisting amdgpu entirely, compared to module deps on amdgpu)
I’ve covered the IOMMU to the best of my ability, above. I’m also going to say that people have reported that the groupings of the board are quite terrible. ACS Override patches could be used, but I would recommend getting the ASUS X399 board instead (just put the boot GPU in the top slot). It’s just the better board.
Regarding the 64GB of ram, do you think that will be enough? I’m concerned that with your plans you might benefit from more. That said, I know that 128 is a bit of a ridiculous jump.
That’s doable, but remember that nvidia doesn’t like to pass through non-quadro cards. There are workarounds, but if you want to just switch the card allocation in libvirt, you’ll need to keep those workarounds in the VMs at all times. My recommendation is to have one VM for AMD and one for Nvidia. I think you’ll have a much better experience that way. GPU drivers in certain OS can be funny if you swap the cards around too much.
You can probably save a few bucks by not buying NVMe here. If you think it’s necessary, go for it though.
So, to sum it up, my recommendations are as follows:
Thanks for your input @SgtAwesomesauce. Based on what you discussed I’m thinking of the following config. It deviates a little from your suggestion I will try to explain my reasons. I’d appreciate your input. (I can’t figure out how to make a fancy table. Sorry.
I decided to go with two GPUs for CI initially one for each brand, leaving my forth pcie slot unoccupied once I figure out if I should add another card for specific testing. Maybe an older generation card eventually.
Threadripper 1950X
Asus ROG Zenith Extreme (Per your recommendation)
64 GB Corsair Vengeance LPX 3GHz (4x16 leaves me room for another 64GB if it turns out to be necessary)
LiqTech TR4 240
Thermaltake Core X5 (nice, roomy case and the motherboard is horizontal)
Seasonic Prime Ultra 1000W Titanium
1x ASUS R5230-SL-1GD3-L (Top x16 pcie slot, boot cpu. Still AMD, see below for why it doesn’t matter)
1x ASUS RX560-O2G (2nd x16 pcie slot, for CI test runs)
1x ASUS Phoenix GeForce GTX 1050 (3rd pcie slot, for CI test runs)
2x Samsung 960 EVO 1TB (To go on the dimm.2 slots, evo because of $$, will also be boot drives, still RAID1)
2x Hitachi deskstar NAS 4TB (raid 1, like before)
1x Intel Optane SSD 900P (u.2 but plugged through a converter into one of the m.2 slots in the middle. The u.2 slot shares 4 pcie lanes with the bottom x16 slot. If I add another GPU this would be an issue)
Does this all seem to make sense? Assuming I appropriate install instances with AMD and Nvidia drivers and apply the kvm hypervisor hiding stuff to combat the ‘error 43’ stuff.
Realistically, you won’t run into issues. Even a 1080 Ti will be fine with only 8 PCIe Gen 3 lanes. The available bandwidth of an x16 slot eclipses the required bandwidth of even the best GPU on the market. The GPU will fall back to x8 and you won’t notice a difference. Especailly with a low-end GPU.
I wasn’t 100% sure what you wanted to do in regards to storage, so I just went balls to the wall. If you’re happy with the 960 EVO, that’s perfectly fine. Fair warning, the dimm.2 slots are known to be a bit finicky.
GPUs look good to me. I only chose the Nvidia GPU because at the time, I understood that you wanted all AMD. Now that the requirements have changed, you can choose whatever boot GPU you’d like.
Good case choice. I only chose corsair because I’ve got a lot of experience building in their cases and I’ve had good experiences.
All in all, it looks like a sane setup. I think this is good, but remember to plan your fans out for positive air pressure to avoid as much dust as possible.
The bottom pcie slot is physically x16 but electrically x8. If the u.2 port is in use it drops to x4. I thought this may be a problem. Would x4 still be fine you think? If so I can just not use the dimm.2.
Do you have a good guide on how best to do this? I’d want to avoid too much overpressure as well, right?
Just for installing them or also at runtime? Do you have some more information on that for me?
Thank you so much for your input so far. I really appreciate it.
Ehh, x4 should be fine for a low-end GPU. I’ve run a passthrough’d GPUs in x4 before without problems, but it was an R9 380, so it wasn’t a beefy card by any definition.
This guy is talking about it, I don’t have access to TR hardware, so I can’t test it myself, but I’ve always been a bit skeptical of the dimm.2 slot.
Eh, I’ve got one exhaust, 3 intake and a few blank slots in my meshify C. It works well. The idea is that you filter all your intake and allow the positive air pressure to make sure that no dust can seep in through the cracks.
One unfiltered exhaust fan is worth about 1.35 filtered intake fans and one filtered exhaust fan is worth about 1.25 filtered intake fans, so it’s basic sums to figure out your air pressure in the case. (those numbers were from my findings after doing a ton of different configs on a few different cases, YMMV)
@SgtAwesomesauce You do your name justice, you are totally awesome.
Thank you so much for your input. I’m still going back and forth a bit on the exact motherboard. I’ve sent a PM to @wendell to ask why the X399 Designaire from Gigabyte got an ‘A-’ in the iommu grouping department. If it is because of fuckery with the nvme drive slots I’ll go with the ROG ZENITH EXTREME despite the ALL CAPS NAME and literally all of the RGB
I could maybe do a build log on the thing. People may like a little threadripper porn?
I had another hare-brained idea. I could maybe pass through the ‘boot gpu’ also actually. I found this tiny, tiny, PCIe x1 serial adapter with detachable cables. The ROG ZENITH EXTREME has a x4 and a x1 slot through the PCH but it is unusable with dual slot gpus. However, this thing is so so small that I think that if I don’t use the faceplate for it at all It’ll squeeze in just fine.
This would let me add 4 gpus for ci testing instead of 3. So maybe a polaris (RX560) and a gcn1 card, and for the green team a kepler and a pascal (1050) maybe?
I haven’t had any issues with the DIMM.2 slot on the Zenith personally. I’ve got a 2TB 960 Pro, and two other 500GB NVMe drives in my build (950 Pro / 960 Evo).
I have tried passing through whole NVMe drives to a VM, and that works nicely too (although went back to standard image files since it’s a bit easier to manage backups and whatnot).
I think I may give it a go. The card is only 30 euros and from eyeballing the shape of the 1050 from asus I think it’ll fit. Worst case it won’t fit I guess.
Just on the ROG ZENITH EXTREME - board looks pricey on the face of it, but in reality it’s not bad at all. Comes with a 10 GbE NIC. Which if you have a NAS box somewhere, you just need to buy a second one down the track a bit (rather than 2) and get vastly improved file transfers to it.
Well that’s how i am attempting to justify it to myself anyway
I was mainly considering it because of the iommu grouping supposedly being good. I’ll eventually fill up all of the X16 slots with double wide cards I imagine so I won’t even be able to fit the 10gbit card But according to @wendell it has the same problem as the gigabyte boards in that it groups nvme slots with x16 slots. I may just go for the x399 fatal1ty as it is the only board with a serial port header on it. If I can confirm that it doesn’t have any ADDITIONAL problems at least. Otherwise it’ll have to be the X399 Designaire.
tangent… If you’re looking for a serial port header - consider an airconsole.
That was my fix for rs232 serial.
Basically its a big battery with a USB/serial port on it running Linux. Connect to it via bluetooth/wifi. Works with iPad, Mac, windows, web browser terminal session, etc. Doubles as a battery charger for your phone or an ad-hoc wifi access point (has both wifi and ethernet on it).
I don’t work for them, but if RS232 is a thing for you, you might be interested. Avoids having to run serial cables to boxes, etc. Really useful if you do laptop-into-router/switch in a datacentre or whatever too (serial cables always too short). Just plug the thing in, go sit down somewhere convenient with your laptop or tablet
edit:
via bluetooth on the mac (once paired) at least it just presents as a serial device under /dev. just run screen /dev/devicename and bingo… seems MUCH more reliable than the USB-serial adapters you find on ebay, etc.
I was planning on doing something similar with an RPI. But having a ‘real’ serial port (not a USB one) is important to catch early boot stuff. If I can’t get a serial port header I’ll try to fit a pciex1 serial port card into a slot wedged between two GPUs probably
Sorry just re-read your post regarding the IOMMU grouping…
ASROCK board has better grouping (than the zenith) without the NVME / PCIe x16 slots in same group?
I’m agonising over what to get at this point. Ryzen 2700x is ample CPU wise for me, but the PCIe lanes are inadequate, and I’d much rather have 8 RAM slots, so i can buy say 32 GB now while ram is super pricey and upgrade to 64 or maybe 96 later when the price drops a bit (without throwing out my existing sticks).
Hence for threadripper i’d be looking at the 1900x until i need more. but i do mess with VMs a fair bit - not heavy usage for CPU but a lot of the stuff i’m wanting to work with needs large amounts of RAM (e.g., Cisco VIRL - 16 GB minimum required; virtulisation of ESXi/vCenter to plug into VIRL or GNS3, etc.)
So threadripper is attractive for the lanes and additional RAM slots, plus by now the platform is likely more mature than Ryzen 2xxx as far as Linux support goes (i’d hope).
so right now my choices are
X399 taichi
X399 ROG ZENITH
x470 Crosshair VII
Definitely feel X399 is where i want to be, assuming the Taichi works as well or better than the Zenith for IOMMU maybe thats the ticket. I’ve just heard varying reports of BIOS bugs/issues from elsewhere (and the ASUS BIOS definitely looks more comprehensive), but those issues were not IOMMU specific… and maybe resolved by now. I don’t have 10 Gb network at home, so the Zenith is very pricey for a lot of stuff i don’t need…
It’s going to be a dual purpose gaming box, preferably via looking-glass, hence the IOMMU queries… don’t trust windows as my primary OS for anything i care about (the Windows side would be strictly games/disposable throw-away test VMs only.
From what I understand all of the boards have this problem. However if you only need to pass through one gpu I think you’ll be fine with almost all of the boards now.
I’m trying to work out what board will let me do passthrough of 3 or 4 gpus. This is apparently where things start to get sketchy.
awww, thanks. I’m just doing my part.