In the purpose of keeping this short and sweet, I’ll keep things simple:
I won in a raffle at work the Ryzen Threadripper 2990WX (32 cores / 64 threads) CPU.
Based on this CPU, I would like to build a good 3/4 gamer / 1 CPU machine, kinda like LinusTechtips.
I have no other use of this CPU and it’s huge core count then to run multiple VMs in parallel.
I work as an IT manager and have a Software Engineering background, however I have not dabbled with
Linux for the past 15+ years. I’m been mainly on Windows machines for work / gaming and Mac for daily stuff: internet / email / etc.
What I would like is that the community help me select the best components for my build and installation to make this project possible.
I have not purchased any other components yet, in fact the only thing I have right now is the CPU. Money is not an issue and I’d rather go
with quality components if possible.
I have however set a few personal objectives if possible:
I don’t want to liquid cool my system (needs to be air cooled i.e Noctua fans, etc.). I live in a fairly cold region, so heat is not much of an issue.
Needs to be a stable system i.e. runs 24/7 with minimal downtime. Both hosts and VMs would be running with minimal downtime.
Main host needs to be Linux. I would like to use Arch (however I have never installed before) or possibly Ubuntu if Arch is too complicated. At this point I would consider myself noob or intermediate tier Linux user (due to past experience on Linux).
I would like to go with Team Red if possible for GPUs as I would like to support Open Source i.e use Mesa ?
I would like to avoid installing Windows 10 if possible as a VM.
I would like to run 3 or 4 VMs in parallel (see objective 2).
All my games are on Steam and I run them in 1920 x 1080 resolution.
Please help me with any suggestions you might have for this build.
If you want a system with absolute minimum downtime, Arch is probably not the way to go. It has kernel updates all the time, and those need reboots to apply.
Arch does have some nice AUR packages for passthrough, namely linux-vfio(patched kernel with ACS and i915 patches) and QEMU-patched. They auto-update but you have to build them yourself just as with many AUR packages.
I would generally recommend Ubuntu or a Ubuntu spin(Kubuntu, Xubuntu, Ubuntu server, etc) to at least start with. Keep your home on a separate partition then you can switch distros without too much issue if you decide you want Arch later on.
For the host, yes.
For the guests, not necessarily. AMD cards have a reset bug that can’t really be easily worked around, Nvidia cards have code 43 but that can be(currently) easily worked around.
If you want windows, windows 8/8.1 still works pretty much as well as 10 for passthrough. It has DXGI so looking-glass works and is also compatible with OVMF(qemu uefi).
Or if money is truly not an issue, then you could look into windows 10 LTSB/LTSC as it does not have many of the things that people find problematic with Windows 10.
What is the reset bug ? Does the reset bug apply to all AMD cards ?
As for GPU / cards, I was looking at the following cards:
Radeon VII
Vega 64 or 56
RX 590 or 580
Pretty much all the above cards will be able to run my games in 1080p without much issue at 60fps or above.
However, my main issue with the card selection is that since they will be stacked i.e 4 cards, I would need to get blower style cards to reduce
the heat impact of each card on the other.
On the green team, I was thinking of the RTX 2060 card.
One thing that is still not clear regarding GPU passthrough, for a system with 4 cards, is can all the cards be identical i.e. same vendor / model ? Or does each card need to be different (i.e. not same model) ?
Would the following setup be feasible:
Main host: Radeon VII
VM1: RTX 2060
VM2: RTX 2060
VM3: RTX 2060
As for Windows 10, I should have said Windows in general. I would like to transition to Linux full time and support the Open Source community as much possible.
It is where the card will stop working after a VM is shut down. The exact severity depends on the card. Sometimes it is with any shutdown, sometimes a reboot is fine, sometimes only with a hard reset/shutdown/VM crash. Linus had this issue with 7 gamers and the R9 Nano.
I am actually not sure if there are any good enough(ie r9 290 or rx470 or better) AMD cards that are totally unaffected.
Cards of the same model(not just specific SKU but same GPU core ie rtx 2060 vs 2070) have the same PCI-id. The PCI-id is used to specify the driver, but you use one driver for the host and another(vfio-pci stub) for passthrough. So use a different GPU for the host, but it is ok to use identical GPUs all for passthrough since passthrough to each VM is specified with the PCI address not the PCI ID.
Sure, I’m not sure how stable the Linux drivers(given the state of the windows drivers) for the VII are yet, but they definitely will improve with time. If you get RTX cards, get ones with a USB C port(virtual link is the branding I think). That way you can just break that out with a USB hub to for each VM to allow easy hotplugging of USB devices. Otherwise, you would have to get PCIe USB cards to passthrough to each VM(and find slots on the mobo for them) or deal with other ways to use USB devices in the VMs.
Biggest issue I’m seeing is mostly for the USB ports you’ll need to also pass through to the hosts. Not sure about the USB-C ports on GPUs, but I seem to remember them being just for video out or power for VR headsets or something similar. If they do work with peripherals, that would be awesome to know
On my mobo I have several extra USB Controllers & I passthrough an entire USB 3 controller to the VM with a KVM switch (with keyboard / mouse) plugged into one slot & a gamepad plugged into the 2nd USB slot.
Also passthrough a GPU with a UEFI BIOS (AMD R9 380 or later / most Nvidia) - I think the new Radeon 7’s were accidentally shipped without a UEFI BIOS.
The GPU has no reset but error, but I’ll see if it works for the USB controller over the weekend when I have the time. Thing is, this will happen to any VM that has the USB controller and is OS agnostic (happens on Linux, FreeBSD, and Windows 8.1 and 10), which makes me thing it has to do with the fact that it is on the motherboard, rather than provided by the CPU
