One Workstation to replace my Homelab with VFIO - 4 VMs & n Containers

zarakynel · August 13, 2019, 3:03pm

Hey Guys,
this is my first post to this community therefore I would like to introduce myself a bit.
“I am a mid thirty computer enthusiast. Since my early days as a kid I was playing games and writing software. That might be the reason why my biggest hobby is also my profession. When I am sitting in front of a computer I like to read Warhammer40k Books and travel with my wife to see as much of the world as possible.”

Your time is as valuable as mine so lets jump straight into the topic without more boring information about myself (just sort that this is kind of board etiquette to introduce myself with the first post).

Background:
I am currently building a Workstation for my new apartment.
The basic idea is to run a single Workstation instead of running multiple Computers for dedicated use cases or tasks as I have done before. Thanks to the amazing work of this community pointing me to VFIO builds I changed my buying list for the new apartment which consists of five different systems and decided to combine all of them in one beautiful machine that might help me saving space (in my new home) and maybe a little money (which makes the project more wify compatible).

At this point I have a lot of questions which are unanswered and things that I need to consider before I can do the next purchase. Your feedback on every level is very welcomed and appreciated!

Goals:

Provide a Workstation PC for me (Windows 10)
– Needs to support Gaming (Blizzard, Steam, Origin (just for Sims 4) the full picture)
– Development Environment for Video Games with Unity3d
Provide a Gaming PC for my wife (Windows 10)
– Needs to support Gaming
Provide a Development PC for me (ArchLinux)
– For all non game development activities
Replace the existing self-build-NAS (Intel i3, 8GB, Nvidia 580 Ti) and Plex Media Server
Provide an environment for my whole infrastructure besides ubiquity stuff
– Nameserver for my home network domain
– Paperless Server to get rid of documents
– Next Cloud environment to get rid of google drive
– Game (Steam, Blizzard) Cache Server
– OpenHab environment to control my smart home and IoT devices
– CI/CD Environment for my software projects including git repository
– PI Hole to get rid of annoying ads
– I like to have my infrastructure as code therefore I need a host OS that is easy accessible with Ansible or Terraform

Constraints & Boundaries

Design the build for purpose not for perfection
Keep costs in mind (one time for the hardware as well as running costs and maintenance)
Use an incremental approach meaning I would like to take one step after another with visible results after each step
There is not really a budget limit with Display and Keyboard/Mouse I would like to end up at around 5k Euro
The planned life cycle is 5-6 years with some GPU Updates in future if needed

Hardware:
There are parts that I already own and I would like to use these for the build if possible:

PCPartPicker Part List

CPU: AMD Threadripper 2950X 3.5 GHz 16-Core Processor
CPU Cooler: Noctua NH-U9 TR4-SP3 46.44 CFM CPU Cooler
Motherboard: MSI MEG X399 CREATION EATX TR4 Motherboard (already purchased)
Memory: Corsair Vengeance RGB Pro 64 GB (4 x 16 GB) DDR4-3200 Memory (Start Setup)
Memory: Corsair Vengeance RGB Pro 64 GB (4 x 16 GB) DDR4-3200 Memory (added later if needed)
Storage: Corsair MP510 480 GB M.2-2280 NVME Solid State Drive (Gaming VM)
Storage: Corsair MP510 960 GB M.2-2280 NVME Solid State Drive (Workstation VM)
Storage: Western Digital Red 8 TB 3.5" 5400RPM Internal Hard Drive (Start Setup)
Storage: Western Digital Red 8 TB 3.5" 5400RPM Internal Hard Drive (Start Setup)
Storage: Western Digital Red 8 TB 3.5" 5400RPM Internal Hard Drive (will by 1 HDD every 2 month while movie db growth)
Storage: Western Digital Red 8 TB 3.5" 5400RPM Internal Hard Drive (will by 1 HDD every 2 month while movie db growth)
Storage: Western Digital Red 8 TB 3.5" 5400RPM Internal Hard Drive (will by 1 HDD every 2 month while movie db growth)
Storage: Western Digital Red 8 TB 3.5" 5400RPM Internal Hard Drive (will by 1 HDD every 2 month while movie db growth)
Video Card: MSI Radeon RX 5700 XT 8 GB Video Card (Gaming VM)
Video Card: MSI Radeon RX 5700 XT 8 GB Video Card (Workstation VM)

Generated by PCPartPicker 2019-08-13 13:13 CEST+0200

Critical Paths & Open Questions

I have a lot of questions in my mind and if you are able to answer them I would really appreciate your support and I would like to learn from your experience with similar topics.

What kind of OS do you recommend for the Host System? I am currently thinking about proxmox or UnRaid because I would like to run the host headless and both distros provide web interfaces for remote access so that I don’t have to provide a GPU for the host. The OS SSD is currently missing because some OS are able to boot from USB like UnRaid therefore I would add another SSD for the Host OS if needed.
Do you think that the approach of building one very powerful workstation that runs 24/7 to cover all my use cases has advantages over having dedicated systems for every use case? (Plex Media Server, NAS Server, Gaming PC, Workstation PC, Server running Infrastructure)
I would like to use the 2nd GPU for the Gaming VM of my Wife and if she is not playing for the Linux Development VM. As far as I understood the VFIO scenario that is possible unless I try to run both VMs in parallel - Is that true?
What kind of Environment would you prefer on the host system to cover all the mentioned software components for the infrastructure like paperless, Plex, DNS etc.? I am currently thinking of Docker with Portainer to have a Webgui and support for Infrastructure as Code.
What is you opinion on the selected CPU do you think that the 16 Core variant is to powerful for the mentioned scenarios so that I can save some bucks choosing with the 12 Core?
I am currently plan to pass-through a GPU and the M2 SSDs to the Gaming VMs because as far as I understood that is possible and brings nearly bare metal performance to the system. Do you have experiences to share with this configuration?
The last days I often read that not every AMD GPU is capable of doing GPU passthrough. Do you have experience with the model I selected? Shall I go for nvidia and hope that they never block GPU passthrough completely?
I am not a fan of hardware raids because I hate loosing data when the controller is dead so my question is what kind of storage solution would you prefer in the given scenario? I prefer UnRaid in a VM if possible with Hardware passthrough to a storage controller and the attached devices to become the ability to mix different kind of HDDs.

Guys I really appreciate your help so thank you for taking time reading all my stuff!

rockking · August 13, 2019, 6:13pm

unraid will probably be a little more turnkey versus proxmox. either will do well though.
either approach is fine. personally i think going one big box over multiple smaller ones is a easier way to justify to the wife that you need some really expensive hardware
yes that should work. i know VFIO has extensions that allow vGPU (another thread i saw asking about this). but i have barely played with VFIO let alone further stuff. but from my somewhat basic understanding is as long as you arent trying to run both VMs at the same time it will work. arguably i would opt to maybe use your GPU as the one that flip flops. but thats up to you.
unraid i believe uses docker for its internal app structure anyway. which may lead to a plus for using it over proxmox.
i would get more cores still but it should be enough
linustechtips on youtube has a very long playlist of videos basically doing this for their editors. it covered some of the issues they ran into. good watch for you
from what i understand of this is it was mainly a vega thing that caused issues. but i havent tried anything with VFIO in a long time. nvidia gpus require special software flags to hide some CPU attributes basically so the nvidia driver doesnt give error 43 or 53. one of those. but basically they check if the GPU is in a VM and error out if it is. its stupid.
unraid i believe also has a nas function and basically formats each disk as its own and then logically handles locating the file you want when you try to access it. so again probably another plus for using unraid over proxmox.

TheCakeIsNaOH · August 13, 2019, 9:45pm

Many AMD GPUs have a broken reset function to one extent or another. This means that they need a host reboot every time you reboot the guest.

Some cards(often from a aib) just work, and others have workaround patches in the linux kernel. Vega cards are fixed, although the patch is not totally finished or upstreamed yet, and the 5700(xt) should hopefully be fixed in the next couple months.

zarakynel · September 10, 2019, 2:28pm

Time passed by and I have received all the parts I needed to build my new machine “Zarakynel”.
And I am very impressed from the result. I am currently writing from a VM running Win10 with GPU passthrough and I would like to share my experience so far with the selected setup.

I bought most of the Parts mentioned early in my first post with some little changed like other SSDs (Intel instead of Corsair) and other HDDS (now I use shucked WD 8 TB drives from WD My Book because they are much cheaper then the WD Red ones).

I decided to go with Unraid as main OS due to the fact that I really like there array concept with disks that can be accessed outside of the array in case of an error.

The GPU passthrough is working for an AMD RX580 but unfortenatly the NVIDIA 1070 that I got from a friend is showing Code 43 after driver installation and I need to fix this issue.

The next step is to add more drives and services and integrate the whole smarthome into openhab.

My Experiences so far:

Power Consumption in idle is a little bit high with 150-200 Watts running 8 Docker containers - therefore I decided for auto S3 sleep after 10 minutes of array inactivity and WoL with a smartphone App to get the system back running when I need it (in S3 it takes 11-20 Watts which is fine in my opinion). Full load power consumption in games under windows VM is about 600 Watts which is ok for a system like that.

Unraid is working very well and is also easy to configure with all the community plugins
GPU Pass through with AMD works nice and charming

GPU Pass through with NVIDIA is currently not working

Pass through of a Blue-Ray USB writer is also possible (I make backups of my movies for plex)
The system is very quit with Noctua Fans and the Be Quiet Dark Base 700 Midi Tower in Reverse Configuration

Some Pictures:

I am currently in a hurry but if you guys are interested I will continue this thread.

TheCakeIsNaOH · September 10, 2019, 4:01pm

For the nvidia code 43, you first need to hide that the vm is a vm, not sure how to do that in unraid, but there should be a guide somewhere.

Then you may need to give the vm the vBIOS rom from your card. Again, there should be a guide for unraid somewhere, I’m just not a user so I do not know how accurate or upto date guides available might be.

FaunCB · September 10, 2019, 4:19pm

I’d go one step newer and do threadripper 3 when thats bumpin. Otherwise, 2990. As well, why 5700’s? Its a little hard to get atm. Course, so are vega’s. Could do 2 2070’s and get a bit more of a boost. Unless you have the gpu’s already, A+ if you do.

Also I’d use unraid. I’m seeing more 6 gamer 1 cpu rigs popping up, almost all using unraid. Proxmox is good for like teaching kids at school how to poke at linux.

tenthousand · September 10, 2019, 5:31pm

Thank you for this post. This is the kind of system I’m thinking about building as well.