I’ve done some searching on and off the Level1 forums for ESXi and x570 and found some information, but not enough where I’m confident enough to pull the trigger on a build.
To begin, let me give a brief rundown on my situation. I’m looking to hop on the Ryzen platform for an upcoming virtualization build. The aim here is to have 1 tower that can virtualize 2 “workstations” with GPU passthrough simultaneously. I am fully aware that ESXi isn’t the “best” choice for this, as things like Unraid and qemu/kvm exist in far more supported scenarios, especially since Ryzen 3000/X570 is still quite new and will never truly be supported by ESXi. (consumer platform and all that). I haven’t shut out Unraid or qemu entirely, but my ideal scenario for a variety of reasons would be to use ESXi. This is for a home lab/workstation, so lack of enterprise support is OK with me.
The CPU I have in mind is the Ryzen 9 3950X, a 16-core beast, 64GB of memory, and 2 GPU’s (one for each VM, undecided which GPU’s at this point). When it comes to ESXi, I definitely know it’s picky about hardware (want to have an Intel NIC, etc). I’m trying to find the best X570 motherboard for ESXi for under ~$400 USD, ideally one with the best IOMMU groupings. I found a post on Reddit that has ESXi running (with GPU passthrough) with a Gigabyte X570 Aorus Elite motherboard, but not with very much information about IOMMU and passing through other devices, like USB controllers.
Any advice here would be VERY greatly appreciated, even if it doesn’t strictly pertain to ESXi, but to other hypervisors.
Thank you for your time, and for the amazing forum!
Thank you so much for the suggestion! The extra PCI lanes are definitely nice, and from a quick 5 minutes of research on that board, the IOMMU grouping seems pretty ideal.
I will definitely be pursuing this as an option! The appeal of the latest & greatest X570 chipset is real, but may not be the right tool for what I’m trying to accomplish here. I didn’t realize second gen Threadripper had become this affordable!
I’m running an ESXi hypervisor build on an x470 Taichi Ultimate with Ryzen 3700x (was 1700 before upgrade on release). Be aware for ESXi that you can at max assign 8 CPU threads with the free license per VM, so the 3950x will have half its cores/threads not being utilized unless you plan on adding additional non-GPUs. Unfortunately the biggest thing you’re going to have issues with is USB ports. There are just one set of USB ports that you can passthrough to one VM on every x470/x570 motherboard I’ve seen, and you can only passthrough the PCIe slots that get their lanes directly from the CPU on VMware (this is where the kvm/qemu with the ACS patch wins).
Try to find a board that has 2 sets of USB ports that can be passed through to different VMs, otherwise you’ll need x299 or x399 to add the additional USB card you’d need for the 2nd VM.
I have been running 2 different x570 boards and Zen2 CPUs with kvm+qemu.
The IOMMU groups for CPU and chipset are split on each, unlike the x370 and x470. No ACS patch is needed.
Only problems are with Zen2 CPU.
USB and Audio controller that are integrated do not return from FLR.
I think there is a bus reset problem that was for Zen1 threadripper where the bus state was not reloaded after its reset. I have a patch in the kernel for it but dont remember if it is a problem.
Did not test Zen1(+) in the board to see how it would behave.
After doing some more reseach, I ended up going with the suggestion in the second post above; the X399 Taichi and the 2950X CPU.
@2bitmarksman I actually plan on running 2 simultaneous VM’s, each of which having 8 cores, so I’m totally fine with that limitation in the free license. As for the USB ports, I plan on passing through 1 GPU and 1 USB controller to each VM, so also a non-issue for me. I grabbed 2 x 4 port 5gbps USB controller cards off Amazon, which I’ve used to passthrough to a VM on another host I have pretty flawlessly with ESXi .
The IOMMU groupings seem very solid on the X399 Taichi from my research. The components are all arriving today and tomorrow, so the build will be commencing soon I will definitely keep this post updated with my progress, and I absolutely welcome other responses/advice!
I think that 2950x will serve you well. Don’t jump right in with a bios update on the board, see what your groups are like first, then update if something’s broken.
I really enjoy my 2950x server, just haven’t found quite the right chassis yet.
X399 and 2950 is a good choice, but just to make sure I’m understood:
2950 is 16 cores and 32 threads. The free version is limited to 8 threads per CPU, so with core pinning, 4c/8t. So you’ll have 8c/16t not being utilized, or free to do other things, like some form of NAS/HTPC/server duties
Yes, you are correct (unfortunately ) . I’m honestly debating on purchasing a full license of ESXi (roughly $1K). I’ll see how everything goes on the free license first, of course. To your point, I could definitely make use of the extra threads on the free license with other things; TR can definitely handle it!
@gordonthree I ended up going with the Fractal Define XL R2. It’s a full tower chassis that’s served me very well in the past. Tons of room for airflow, components, radiators, it’s quiet, and looks like a pro workstation on the outside. I definitely recommend giving it a look if you’re going for a workstation feel.
I’m in search of a server chassis that supports a 360mm AIO and 10+ easy to service 3.5" drives, tool-less would be perfect. Many can do one or the other, seemingly none can do both without extra parts (raid cages for example).
I’m thinking my ideal case is a unicorn. My Phanteks would be OK if I didn’t have to completely disassemble it to service even a single drive. For whatever reason the designers thought it would be cool to have the drives be cabled on one side but fastened on the other side of the chassis, so both doors come off to do maintenance.
I sent an email to fractal, but I think they preclude using a rad when the case is in “storage” configuration.
Hey new member here, I’m currently getting parts together to move my aging Dell Poweredge r510 over to a R7 3700x, x570 ESXi 6.7 rig, also I have a unused ESXi Standard 6.7 license if anyone needs it I can sell it for much cheaper than $1k, it’s nice to have access to all threads on board.
I will definitely keep this post updated with my progress, and I absolutely welcome other responses/advice!
