Part 1: Impetus and Part List
Welp, I had been moving toward hyper-convergence with a beefy LGA2011-3 motherboard and a 14-core 2680v4, but I had a change of heart. I guess I prefer “dumb” fileservers that run ZFS and export network shares and not much else. I was running out of PCIe expansion slots and lanes—with several expansion cards running downgraded and/or through the PCH. I’d also grown disappointed with the lightly-threaded CPU performance of Broadwell.
I had containers and VMs running kind of haphazardly across my homelab and I was feeling pressure to consolidate and optimize. I started thinking about what I wanted to do and what I had to work with. First I had to pull out from the fileserver the SSD-backed zpool for VM images, the GPU and scratch disk for Plex, and a few other odds and ends. That along with my spare parts bin gave me an initial part list:
- Ryzen 9 3900X
- 2x32GB DDR4-3200 CL22 ECC UDIMMs
- 4x500GB SATA SSDs (zpool for VM images)
- NVIDIA Quadro P620 2GB (Plex)
- 256GB M.2 NVMe SSD (Plex)
- 256GB SATA DOM w/ pin 8 power
- Intel X520-DA2
- Renesas USB 3.0 controller
- Noctua NH-U9DX i4 3U
I happened to have a 3900X on hand that I had just pulled from a build. Dual-CCD Ryzen 3000-series processors are ideal for virt because they give you four discrete L3$ regions to play with. You can significantly reduce cache misses and flushes by pinning a VM to some cores within an L3$ region and isolating it from the host and other VMs. (Or at least that’s how I understand it.) Other nice-to-haves include PCIe Gen4 support, relatively-low power consumption (especially in ECO mode), and great IPC.
I decided to build in the Fractal Define 7 Mini to accommodate the 3U tower cooler, give me lots of expansion options, and sit comfortably next to my 6U network rack. X570 is the only AM4 chipset with consistently-usable IOMMU groupings, and I wanted something with a BMC and an onboard NIC with SR-IOV support. Unfortunately, the only ATX (micro or otherwise) board that met all those requirements was the ASRock Rack X570D4U-2L2T, which was effectively not available at the time. The X570D4U non-2L2T wasn’t much better with an eye-watering retail price of $400 at Newegg.
I tried to figure out if I could make the X470D4U work to save a few bucks, but it had too many compromises. Apparently a lot of the PCH-attached devices get lumped together in fewer IOMMU groups, cooler compatibility is poor, the SATA DOM port doesn’t supply pin 8 power, I’d lose PCIe Gen4 support, and the M.2 slots are extremely limited. Then I thought about using a sub-$200 consumer X570M board along with a Pi-KVM, but the Pi-KVM was going to cost at least $190 to build even though I already had a Pi 4 on hand.
So I bent down, grabbed my ankles, and bought the X570D4U from Newegg. I still don’t feel great about it. 
Stay tuned for Part 2!
