RAM prices are very silly. I am getting a system together to perform LLM development with 2x PNY RTX 5000 72GB GPUs
The GPUs have been ordered. My workloads would be centered around R&D with LLMs, nothing on this box would be considered production, but I don’t want to spend too much time getting bogged down with firmware revisions on my motherboard bricking something like PCiE bifurcation.
My intention is to run qemu+KVM (maybe proxmox) on the bare metal, run a VM hosting development infrastructure (gitea etc), and two LLM VMs each with a GPU passed through. I would estimate another 3 VMs for the actually development work.
The dev work itself would focus on infrastructure support with LLMs acting as an assistant for maintenance, monitoring, and documentation. I want to build tools around Proxmox to fix the needs of my organization.
Other projects would revolve around report generation, misc automatic research tasks, and code generation and review.
I would like help from you smart people out there to design a system to adequately support this work. I am looking in the neighborhood of 128 GB of RAM, but by the time I start considering motherboards and CPUs, it all gets a bit fuzzy for me.
Happy to make this a conversation during and after the system is built for any other curious minds.
If it doesn’t need to be cutting edge, you could do what I did and order a used 3960x (Threadripper) system off eBay. It was <$1k with motherboard + CPU and new enough that it has PCIe 4.0.
Using DDR4 will save a few bucks and it’s up to you if ECC is worthwhile
I ended up going with an HL15 Beast. I had some experience with the HL15 tame in a lab, and they worked well enough and this model has some handy SSD slots and plenty of room for the GPUs. A lot of what was on market was a bit painfully priced because most OEM providers are running DDR5. Building a custom machine would not have worked in this case, unfortunately. Especially considering the bulk of the work that will need CPU+RAM is low-intensity virtual machines. The place I need the best performance will be where the GPUs are anyway.
I spent the better part of the weekend building out the network on it, running Proxmox, but today I finally got to load up the models I want to work with for this project. Qwen3.5 122B A10B, Qwen3.6 35B A3B, and Qwen3.5 9B are running simultaneously to serve three separate Turnstone instances. Besides directed stress testing, I won’t be using each at the same time. It was really exciting getting them loaded and watching the graph creep up on nvtop!
The way this has shaken out so far is I elected to have one VM that runs inference behind on Virtual FW, and then there are distinct instances behind other virtual firewalls. Each of these are sitting behind a “core” FW so I can poke, capture, and measure the implications of having inference offsite. I thought it is the same way OpenRouter would provide the horsepower to something like Hermes of Turnstone so why not be able to see how the whole thing would work. The whole networking situation makes me think of the Forbidden Router video on L1T. I may have made it more forbidden-er.
Initial testing with Turnstone has been really rewarding, t/s are zooming! Over the next week I will be finishing the routing from the inference zone to the other application zones, building the tooling that Turnstone will lean on, and pilot it’s ability to be an infrastructure assistant.
I have been testing this machines fairly intensely over the last few weeks. With the release of Qwen3.8 27B, I have been kicking a lot of heat out of this box and it has held up well. The model has an annoying habit of insane reasoning blocks. Between that and my migration to vLLM, I have spend a lot of time troubleshooting.
45 Drives shipped it with the wrong CPU, so I was just working with 8 cores to begin with. Now I have something that gives me more room to stretch.
I am still playing around with the right way to segment my different services like MCP, Gitea, Turnstone, the vLLM workloads etc. onto different VMs. Once I smooth out the rough edges on the model and my turnstone instance, I’ll move to integrating a Proxmox MCP server. I am a long way from an infrastructure aid, but I am learning a lot as I go