I am in the process of designing a local AI environment designed specifically to assist my daily software development workflow with high-context LLM’s. I need advice on optimizing a minimal but robust service stack for model inference.
The primary objective is leveraging multi-GPU compute resources for LLM deployment while minimizing operational overhead and maintaining architectural stability.
Infrastructure & Environment:
Host OS (Local Workstation): Arch Linux. This should host the development tooling but I would rather not run un-needed services here.
Deployment Infrastructure: A separate bare-metal server running Proxmox. I plan to encapsulate the LLM serving components within isolated LXC containers or dedicated VMs on this Proxmox host for stability and easier management of service endpoints.
Architecture Goal: Achieve a single, reliable API endpoint that handles multiple users (2-3) asynchronously.
Workstation Hardware:
CPU: AMD Ryzen 9800X3D (8 cores/16 threads)
RAM: 32GB DDR5-6000 Dual Channel (2 x 16 GB)
Motherboard: Gigabyte B850 AI TOP
GPU’s
Primary: ASUS PRIME 9070XT (PCIe 5.0 x8 slot)
Secondary: ASRock Creators AMD Radeon R9700 AI PRO (PCIe 5.0 x8 slot)
PSU: 1050W unit capable of PCIe Gen 5.1 requirements and utilizing native 12-pin GPU connectors/headers.
Workload/Model Expectations:
For now, the models will be exclusively utilized in a development workflow via OpenCode.
Tasks include:
Generating jest tests/Apex tests for complex JavaScript/Lightning Web Components(LWC) logic.
Analysis through large-scale log parsing.
Reviewing and proposing fixes/optimizations for intricate Apex trigger chains and controllers.
Implementing net new features to the packages. Features will likely encompass LWC’s with HTML/JS and server side Apex code annotated with @AuraEnabled.
Personally, accuracy and contextual relevance are more critical than peak token generation rate(up to a certain point, can I get 40 t/second?). Salesforce Skills have come in handy for documentation lookup which adds to the contextual relevance of the models output.
Based on an estimated total VRAM pool of ~48GB across the GPU slots, my limit is likely around 35B parameter models when using quantization (Q4/Q6).
What minimal software stack do you all recommend to get this setup operational?
I want to keep complexity low while maximizing stability of my workstation since I use it daily for work and gaming.
The main goal of llama.cpp is to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.
Plain C/C++ implementation without any dependencies
…
Depending on your model of choice, Q4 or even Q6 might be cutting it. For example, Qwen models tend to be more sensitive to quantization.
I’ll second the earlier recommendation. Llama.cpp can run in router mode which is good enough for small deployments. If you have very strict requirements, you might want vLLM but there be dragons.
you could squeeze in qwen3.6-27B which is generally considered “better” than 35B-a3B.
however it is a lot slower and even more slow as you won’t be able to enable MTP with multiple users (needs to be 1). if the token/s isn’t a concern then it is worth a try - i do admit the speed of 35b-a3b is awesome. for reference I am able to run qwen3.6-27B q8_0 weights with fp16 kv cache MTP at about 222k context with 56gb vram - including mmproj / vision (and a few other things eg sunshine etc). while i personally use llama.cpp and recommend it - if you really have 2-3 concurrent users maybe vLLM might be more perfomant - i can’t say from experience i have just heard that it is.
Wouldn’t vLLM default to the lowest common denominator in terms of VRAM i.e. in his setup it would be 16gb of the 9070XT?
Personally, I’d suggest using llama.cpp for getting the baseline performance numbers for a setup like yours.
For the actual LLM, Qwen 3.6 27B Q6 with unquantized KV cache would be my preference. The MoE (3.6 35B A3B) is way faster but might lose the plot occasionally.
I spent several weeks fiddling with this before @digitalscream was able to sort out everything for me with a nice explanation of why things should be done the way they are with AMD in the context of serving up models. The TL;DR is, llama.cpp is dead easy to get working and working right while you will end up fighting vllm the entire time, and its not even faster unless you’re getting really exotic with the setup.
