Hi, I am looking into building a home server where one of the most demanding use cases would be LLMs and gaming over a network (via Steam Link or Moonlight). I was looking at the Framework Desktop, but I’m not sure if it’s a wise choice.
It can fit larger models, but it’s not reasonable to run them.
Gaming-wise, it’s not a really good deal.
I was also considering using multiple Intel Arc B50s or an AMD 9600 XT.
So, my questions are:
Which LLMs have a good performance-to-price ratio for the required hardware?
What is the best hardware for AI in the Framework Desktop’s price range?
It might help to share what use case you have for the LLMs. The Framework desktop has a pretty wide range for pricing depending on configuration, so it might be helpful to share what your exact target price is.
I used old gaming hardware (I already owned) for the machine that runs my local LLMs, and eventually bought a RTX 4000 Ada SFF (20 GB VRAM) to replace the small GPU that was installed. I can run models up to ~30b parameters using Ollama, but my primary workhorse models are the 12-14b parameter ones. I’m using Ollama + Open WebUI, and recently linked up SearXNG to do web searches. My default config is QWEN3:14b and QWEN3:1.7b for the web search queries (small and reasonable context window). Usually I can ask a question and let it run, and when I come back a few minutes later, I have my answer and some sources.
I haven’t used local for agentic coding, but I have used some of the local models with other agentic tasks, like helping to automatically update/correct documents based on related data/documents. I used several models sequentially (with a small loop) to make that work so that I didn’t run out of GPU memory.
Honestly, I woud keep your gaming separate from your LLM machine if you can. I have a machine for gaming and if I tried to run LLMs on it, my electricity bill would be insane. The old hardware I’m using to host my LLMs is not the most efficient, but it can idle reasonably low, and even under a load, it’s not too bad. The big difference is the GPU. I had a RTX 2060 in there originally, which will run at ~150 W under load (idle around 15-20 W), but the RTX 4000 Ada SFF runs at a max 75 W (idle around 5-7 W), which doesn’t sound like a lot, but it can make a big difference over time. You can fiddle with undervolting the GPU, which is okay if you have the time, but if it’s unstable for gaming, you’ll end up reverting anyhow.
My advice would be to get a machine for the purpose you’re more interested in having sooner, and get another one later. You of course don’t have to do that, it’s just my recommendation.
Also, (for LLMs) the Intel cards definitely seem like they could have a decent price to performance value, but also don’t forget about the new AMD entry. Little more pricey, but gets 32 GB of VRAM and not too bad on the power draw.
I didn’t know about the AMD R9700 and it probably makes the most sense for me because of the VRAM for LLMs and because of AMD support and gaming on Linux.
A separate gaming computer doesn’t make much sense for me, it’s not a priority for me.
The total cost is an estimate, prices are based on Czech Republic prices with tax included…
forgive my ignorance but aren’t “dual” gpu’s less valuable because you can’t just combine VRAM on frontends like ollama, LM Studio? and if you are using AM5, you are gonna be put out on PCIE bandwidth, generally only 16-20 lanes available.
You can’t combine it like a singular memory pool on consumer hardware but you can spread layers across the GPUs for inference.
It works pretty well for generic LLMs. Video/image generation is different though.
Additionally, lanes will not matter that much other than initial loading into GPU memory and unless you offload some stuff onto the CPU. 16x + 4x is okay but obviously x8 + x8 is better.
Note that those models you listed are all very old. 2 years are positively ancient in this space.
If you don’t need the token generation rate to service a small business and are willing to try MoE models, the best current setup might be a single powerful GPU for prompt processing and KV cache, and as much system RAM as you can get for the model itself.
The latter is, unfortunately, currently a bit expensive for any useful amount, but it is still much cheaper - by some technical definition of “much” cheaper - than VRAM per gigabyte.
I have tried it and I agree it’s good. I swap around with a bunch of models, but since the footprint of QWEN3:14b is slightly smaller than GPT-OSS:20b, it’s been my default (for now).
This looks like a decent build, but if the CPU could be upgraded, I would also recommend that. I know that’s not always possible, just a recommendation. Not only would you be on a more recent CPU architecture, but you’d also be able to use DDR5. Memory prices have sky-rocketed, so it could be you end up paying more for DDR4 over DDR5 as well, but you’ll have to research based on your local availability/pricing.
Kind of tempted to go for a R9700. Currently my 6750XT 12GB is limited to about ~14B models. 30B models are painfully slow.
Tho I am currently experimenting with gpt-oss-20b as suggested in this thread, and looks like I’m getting about ~20 tokens/second which is usable. Will keep playing with it.
Alternatively a separate AI Max+ 395 system (16C32T). Or this article suggests an upcoming 392 (12C24T) and 388 (8C12T) variations, with less CPU cores but similar performance for inference:
What would be the rough performance difference between the 32GB on the R9700 vs a 64GB or 128GB variation of strix halo mini pc? The stix obviously having more VRAM
Would be cool it L1T could get their hands on one of those 96GB Huawei Atlas 300I Duo for testing.
