I’d like to get into self hosting LLMs just to learn, tinker, and maybe integrate into other projects but don’t really know where to start of what hardware to run it on. I’d want to gauge the communities option of the best place to start.
I’d prefer it isolated from the rest of my setup i.e. proxmox cluster for self hosted app on some lenovo mini’s, my decently spec’ed PC (Ryzen 7 5700x3d, radeon 9070xt, 64GB DDR4).
Something like a spark is a bit outside of my budget, but a spec’ed out mac mini would be an easier spill to swallow. With a mac mini I feel like I would just be leaving a lot on the table being locked to macOS and paying the Apple premium and it would be harder to repurpose later if I decide I’m not interested anymore or it’s not powerful enough. That really only leave me two options. Build a custom server from new/used parts which may or may not be cheaper given the current market or run it on my current PC.
I’d like to hear people’s thoughts, and apologize in advance if this is a reoccurring topic.
Sounds like you are right at the start of your journey, I’d suggest not “committing” to a direction until you’ve had a chance to actually kick the tires a bit and decide what aspects of LLMs interest you, and what you want to learn more/use more.
Honestly, I’d start with your current PC. Running something like Ollama or LM Studio is free, and doesn’t commit you - but still plenty of learning possible. Or maybe you prefer to generate images/video - you can try this out with ComfyUI. None of these will be fast, especially relative to a web subscription to one of the frontier models like ChatGPT/Claude/Gemini/etc, but it will help dial in what parts of LLM you want to dig into next.
The Mac minis are fun, but I’d only do that if you had non-LLM reasons to want to get into the Apple ecosystem. As for building a custom server, I would suggest stay the heck away from this option until you know what you want to focus on more. At most you might consider adding another Lenovo Mini, maybe with some kind of M.2-to-Oculink with an external GPU if you find the Lenovo is unbearable slow (it probably will be :). At least this way if you decide to invest more heavily in e.g. Spark, Strix Halo, Mac Studio, some giga-workstation, or even clusters of all these - then you could always just add the “AI” Lenovo mini to your existing ProxMox cluster.
Test it out in your current PC first to see if what you want to do is even feasible.
If you want to continue from there, I’d suggest getting something like a 9060 XT 16GB card and using it as a secondary GPU in your current setup if it’s gaming performance you’re concerned with. Other alternatives are things like used RTX 3090s in an X99 system, for example.
You can get pretty far with affordable hardware or hardware you already own.
For reference, a DGX Spark is aimed more towards things like running chat bots for small businesses and handling concurrent users in a minimal footprint afaik. Most people are much better off doing a custom build than using one of those.
The trouble is, Spark and Strix Halo are good for AI development work (ie “does it work”, rather than “I want to use this to do stuff”), and to a certain degree training and fine tuning, but they’re not quick enough for stuff like agentic coding. They’re great for YouTubers making “ZOMG! IT WORKS!” videos running with 8k context in benchmarks, but when you’re doing even medium-sized agentic work with 100k+ context, the acceptable-ish 25-30t/s from the benchmarks drops to 15t/s and you’re left screaming at it to work harder.
Your best bet is to look at the kind of models you want to run and what you want to do with them, and then work out where you’re going with it - you don’t want to get to a certain point and then hit a wall where you have to throw it all away and start again.
For most purposes, used 3090s are by far the most cost-effective approach - they’re excellent performers, and they’ve got enough VRAM to be able to run Q4 quants of 30B models even with one GPU. The only problem with them is cooling; if you want to run two (48GB VRAM is ace), then you’re going to need a motherboard with the second PCIE slot at least four slots away from the main x16 slot. Those are…not easy to come by. Either that or water cool them, which is its own set of problems.
If I was starting again, I’d probably have started with two or three used 3090s on a cheap Threadripper build.
Lastly…RAM. If you’re going to be swapping models in and out, then you want plenty of RAM - 64GB minimum, preferably 128GB+ - purely because constantly loading huge models from disk will put a dent in your day.
With all of that said, I’d probably still end up where I am now - a pair of R9700s. The recent updates to llama.cpp’s Vulkan backend have made multi-GPU stuff with the Qwen3+ models really performant, and 64GB VRAM means I can run Qwen3-Coder-Next-80B and Qwen3-Coder-30B at the same time (I use the former for agentic code work, and the latter for inline edit completion) at sufficient quality with enough context to make it worthwhile.
This is pretty universal.
I used to run a pair of Tesla P40s, keeping them cool without exceeding 80dB and without liquid cooling took a lot of trial and error. Best to get some blower-style cards.
Side note: been thinking of making a hole in one of the P40 heatsinks and putting a GPU blower fan in it
Yes, exactly - it’s possible to mitigate the effect if the cards aren’t too big…but 3090s are the worst of all worlds when it comes to running two or more. Big honkin’ lumps of metal are just not very helpful in this context.
That was one of the main reasons I ended up going for the R9700s. The good news is that you won’t lose money buying used 3090s to get started and then ditching them to upgrade later on.
Just…don’t do Intel. That’s where I started, and it was a disaster; the hardware’s great, the software support is terrible.
Intel Arc works very well in ComfyUI, but getting LLMs running is pure suffering.
I gave up on the latter and found much better perf running Q4_KM 30B and Q5_KL 22-24B models on the 7900XTX without KV offload. Wanted them for SR-IOV anyway, so didn’t matter in my case.
I have yet to find a usecase for any of it outside of character tokens for my TTRPG group though, setting up the hardware is the fun part lol
two 3090s connected with an NVLink for better tensor parallelism under vLLM / ExLlamaV2 (ollama does not support tensor parallelism at all, llama.cpp is in alpha stage so for now it’s even slower than layer-split). This option is the most cumbersome, but allows to buy parts gradually: start with one 3090, if you are into it and have budget then buy the 2nd…
I have this pet theory that, when the bubble bursts, the big AI companies are going to pivot to become model creation factories and most AI usage will become local.
At a rough guess, we’ll see those unified Ryzen mini-PCs bundled as local AI appliances; the best way to run it will still be with dedicated GPUs for a while, but the appliances will be cheaper. And then high-bandwidth NPUs/TPUs will start to go mainstream for completely dedicated machines.
The only way the AI companies can stop it is if one of them actually manages to create AGI, but that’s a very remote possibility on the grounds that LLMs are an evolutionary dead-end if the goal is AGI.
I don’t think it will stop even if they manage to create an AGI, quite the opposite, it will trigger an arms race similar to how today’s weapons are far more destructive.
As for hardware, yup, dual R9700 seems like a good place to be. Or multiple 3090s on risers.
I had hoped that RAM would become cheaper and 48GB GPUs would become commonplace by the end of 2026. Looks like the opposite happened and in expectation of it I did buy enough RAM for my main system.
Performance is nowhere near what I had hoped for (slow tps) but it’s enough that I can claim that I can run them.
Wishing Intel releases a single GPU card with 48gb of fast VRAM at some point. I’m wary of their software ecosystem even though people claim that it’s usable.
I meant that the only way to stop the bubble bursting and AI predominantly moving local is for them to make good on their promises (threats?) and make AGI work. If they do, then it’s just going to be today’s nightmare increased by an order of magnitude or two.
I think we’re pretty much on the same page here
The trouble is, Intel’s cards just aren’t fast enough to compete with Nvidia and AMD in terms of compute; the A770 had the memory bandwidth, but it was half the speed of even the mid-range options from the other guys. When you add the software support problems to the mix, they’re just not practical.
If Intel would just do the right thing and properly contribute to llama.cpp’s SYCL backend, it would fix the software support problem and be cheaper for them. vLLM’s a losing proposition, because it’s so damn hard to get models working half the time, particularly new ones; that’s not a criticism, it’s their strategy - they’re looking at pro-level model hosting as the design goal. However, Intel are coming in late and they need to grab mindshare, and that means getting enthusiasts in…so llama.cpp should be their target.
It doesn’t help that vLLM, outside of Nvidia cards, is actually slower than llama.cpp on single-user stuff when using Vulkan or SYCL. And, if you’ve got an Intel card, you still have to run Intel’s own version of vLLM because they aren’t contributing back to the main project (mainly, I think, because they don’t want to obey the original projects’ contribution rules).
