Wanted to start this topic on why build a local LLM, what’s the use case, etc… I’ve been thinking about building a local LLM, but I can’t think of a real use case of it thus far. Wanted to see what the community has to say on the topic. I have some spare hardware laying around to include a Ryzen 3950x, 64GB DDR4 3200, 3080ti, 3070ti, and 4070ti video cards.
Making own tools, checking your own code, translating books that contain smut that you bough with cash that you would rather no one know you are reading then the llms are a nice tool. You will have to build a bunch of other infra as well if you do code and haven’t done it before.
The biggest advantage of local llms is that they are private and won’t go down when clippy decides to take most of microsoft online services down.
Its also not magic and will do dumb things so plan accordingly.
Those specs can run Qwen 3.8 flash next with strata at a nice pace, which is a rather capable model.
1 Like
Do you mean train your own model from scratch, or run an open-weight model? I’m going to assume the latter.
The use case is… everything you would use an LLM for anyways. So long as you have enough resources to run Qwen3.8-27B or better (as purkkaviritys said, you can run Qwen3.8-Flash-Next 125B easily) you can get a lot done. The massive cloud models are better for massive project planning and things like that, but if you just need a code assistant or something to help chew through data, the small ones do great. And you aren’t streaming all your data to someone else’s servers, and just trusting that they secure it properly, don’t use it to profile you, and that there’s no MITM attack that can simply decode the stream (there is for Claude Code, for example). I use Qwen-Flash-Next when I need to handle bulk data dumps/uploads that may contain private information, because I don’t like sending all that to Anthropic. Even putting bad actors aside, I just don’t trust Anthropic itself.
If you live somewhere with cheap electricity, it’s also much cheaper. Inference is basically free at work, because it’s a 300,000 square foot office and warehouse building, so the ~400W or so my local box pulls isn’t even a drop in the bucket compared to the HVAC system alone.
2 Likes
I have been in this business long enough to have companies “retire” services I use and enjoy all the time. So if it can be taken away from me, I do my best not to depend on it. It’s why I locally host nearly everything, including an LLM.
Since Qwen3.8-27b is good enough for me to have it do busy work, and it’s basically free to run it as much as I want, I am using it quite a bit. It’s saving me a bunch of time.
Harder one-off task go to a cloud LLM. Only so much a local AI can do. But on complicated stuff I do ask a cloud LLM to figure out the problem and write up a playbook on how the work should be preformed, then hand that off to a local LLM to do the work.
To avoid paying a subscription to an (already) billionaire.
1 Like
Censorship → Local LLMs are often less censored.
Asked Perplexity Pro for a filmography of an adult movie actress the other day. (strictly scientific research)
It refused and instead gave me a wall of text on the legality of pr0n movies in my country.
Few months ago it would have just compiled the list. (which is freely available on the web anyways)
You can use it for a 24/7 ai assistant, hook it up to something like openclaw or hermes
Setting up and using an LLM local is a great learning experience in itelf - then there’s what it opens doors to that you can jump into and learn about after it’s setup and working. Regardless of your interests a local LLM really does give you a lot of freedom to “tinker” more so than piping a cloud LLM into your setup. - People really are only starting to scratch the surface of what they can / want to do with a local LLM. It’s kind of the new “Overclocking” of the time, and I’m here for it.
I live in the US where electricity is roughly $0.16/kw during the summer and $0.14/kw during winter, but will bump up in the next couple of years due to hyperscale data centers being built near by.
I think the goal I’m trying to ascertain is use case per cost benefits. Is it worth building a local LLM? I don’t really code, but it’s not something I’m shy about jump into to learn. I see AI as an automation tool, but there’s not a whole lot of automation I’m currently needing.
I have a custom router, 10 GB LAN, 100TB TrueNAS server, and a Proxmox server for various VM’s
And that’s my hurdle. What the hell do I do with it after I build it? I might go “Field of Dreams” on it and find out after? I built a Stable Diffusion server couple of years ago to fool around with image creation/editing. While it was fun for a duration, it ended up serving no purpose and was scrapped.
That’s basically what I did.
Use case example. I am ripping DVD/BR disc of series for my JellyFin server. Some are hundreds of episodes. I manually just dump the ripped video files into a folder as I was ripping them. I told Hermes to sort though that folder, look around the 1 minute mark where they display the episode title and use vision to get what episode it is, cross reference that to the Wikipedia article to get the series and episode number, then rename the file to my naming convention. Also transcode the video to X format, making sure to get the English audio track. Then put the finished videos in a folder structure based on the season. Then give me a report to confirm I have all episodes and they are all the expected size.
It saves me so much time.
2 Likes
- It’s cool new tech. The most important reason. You can make it a search engine, coder, automate your PC with a little more creativity, like natural or nuanced language and inputs. Once you get a machine that can perform all these tasks, all of a sudden you got plenty of ideas and projects.
- Some of the things you wanna do with a giant frontier LLM don’t warrant the compute or price. Small models like your Qwen 3.8s, Nanbeiges, whatever, they can get you where you’re going while only paying the electric bill.
- Kind of opinionated, but bigger general LLMs/VLMs are kinda like encyclopedias, trying to put all the knowledge of the world into one source. Having this knowledge in a big company or government’s hands, all but your own, could be a little disadvantageous. Doing yourself a disservice, even, I dunno. This part depends on if your cool with the makers of LLMs going scorched earth on the whole internet to bring these to you.