I have been interested in hosting my own LLM for the past year now. The time has finally come where I have hardware laying around which I think is up to the task. I have been reading and watching videos but am feeling overwhelmed about where to start, as I have a specific goal in mind.
Input from the community would be invaluable here. These are my objectives.
Self Hosted LLM (only for me, I won’t be sharing)
I only want\need the model to assist me with IT related tasks (Homelab stuff, Azure for work, light coding, etc) in other words, I would love to super specialize the model and ignore things I don’t need it to do
Be able to train it on my specific environment (share my homelab configs, etc.)
I have a Intel Core i7-13700K and a GTX 4080. I can also have up to 64gb of DDR5 if needed (I have 32GB installed currently).
My preliminary testing on the 4080 seems to indicate I can get some very good tokens per second when using a small, 12GB model. I know if such a small model is being used as a general LLM, the accuracy will be poor, so I am hoping there is some way to specialize it, keeping it fast but also accurate.
I think LM-studio is a great starting point, it leaves the knobs and tinkering a lot more accessible and user friendly compared to ollama. You can use it to try out models, serve them over LAN, and experiment with things like openwebui or other programs that use lm-studio as an openAI compatible API endpoint.
Since you mentioned IT, briefly want to plug llm by simonw for a way to interact quickly with self hosted LLM in a terminal or on scripts. I’ve tried a bunch and this seems to be the best option for now.
For which models to pick, lm-studio makes it easy to try and compare a bunch. I use a 5090 and have the best experience with Qwen3 30B A3B 2507, there are various quantizations and instruct/thinking/coder you can try. GPT-OSS 20B should squeeze on a 4080 I think, it’s a good baseline for comparison. Use lm studio to tweak the context, CPU offload, and optimize for your use case. For the IT customization, a strong system prompt should get you most of the way there without diving too much into the weeds of post training.
Lm-studio has a simple chatgpt-like interface that has a sidebar of chats that contain a list of prompts and responses just like you’d see from a big lab.
Each time you send a new prompt, it will append the history of the chat + your most recent request + the system prompt (can easily be set in lm studio)
So persistence across reboots isn’t really a concern, you could even switch models mid chat and it would “remember” what you were talking about before.
So the system prompt would be something like instructions to provide helpful IT feedback, how long the response should be roughly, formatting, etc and here you could include system details, paths, ip addresses. To take it a step further, you can have a reference file for different parts of system config that you feed to the LLM when relevant. This is partially Claude code works with the CLAUDE.md file.
You could write the reference file yourself, or put the crucial info in the system prompt, or build some tooling to work like Claude code that will gather info and update the reference material as you go.
In theory it’s pretty simple, everything just gets thrown into the LLM context window at request time. Doesn’t matter if it’s in the chat history, copy pasted from an older chat, or fed into the LLM through a markdown file.
Yes the open source large models as well as custom post trained models are available on hugging face. Probably a lot more information and guides as well.
The large open source models already have so much information about programming / IT baked into their training data the challenge is really just how big of a model can you fit on the GPU. There may be some use cases for post training a custom model but I’m not sure how we’ll it would work on a 4080 as you’d already be dealing with a quantized model instead of the full 300B parameter data center model. This is getting a bit above my depth though.
I think for your use case tuning the CPU offload, system prompt, and maybe building some tools or agents would meet your needs though.
+1 LM Studio it’s a great all in 1 model endpoint and UI. I like to use LM Studio to browse models and check for the hardware compatibility. For personal use it is great, but if you want to use it at work make sure you get approval your IT and manager
When you need to use it at work, I recommend checking out ollama
It is MIT licensed, so you’d only need approval from your manager and IT
For general instruction following and structured outputs, gemma’s instruction tuned models are a great small model that you can run on a laptop, gemma 3 seems to do a slightly better job at language translation
Personally I like gemma2:9b-instruct-q8_0
As you explore models, you’ll probably come across the labels. I wrote a explanation a while ago
This is still a great process I follow to quickly try models
For agents and tool calling, I found that gpt-oss is good for SQL queries.
I found that I don’t need to steer it towards tools as hard as qwen2.5 when I developed with it a couple of months ago
These were awesome replies and you have helped me tremendously. I got my initial set up finished. The 4080 is extremely fast when using a 12gb model. I have loaded some documents of my homelab as a knowledge base so the LLM has context when I’m asking questions. It’s pretty damn cool.
I have a ton of testing to do. I’ll update this post when I’m further along. Thanks again folks, I needed the inspiration.
There are also free applications like AnythingLLM that allows you to create workspaces that include documents, graphs, databases, you can then embed information in a vectorDB so that your model running in LM studio can use more sophisticated RAG. I use this for a variety of purposes but one of them is giving your AI a document repository and conditionals like Query vs Chat where Query will not make stuff up and depending on how strict you set the matching logic it will just tell you it did not find anything vs be creative.
Yes. I am using Open WebUI inside a docker container and running qwen2.5:14b. The performance is great but the accuracy is certainly not at ChatGPT levels. I have some honing to do!
I’ve got wondered if there was a way where openwebui could use a smaller model to detect if a users request needs specialized information and hit the knowledge that is uploaded to open webui… could prove amusing if they could figure it out.
A few things that might help you squeeze more out of that 4080 16GB setup:
Model picks for 16GB VRAM:
Qwen2.5-Coder-14B — if coding/IT tasks are your main focus, this is probably the best coding-specific model that fits comfortably in 16GB. Noticeably better than the general Qwen2.5 14B for anything code-related.
Mistral-Small-24B at Q4_K_M quantization — fits in ~15GB VRAM and punches well above its weight for general reasoning. Good all-rounder if you want one model for everything.
DeepSeek-Coder-V2-Lite-Instruct — another strong option for coding tasks specifically, and it handles longer context windows well.
Quantization matters a lot:
Q4_K_M is the sweet spot for 16GB cards. You lose maybe 2-3% quality vs FP16 but you can run models nearly 4x larger. Avoid Q2/Q3 — the quality drop gets steep. Q5_K_M is worth trying if the model barely fits, but Q4_K_M is usually the best tradeoff.
System prompts are underrated:
For specialized IT/homelab use, a detailed system prompt makes a massive difference. Something like: “You are an expert systems administrator specializing in homelab infrastructure, Azure cloud services, Docker, networking, and Linux administration. Provide precise, actionable commands and configurations. Always specify which OS/distro commands are for.” — this alone can noticeably improve response quality for your use case.
RAG for your specific environment:
Since you mentioned wanting to train on your homelab configs — you don’t actually need to fine-tune. RAG (Retrieval Augmented Generation) is what you want. Open WebUI has built-in RAG support now — you can upload your config files, network diagrams, runbooks, etc. and it’ll pull relevant context into the prompt automatically. AnythingLLM is another good option if you want more control over chunking and embedding. This is how you get the model to “know” your specific environment without the complexity of actual training.
The 4080 is a solid card for this — the 16GB is the main constraint but with the right quant and model choice you can get surprisingly close to ChatGPT quality for specialized tasks.