After using my 55W 16gb VRAM GPU in the laptop for ~4-5 months, I can tell you that while it is quite easy to get smaller parameter models running (2B, 4B, even decent 8B), you have to focus on larger Q5-Q8 quants as the quality really takes a hit - especially with toolcalls. As a matter of fact, with only 12GB, I’d say to not plan on any agentic multi-turn toolcalls.
The good news is you can use a frontier model on the cheap to make a plan, and then use local smaller models to execute each step. I’ve never done this, but I hear it’s a common thing now.
When picking a local model for 12/16GB, you have two choices:
- Small parameter (4B, 9B, etc) as Q8_0 for high performance prompt processing and token generation, but nowhere as smart - aka, creative - as larger models.
- Larger models (27B, 30B, 35B, etc) at a significantly compressed Q4_0 or even lower to get it to fit in 12GB (e.g. Q2_0). The compression creates a whole slew of hallucinations to watch out for, but is wild to chat with. LOL. Basically, low compression makes these larger models “too talkative” IMO, overthinking simple problems.
I’m on the side of running high quality (Q8) smaller models. It’s far more consistent - just limit your work for small snippets and it’s fantastic.
With that said here’s my experience.
You want a re-trained models based off of Unsloth’s UD variants. These have heavier weights on coding.
Qwen3 3.5 9B @ UD Q8 model (~9GB + ctx) was excellent - as long as you review all the code (there will be issues here and there, every time, so you gotta check it all).
The Qwen2.5 Coder 7B Q8_0 (9GB) and it’s smaller 3B Q8_0 was great to use because their very fast speed more than makes up for the amount of errors. The cycle starts with codegen, then your review, you tell it to fix, and it cranks out a fix very fast. Think small refactorings and code stubbing, not entire stack full app creation.
I’ve actually written most of my codegen code with this model to date, mainly because lately I’ve been switching lots of newer models.
Skipping ahead, recently I have been running the Kappa 20B 131k MXFP4 model released here on these forums. It’s based on Google’s GPT OSS 20B. Wendell says it fits entirely into 12GB, but my llamacpp conversions seem just a tad too large for 12GB. The jury is still out for VSCode usage as I’ve had great codegen without too many issues, but then I’ve had a few projects go completely haywire. Most likely my settings like Temp.
For llama.cpp (llama-server), you can convert it to GGUF, or I can post mine if you need a starter model. It converts quite easily locally though.
I’m also now trying out two more models, such as the new Gemma 4 12B UD Q6_0 (the Q4_0 may fit in 12GB, if not Q2 would) and Qwen3.6 27B @ UD Q2_0.
That qwen3.6 27B with Q2 is surprising capable! However, it’s pretty slow. The Qwen 35B A3B UD Q2 fits nicely and is a nice speed boost, however it’s providing more wild codegen output and makes me have to review far more code than 27B Q2 is. However, neither will fit on 12GB even at q2_0.
I am not sure which I’m landing on yet. This is part of the fun, finding which model fits your coding style.
I use llama-swap, which makes it easy to swap out models in realtime. So when I use one for A while.