tl;dr;
Anecdotally bigger models can be quantized down more than smaller models without major quality losses. Except for R1 671B, I tend to avoid going lower than a 4bpw quant e.g. no lower than IQ4_XS.
Long Winded
Yeah that is a good question. The answer to any good question is always “it depends” lol.
When thinking about what LLM to use consider:
- What are the breakpoints in your hardware setup
- For example, a 70B@IQ2 can fit into 24GB VRAM and run pretty fast with 8k context, but a 70B@IQ4 won’t fit entirely into VRAM so be much slower.
- How much context do you really need for the job? 8k is honestly enough for a lot of mundane one or two shot questions like writing sample code or a short story. Most of the “long context” models still don’t perform super well beyond 32k context at best imo.
- Do you need “quality” or “speed”? If you need speed, you can use
--parallel N and multiply the context by N so each concurrent slot will run simultenously and increase aggregate tok/sec despite each individual generation being slower.
- Are you feeding in context from a RAG or web search results, or are you relying on the model itself to have knowledge? Many smaller models have decent logic and reasoning capability but very little knowledge. Bigger models often have both builtin knowledge and even better reasoning.
On my personal rig (3090TI 24GB VRAM + 96 GB DDR5@88GB/s + 9950X + single Gen 5 x4 NVMe SSD) these are my most common setups:
Best Quality
If I want the best quality answer, and am willing to wait while it runs for up to an hour or so in the background, I’ll go for the unsloth R1 671B @ R1-UD-Q2_K_XL mentioned above at 8k context and let it chug at 1-2 tok/sec.
I use this to refactor python code, throw a man page at it and ask it to write the command, or for higher quality creative writing and prose.
This is about the only model that will come up with more varied answers when prompted the same. Most smaller models seem to give almost the same answer every time.
Good Quality
If I want okay quality, but somewhat faster I go with something like bartowski/Athene-V2-Chat-GGUF @ IQ4_XS. It doesn’t fit entirely in 24GB VRAM with 8k-12k context but runs around ~4-5 tok/sec. Also because it is not a <think></think> reasoning model it immediately spits out some python code.
Okay Quality Fast
If I want fast or longer context I go with a 32B@~4bpw e.g. bartowski/FuseO1-DeepSeekR1-QwQ-SkyT1-Flash-32B-Preview-GGUF@Q4_K_M. This 32B size is the sweet spot imo for 24GB VRAM systems. I would probably choose this over a 70B@2bpw.
Fast Long Context
For when I want parallel inferencing to scrape multiple websites concurrently and summarize it all or any kind of RAG type thing I go with a long context 8B or even 14B e.g. mradermacher/Qwen2.5-14B-Instruct-1M-i1-GGUF@IQ4_XS. Many of the Qwen2.5 models are quite performant even at smaller sizes and available at multiple breakpoints depending on your target hardware. These models have very little knowledge, but if you inject good clean context they can pick out the best answer from those you provide in the prompt.
Little Guys
The little 1.5B and smaller might make sense for mobile devices or similar where you mainly just want it to complete sentences or do a single very specific task for which it has been custom fine-tuned.
Conclusion
Okay way too much, but it might give a feel for how you can choose the right model for the job. Every few months I go through and replace the models in each class with whatever the new SOTA hotness is trending on huggingface.
Cheers!
lol…