progressEdd's llms, ai, and dev corner

[progressEdd]

I usually just look it up on the internet. The LocalLLaMA reddit and HackerNews has replies with resources.

For your question, I had to look it up because no one has asked me before.
https://old.reddit.com/r/LocalLLaMA/comments/1d32q63/what_do_the_numbersletters_in_the_quantized_model/

These helped me better understand the voodoo magic of quantz

llama.cpp/examples/make-ggml.py at 5f6e0c0dff1e7a89331e6b25eca9a9fd71324069 · ggml-org/llama.cpp · GitHub

GitHub - ggml-org/llama.cpp: LLM inference in C/C++

phi3:14b and phi3:medium is actually (1e67dff39209) 14b-medium-4k-instruct-q4_0

Old quant types (some base model types require these):
- Q4_0: small, very high quality loss - legacy, prefer using Q3_K_M
- Q4_1: small, substantial quality loss - legacy, prefer using Q3_K_L
- Q5_0: medium, balanced quality - legacy, prefer using Q4_K_M
- Q5_1: medium, low quality loss - legacy, prefer using Q5_K_M
New quant types (recommended):
- Q2_K: smallest, extreme quality loss - not recommended
- Q3_K: alias for Q3_K_M
- Q3_K_S: very small, very high quality loss
- Q3_K_M: very small, very high quality loss
- Q3_K_L: small, substantial quality loss
- Q4_K: alias for Q4_K_M
- Q4_K_S: small, significant quality loss
- Q4_K_M: medium, balanced quality - recommended
- Q5_K: alias for Q5_K_M
- Q5_K_S: large, low quality loss - recommended
- Q5_K_M: large, very low quality loss - recommended
- Q6_K: very large, extremely low quality loss
- Q8_0: very large, extremely low quality loss - not recommended
- F16: extremely large, virtually no quality loss - not recommended
- F32: absolutely huge, lossless - not recommended

I now try to use Q5_K_M and larger models. General idea is greater accuracy at the cost of more Vram and storage space. From what I can tell most base models are Q4_0.

ggml-org/llama.cpp/blob/5f6e0c0dff1e7a89331e6b25eca9a9fd71324069/examples/make-ggml.py#L16C1-L37C51

The llama.cpp wiki is also good

Tensor Encoding Schemes

This guy used deepseek-r1 to answer the same question lol
https://old.reddit.com/r/ollama/comments/1ifv422/can_someone_clarify_the_subtypes_of_models/mak8at8/

Here’s a clear breakdown to help you navigate model types and quantization:

1. “Text” vs. “Instruct” Models

   Text (Base) Models:
   These are general-purpose models trained to predict the next token in a sequence (e.g., GPT-3). They’re good for text completion (e.g., writing stories, code, or emails) but may struggle with direct instructions.
   Example: llama-2-7b, mistral-7b-v0.1

   Instruct Models:
   Fine-tuned on instruction-response pairs to follow prompts accurately (e.g., “Summarize this article”). They’re better for tasks requiring structured outputs, Q&A, or step-by-step reasoning.
   Example: llama-2-7b-instruct, mistral-7b-instruct-v0.2

When to use which:

• Use text for creative generation.
• Use instruct for task-oriented prompts.

2. Quantization Explained

Quantization compresses models by reducing numerical precision (e.g., 32-bit → 4-bit). Lower precision = smaller files and faster inference but potential quality loss.
Common Quantization Types (GGUF/GGML format):

Naming Scheme: qX_Y or qX_K_Z
    X: Bits per weight (e.g., q4 = 4 bits).
    K: Indicates "k-quant" (advanced method balancing accuracy/size).
    Z: Block size (S=small, M=medium, L=large). Smaller blocks retain more precision.

Type	Bits	Use Case	Quality-Size Tradeoff
q2_K	2	Extreme compression (low RAM devices)	Lowest quality
q3_K_S	3	Lightweight tasks on weak hardware	Low quality
q4_0	4	Default balance for most users	Good quality, smaller size
q4_K_M	4	Better than q4_0 (recommended)	Balanced
q5_K_M	5	Near-fp16 quality	High quality, moderate size
q6_K	6	Minimal quality loss vs. fp16	Largest quantized size
q8_0	8	Almost identical to full precision (fp16)	No practical gain over fp16
fp16	16	Unquantized (original model)	Best quality, largest size

Key Takeaways:

Start with q4_K_M or q5_K_M for a good speed/quality balance.
Use q2_K/q3_K only if hardware constraints demand it.
Avoid q8_0/fp16 unless you have high-end GPUs.

3. How to Choose

   Hardware:
   <8GB RAM: Use q4_0 or lower.
   8–16GB RAM: q5_K_M or q6_K.
   >16GB RAM: Consider fp16 if quality is critical.
   Task:
   Simple chatbots: q4_K_M.
   Complex reasoning: q5_K_M or higher.

Quantized models are available for both text and instruct variants. Always match the quantization to your hardware and task needs!

Let me know if you need specific examples!

Edit: Credit to my buddy DeepSeek-R1