The Local LLM Matrix: Best Models & Quants by VRAM Tier (<=16GB – 256GB+)

High-Performance Computing Machine Learning, LLMs, & AI
1

The Best Local LLM Matrix

A vram-first reference for running local models on real silicon.

This thread tracks suggested models, mapped by:

  • Usable VRAM footprint
  • Optimal quantization architecture
  • Context headroom via unquantized KV caching

How This Thread Is Managed & Maintained

  1. Weekly Rollups: Every weekend, community submissions, poll shifts, and verified results are reviewed.
  2. Major Release Fast-Tracks: When a significant open-weight release drops, it will be tested. Once context behavior are validated (either through hands-on testing or verified member submissions/DMs), the matrix will updated accordingly.
  3. Direct Submissions & DMs: If you have dialled in an unlisted model, post it or send a DM with your suggestions.

The KV Cache Quantization Rule: When to Use It (and when NOT to)

Baseline Recommendation: Default to unquantized (FP16/BF16) KV cache whenever VRAM allows. Every primary context limit listed in the matrix below assumes full-precision KV cache.

Quantizing the KV cache to FP8 or FP4 halves or quarters its memory footprint, but it introduces numerical noise into the attention mechanisms. Whether that degradation matters depends entirely on the workload:

  • When you SHOULD NOT quantize KV Cache:
    • Agentic Coding & Repo Editing: Quantized KV cache introduces subtle token hallucinations, misses syntax rules, and breaks multi-hop file references.
    • Deterministic Logic, Math & Deep Reasoning: Attention sinks and drift cause chain-of-thought models to lose coherence during long generation steps.
    • Needle-in-a-Haystack (NIAH) & Technical RAG: Critical facts buried in large documents are often skipped or misattributed once context scales past 32k tokens.
  • When KV Cache Quantization IS Acceptable:
    • Creative Writing & Long-Form Fiction: Gist and narrative momentum matter more than verbatim recall. Minor attention drift rarely breaks a story.
    • Casual Chatbots & Roleplay: Conversational flow is tolerant of minor precision loss.
    • VRAM-Constrained Exploration: When stretching a card beyond its native capacity is the only way to run a task, and fuzzy recall is acceptable.

If a workload requires precision, scale context via model weight quantization (such as AutoRound W4A16) combined with RoPE/YaRN (if going over the native model context), while keeping the KV cache in native FP16/BF16.

The Master Matrix (Current Consensus)

Tier 1: ≤ 16 GB VRAM (Single Consumer GPU / Entry Workstation)

Target Hardware: RTX 4060 Ti 16GB, RTX 3060 12GB (tight), RTX A2000, RX 7600 XT, Apple 16GB Unified.

Model & Base Repo Preferred Package / Quant Static Model VRAM Usable KV Headroom Max Native Context Max YaRN / RoPE Context Best Use Case
Ornith-1.5-9B letechlead/Ornith-1.5-9B-INT4-W4A16-AutoRound ~5.6 GB ~9.5 GB 256k 1m General Reasoning, Coding, Long Context
Tier 2: 24 GB – 32 GB VRAM (Dual 16GB, RTX 3090/4090, Dual RTX A4000)

Hardware targets: 2x 16GB GPUs, RTX 3090/4090 24GB, 2x RTX A4000, 36GB Mac Unified.

Model & Base Repo Preferred Package / Quant Static Model VRAM Usable KV Headroom Max Native Context Max YaRN / RoPE Context Best Use Case
Swift-Qwen3.8-27b jamesbrunet/Swift-Qwen3.8-27b-W4A16-AutoRound ~20 GB ~4-12 GB 256k 1m Heavy Code Refactoring, Logic
Gemma-4-26B-A4B-it Community W4A16 ~14 GB ~10-18 GB 256k N/A Instruction Following, Analysis
Tier 3: 48 GB – 64 GB VRAM (Dual 24GB, Quad 16GB, 64GB Mac Unified)

Hardware targets: 2x RTX 3090/4090 (48GB), 4x RTX A4000 (64GB), Mac Studio 64GB.

Model & Base Repo Preferred Package / Quant Static Model VRAM Usable KV Headroom Max Native Context Max YaRN / RoPE Context Best Use Case
Gemma-4-31B-it Community W4A16 / AutoRound INT4 ~18 GB ~30-46 GB 256k 1m Long Document Ingestion
Ornith-1.5-35B-A3B SergiioB/Ornith-1.5-35B-A3B-AutoRound-W4A16-sym-G128-MTP-BF16 ~22 GB ~26-42 GB 256k 1m High-Throughput Reasoning (MTP)
Swift-Qwen3.8-27b jamesbrunet/Swift-Qwen3.8-27b-W4A16-AutoRound ~20 GB ~28-44 GB 256k 1m Deep Codebase Context
Tier 4: 96 GB – 128 GB VRAM (Workstation Multi-GPU / High-Density Compute)

Hardware targets: 4x–6x 24GB GPUs, 8x RTX A4000 (128GB), 2x 48GB Ada/Ampere.

Model & Base Repo Preferred Package / Quant Static Model VRAM Usable KV Headroom Max Native Context Max YaRN / RoPE Context Best Use Case
Swift-Qwen3.8-27b jamesbrunet/Swift-Qwen3.8-27b-W4A16-AutoRound ~20 GB ~76-108 GB 256k 1m Extreme RAG & Repository Ingestion
Qwen3.8-Flash-Next Intel/Qwen3.8-Flash-Next-W4A16-RTN-AutoRound ~168 GB ~23-55 GB* 256k 1m Fast Agentic Workflows
Ornith-1.5-35B-A3B SergiioB/Ornith-1.5-35B-A3B-AutoRound-W4A16-sym-G128-MTP-BF16 ~22 GB ~74-106 GB 256k 1m High-Throughput Production Serving
  • ‘*‘ See “Technical Architecture Note” below for Qwen Next Architecture information.
Tier 5: 196 GB – 256 GB+ VRAM (Enterprise Nodes & Full Workstation Clusters)

Hardware targets: 8x 24GB GPUs (192GB), 8x 32GB/48GB/80GB Enterprise Nodes.

Model & Base Repo Preferred Package / Quant Static Model VRAM Usable KV Headroom Max Native Context Max YaRN / RoPE Context Best Use Case
DeepSeek-V4-Flash-Vision-Exp Intel/DeepSeek-V4-Flash-0731-W4A16-AutoRound ~156 GB ~40-100 GB 1m N/A Multimodal Document & Video Analysis
GLM-5.3-Flash Intel/GLM-5.3-Flash-W4A16-AutoRound ~176 GB ~20-80 GB 1m N/A Complex Agent Systems
Qwen3.8-Flash-Next Intel/Qwen3.8-Flash-Next-W4A16-RTN-AutoRound ~168.0 GB ~28-228 GB* 256k 1m Multi-Million Token Ingestion
Swift-Qwen3.8-27b jamesbrunet/Swift-Qwen3.8-27b-W4A16-AutoRound ~20 GB ~176-236 GB 256k 1m Complete Library & Git Ingestion
  • ‘*‘ See “Technical Architecture Note” below for Qwen Next Architecture information.
⚠️ Technical Architecture Note: Qwen3.8-Flash-Next N-gram Lookahead & Memory Offload

Qwen3.8-Flash-Next implements a high-throughput n-gram lookahead speculative table. This structure requires approximately 95 GB of unquantized index memory in addition to the ~73 GB base W4A16 model weights:

1. Host RAM Offload Mode (Tier 4 / 128 GB VRAM):

  • VRAM Allocation: Only the ~73.0 GB model weights live on the GPUs.
  • System RAM Requirement: The ~95 GB n-gram table is mapped directly to pinned host memory over PCIe. Your host machine must have at least 128 GB of system RAM.
  • Bus Requirement: PCIe Gen4 x4, PCIe Gen3 x8 or higher per card is strongly recommended to prevent host-to-device speculative verification latency from bottlenecking token throughput if offloading n-gram to host memory.

2. Pure VRAM Mode (Tier 5 / 256 GB VRAM):

  • VRAM Allocation: Can load both the weights (~73 GB) and the n-gram table (~95 GB) entirely in GPU memory, consuming ~168 GB of static VRAM before allocating the first token of KV cache.
2

Community Submissions, Hardware Feedback & Thread Evolution

Rather than running rigid polls right out of the gate, this thread is designed to gather real-world, operational data from the community. Automated benchmarks and leaderboards miss how setups actually behave day-to-day - I want to know what you are actually loading up, what you use it for, and how you have it tuned.

Use this space to share your daily drivers, niche workflow solutions, hardware limits, and suggestions on how to improve this living guide.

1. What I Want to Hear From You

Whether you are pushing multi-GPU tensor parallelism in headless servers or running daily local workflows on consumer cards and unified memory laptops, chime in on:

  • What model & quantization do you actually rely on? (e.g., AutoRound W4A16, GGUF Q4_K_M/IQ4_XS, native FP8, unquantized BF16).
  • What is your primary use case? (Agentic coding, large repo ingestion, technical RAG, multi-step math/logic, casual chat, long-form creative writing).
  • How are you serving it?
    • Backend / runtime: vLLM, direct llama.cpp, LM Studio, Ollama, Unsloth Desktop, etc.
    • Architecture notes: Single GPU, multi-GPU split (tensor parallel or row/layer offloading), host RAM offloading, custom RoPE/YaRN factors, or speculative decoding.
  • Context & KV behavior: Did you keep the KV cache unquantized (FP16/BF16), or did you drop to FP8/FP4? Where does the model hold coherence versus fall apart?

2. Standardized Submission Template

To help keep submissions consistent and allow our parsing tools to aggregate community picks into Post #1, use this format when posting your setup:

[COMMUNITY-SUBMISSION]
tier: [≤16GB | 24-32GB | 48-64GB | 96-128GB | 196-256GB+]
hardware: [e.g., 8x RTX A4000 16GB, 2x RTX 3090, 1x RTX 4060 Ti 16GB, Mac Studio 64GB]
backend: [vLLM | llama.cpp | LM Studio | Ollama | Unsloth Desktop | Other]
model_name: [Full model repository or exact filename]
quant_package: [e.g., AutoRound W4A16, GGUF Q4_K_M, EXL2 4.0bpw, FP8]
download_url: [Hugging Face or model source link]
primary_use_case: [e.g., Coding & git refactoring, Deep reasoning, Creative writing, RAG]
stable_context_tested: [Actual token count tested without OOM or coherence collapse]
kv_cache_setting: [Native FP16/BF16 | FP8 | INT4]
throughput_tps: [Approximate generation tokens/sec]
setup_notes: [Launch flags, rope/yarn config, host RAM offload, prompt prefill behavior, quirks]
[/COMMUNITY-SUBMISSION]

(You don’t have to fill out every line if you’re just dropping casual thoughts, but using the block makes it easy to integrate your numbers directly into the master matrix.)

3. Thread Feedback & Suggested Changes

This living guide is built for the forum. If there are things missing, unhelpful, or formatted poorly in Post #1, let me know:

  • What metrics should this thread track? (e.g., Time-To-First-Token prefill latency, PCIe bandwidth bottlenecks, power draw/watts-per-token, speculative draft model pairings).
  • VRAM tier restructuring: Do the current brackets (≤16GB, 24–32GB, 48–64GB, 96–128GB, 196–256GB+) reflect how you build systems, or should I adjust them (e.g., dedicated 24GB single-card tier, 48GB dual-card tier)?
  • Deployment examples: Would container recipes (Docker Compose), systemd service templates, or specific vLLM / llama.cpp multi-GPU launch commands be helpful to include in collapsible details tabs?

Drop your configurations, battle-tested setups, and suggestions below!