DeepSeek Deep Dive R1 at Home!

High-Performance Computing Machine Learning, LLMs, & AI
, ,
138

Here we are around 3 months later and deepseek-ai just released updated improved model weights: DeepSeek-R1-0528!

With Wendell and the l1t crews’ help with hardware I managed to cook up some ik_llama.cpp exclusive quants and release them on huggingface at ubergarm/DeepSeek-R1-0528-GGUF if anyone is interested.

They are a bit large for gaming rigs but more targeted for homelab folks. The bigger one is good for 368GiB RAM rigs and the smaller one for 256GB RAM systems. I am gonna download the smaller one and try to run it at home anyway using mmap() off of my NVMe drive as much discussed above already. Hopefully I can hit ~5 tok/sec.

Quality benchmarks (lower perplexity is “beter”):

  • DeepSeek-R1-0528-Q8_0 666GiB
    • Final estimate: PPL = 3.2130 +/- 0.01698
    • I didn’t upload this, it is for baseline reference only.
  • DeepSeek-R1-0528-IQ3_K_R4 301GiB
    • Final estimate: PPL = 3.2730 +/- 0.01738
    • Fits 32k context in under 24GiB VRAM
  • DeepSeek-R1-0528-IQ2_K_R4 220GiB
    • Final estimate: PPL = 3.5069 +/- 0.01893
    • Fits 32k context in under 16GiB VRAM

Cheers and happy weekend!

3 Likes