Full DeepSeek Q1 with the "IK" version of llama.cpp on AM5 -- No Distills; Just a Quant

I wanted to try this as soon as I built my Threadripper Pro rig.

I was able to fully load up my RTX Pro 6000, and two RTX 4000 ADAs for 136gb of vram total. Not quite the whole model, and not running everything at full speed, but it’s working!

It’s not about to die like my Ryzen 9 build, so that’s good.

Results:



./build/bin/llama-server \
  --model DeepSeek-R1-0528-IQ1_S_R4-00001-of-00003.gguf \
  --alias ubergarm/DeepSeek-R1-0528-IQ1_S_R4 \
  -fa -fmoe \
  -mla 3 -amb 512 \
  -ctk q8_0 \
  -c 16384 \
  -ngl 99 \
  -ot "blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42)\.ffn_.*=CUDA0" \
  -ot "blk\.(43|44|45|46|47|48|49)\.ffn_.*=CUDA1" \
  -ot "blk\.(50|51|52|53|54|55|56)\.ffn_.*=CUDA2" \
 -ot exps=CPU \
  --parallel 1 \
  --threads 16 \
  --host 0.0.0.0 \
  --port 8080

I really jammed it in there. This leaves really no room for context.

This also happend: /home/aatchison/ik_llama.cpp/src/llama.cpp:18969: Deepseek2 does not support K-shift
Found this explanation: Bug: src/llama.cpp:15099: Deepseek2 does not support K-shift · Issue #8862 · ggml-org/llama.cpp · GitHub

BTW, did anyone else notice that the CUDA[N] enumeration is backwards from what the system uses?

 nvidia-smi -L
GPU 0: NVIDIA RTX 4000 Ada Generation (UUID: GPU-b4bd27db-5f40-e482-e6af-9eb03b431c4b)
GPU 1: NVIDIA RTX 4000 Ada Generation (UUID: GPU-653631bd-3198-efb1-2fad-6625508e4e9a)
GPU 2: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (UUID: GPU-8bcb061d-aad7-d9d8-b271-25c0a1bd34fd)

It’s also a bit strange, since that’s the opposite order they are physically in the pcie slots.

I wonder if ik_llamma.cpp/llama.cpp is prioritizing the most capable card as 0.

4 Likes