DeepSeek R1 671B backed by fast read IOPS?

ubergarm · January 31, 2025, 11:02pm

Yeah the landscape of inference engines and models is overwhelming. I’ve tried many of them out locally and in general llama.cpp is the upstream source for many of the other projects. It also has some of the best capability for doing partial offload between GPU/RAM and most importantly for this test is it supports mmap() letting the big model files sit on a disk while taking advantage of the filesystem cache.

Your 128GB is not quite enough to fully hold even the smallest dynamic quants of the real R1, but if you have a decent disk also you could give it a try for fun anyway.

The smallest version of the actual real DeepSeek R1 that I know of is UD-IQ1_S 1.58bpw that weighs in at 131GiB.

You could start with that and give it a try before moving up to a larger sized quant.

If you are interested, give this a spin:

Procedure

First, download all these 131GiB of GGUF files and put into a directory on your fastest drive: unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S

While that is going, get yourself setup with llama.cpp inference engine like so:

# download the source (i rebuild daily at this point lol)
git clone [email protected]:ggerganov/llama.cpp.git
# if above doesn't work use the url instead e.g.:
# git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

# build for CPU inference only to keep it simple for now (no CUDA etc.)
cmake --build build --config Release -j $(nproc)

# confirm it built by looking for all the fresh binaries
ls -lah build/bin/

# try to start up the API server (which hosts a nice web interface too)
./build/bin/llama-server \
    --model "/mnt/fastdrive/models/unsloth/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf" \
    --ctx-size 8192 \
    --override-kv deepseek2.expert_used_count=int:8 \
    --cache-type-k q4_0 \
    --cache-type-v f16 \
    --parallel 1 \
    --threads 16 \
    --host 127.0.0.1 \
    --port 8080

--model points to the first file of whatever GGUF model you decided to download
--ctx-size more is better but takes up precious RAM quickly, try between 2k to 16k to start out
--override-kv can reduce expert_used_count from default of 8 to say 4 for faster inferencing at a cost of likely lower quality
--threads i’d recommend to set to however many real cores your CPU has (not SMT/hyperthreads)

If our good friend OOMkiller keeps killing the process, you can check with sudo dmesg -T. I tried fiddling with the various sudo sysctl -a | grep overcommit settings, but in the end used Linux cgroups to put an artifical RAM cap on the llama-server process which made it decide to mmap() instead of malloc() and barfing.

So only if you are OOMing, you could try prepending this to the command e.g. sudo systemd-run --scope -p MemoryMax=88G -p MemoryHigh=85G ./build/bin/llama-server .......

Once it is up and running, it will spit out the IP address and port. Open that up in your web browser, in this case http://127.0.0.1:8080 and you can try it out.

If this is your first time running an LLM, you could start out with a smaller GGUF model like one of the little Distill variants mentioned by @eastcoastpete . Keep in mind the Distill’s are NOT R1. It’s confusing haha.. I’ve had good luck with the IQ4_XS quant of bartowski/FuseO1-DeepSeekR1-QwQ-SkyT1-Flash-32B-Preview-GGUF which is a 32B CoT frankenmerge of some recent models including the Qwen2.5 Distill…

Good luck and curious how you make out! Holler if you get stuck on anything! Cheers!