PC Parts Picker but for Servers and self hosted LLMs?

[progressEdd]

I’d check this thread for deepseek R1, you probably could ask @ubergarm some specificquestions after you read through it

DeepSeek Deep Dive R1 at Home!

Intro There’s a lot of interest in running a version of the full DeepSeek R1 671B LLM at home on high end gaming rigs, workstations, and moderate home lab setups. This thread is a deep dive dump of a couple weeks experimentation including benchmarks, test metrics, and some thoughts on current limitations and potential future improvements. Key Find…

If you want to dip your toes, you can always try a smaller model on a gaming gpu. For every 1 GB of vram, you can run 1B parameters

progressEdd's llms, ai, and dev corner

I usually just look it up on the internet. The LocalLLaMA reddit and HackerNews has replies with resources. For your question, I had to look it up because no one has asked me before. https://old.reddit.com/r/LocalLLaMA/comments/1d32q63/what_do_the_numbersletters_in_the_quantized_model/ The llama.cpp wiki is also good This guy used deepseek-…

Help installing local LLMs

Heuristic for model size, parameter count, and GPU VRAM Agreed. To add a heuristic, I usually look the file size on the downloads is approximately the amount of VRAM you need. [image] Another is that for every 1 Billion parameters ~ 1GB of Vram. Though the model size is a much better indicator of if the model will fit in your gpu. There’s also …

Self plug for referencing how parameters and Quantization works