PC Parts Picker but for Servers and self hosted LLMs?

High-Performance Computing Workstations & Servers
,
1

I’d like to do some shopping for either used or new parts to build a beefy home server/lab I can run modern LLMs on locally. Most of my home servers in the past have been retired gaming machines doing the home server function as a half-life. Which is fine for plex and navidrome, but the memory requirements for LLMs is so high that I’d like to shop motherboards that can support 100GB+ that seems to be required as well as the PCIe pass through for GPUs.

How does one go about planning out their home server builds?

I don’t see a lot of recent posts on this topic but maybe I am bad at searching.

2

LLMs don’t only have high memory capacity requirements - they also have high memory bandwidth requirements.

Specs for the latter aren’t quite as well advertised. What I have found working for me is to look up memory bandwidth specs from server CPUs of the generation you’re considering. Note, that all memory channels of such a server need to be populated to reach these advertised memory bandwidths.

Examples:

  • Haswell/Broadwell: 68 GB/s, 4 memory channels
  • Zen 3 EPYC (Milan): 204.8 GB/s, 8 channels
  • Zen 4 EPYC (Genoa): 460.8 GB/s, 12 channels
  • Zen 5 EPYC (Turin): 614 GB/s, 12 channels
3 Likes
3

So here’s what I’ve sketched so far, what would be good for GPU?

Supermicro MB MBDH12SSLCTO Socket SP3 AMD EPYC 7003 Milan/EPYC 7002 Rome Max2TB DDR4 PCIE ATX
https://www.newegg.com/supermicro-mbd-h12ssl-ct-o-amd-epyc-7003-7002-series-processor/p/1B4-005W-00909?Item=1B4-005W-00909

$755


AMD EPYC 7642 2.3 GHz 256MB L3 Cache Socket SP3 225W 100-000000074 Server Processor
https://www.newegg.com/amd-epyc-7642-socket-sp3/p/N82E16819113636
$1000


RIMLANCE RAM 256GB(4X64GB) DDR4 3200MHz PC4-25600 2Rx4 1.2V 288-PIN ECC Registered RDIMM Server Memory KIT Compatible with Dell PowerEdge XR11 Rack Server
https://www.newegg.com/rimlance-256gb/p/1X5-00FS-007J3?Item=9SIAU3BKEF7023
$515


4x 4TB SSD 
Silicon Power ACE A58 SLC Cache Performance Boost 2.5" 4TB SATA III 3D NAND Internal SSD (SU004TBSS3A58A25SN)
https://www.newegg.com/silicon-power-1tb-ace-a58/p/N82E16820301489
about 755.96

power supply 
$250
4

I did a thing about this, as @jode said:
newer EPYC is better, especially for RAM intensive applications.
Latency is not king per say, but the more the marrier as most algo’s are optimized for 4-6 year old hardware (that the undergrads are running)

https://forum.level1techs.com/t/2025-reusing-servers-and-workstations-wiki/

5

If you’re going to run the LLMs on GPUs, then system ram speed/latency is less of an influence on performance. The system ram is just used for offloading onto, then back from, the GPUs.

When I built my system I wanted to prioritize idle power efficiency. So I went with a lower TDP CPU. EPYC 7313P (155w).

I see very little action on my CPU when the GPUs are doing their thing.

I bought six XFX RX 7900 XTX GPUs. AMD GPUs are supported by ollama. So inference is easy. Beyond that, I’ve had to find the right combo of drivers and tools to create my own training runs. I got it all working. AMD support continues to improve.

1 Like
6

If you haven’t bought the hardware yet you can get the CPU for about $300 via eBay and could probably do better than $500 for the memory. In the process of building a new Epyc machine myself and got a 7252 for $50, lowest on newegg was $300-$400, just want to be sure it’s not a locked processor.

1 Like
7

I’d check this thread for deepseek R1, you probably could ask @ubergarm some specificquestions after you read through it

If you want to dip your toes, you can always try a smaller model on a gaming gpu. For every 1 GB of vram, you can run 1B parameters

Self plug for referencing how parameters and Quantization works

1 Like