What’s the cheapest way to get a high scaling efficiency tensor parallel rig for 8 GPUs?

High-Performance Computing Machine Learning, LLMs, & AI
1

Hey all, posing the question up front: what’s the cheapest way to get a high scaling efficiency tensor parallel rig for 8 GPUs?

So, I recently impulse bought 8x r9700 with the plan to run tensor parallel tp8 on ~200GB models. It seems like for inference performance the best implementation would be to optimize for low latency and decent bandwidth through a pcie 5.0 fabric where p2p communications never hit the cpu. I’m thinking on skimping on the host as a result (cheapest cpu+mobo that exposes a pcie 16x 5.0 slot and 64gb ram).

I’m looking at the githib project “local-inference-lab/rtx6kpro/blob/master/hardware/topology.md” for inspiration using a single Microchip Switchtec PM50100 running 8x lanes per gpu flike this:

Ryzen host (64gb ddr5 ram, cheap mobo + processor)
│
PCIe 5.0 x16 slot
│
x16 → 2× MCIO x8 card
│
▼
┌────────────────────┐
│ C-Payne PM50100 │
│ 100-lane Gen5 │
│ switch │
└────────────────────┘
│ │ │ │ │ │ │ │
x8 each over MCIO
│ │ │ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼
PCIe x16 mechanical
endpoint adapters
│ │ │ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼
R9700 × 8

And housing all of the bits in a small cheap gpu cluster housing like a MM-A515-CPW and swapping the electronics.

I reached out to guva systems to see if I could get a quote for a true 16x lanes per gpu switch since thats a clear optimization option, but I’m unsure what the performance unlocks for vllm optimization would be.

I’m a bit of a greenbeard in this area though, so I could use advice on hardware choice optimizations really matter for a vllm-radiance deployment on qwen3.8-next-flash or quantized glm5.3-flash like I am currently targeting.

2

Just realized the title is now misleading to the post content and I can’t edit it, I wandered a bit while researching this post. :sweat_smile:

3

Realizing now the importance of caching and how thats usually done with system ram. Also seeming to find that a pcie 4.0 host might still be able to support a pcie 5.0 gpu-gpu fabric. So now I’m considering a cheaper pcie 4.0 threadripper base system with 256 gb system ram. Likely need a new housing solution or run a rack.

4

For an 8-GPU TP8 setup, focus less on the CPU and more on the PCIe topology. If the GPUs can communicate directly through a Gen5 switch without routing P2P traffic through the host, that should matter much more for tensor-parallel inference than having an expensive CPU.

The important thing is to benchmark the actual topology rather than assume x16 per GPU will automatically translate into better vLLM performance. Check GPU-to-GPU bandwidth/latency, P2P support, NUMA effects, and how the workload scales from 4 → 8 GPUs. Also make sure the motherboard and BIOS expose the required PCIe lanes and ACS/IOMMU settings correctly.

For ~200 GB models, memory capacity and inter-GPU communication can easily become the bottleneck, so I’d validate the switch configuration with NCCL P2P tests before spending more on the host. A cheap CPU is reasonable if it doesn’t become the bottleneck for tokenization, scheduling, or data movement.

1 Like