Hey all, posing the question up front: what’s the cheapest way to get a high scaling efficiency tensor parallel rig for 8 GPUs?
So, I recently impulse bought 8x r9700 with the plan to run tensor parallel tp8 on ~200GB models. It seems like for inference performance the best implementation would be to optimize for low latency and decent bandwidth through a pcie 5.0 fabric where p2p communications never hit the cpu. I’m thinking on skimping on the host as a result (cheapest cpu+mobo that exposes a pcie 16x 5.0 slot and 64gb ram).
I’m looking at the githib project “local-inference-lab/rtx6kpro/blob/master/hardware/topology.md” for inspiration using a single Microchip Switchtec PM50100 running 8x lanes per gpu flike this:
And housing all of the bits in a small cheap gpu cluster housing like a MM-A515-CPW and swapping the electronics.
I reached out to guva systems to see if I could get a quote for a true 16x lanes per gpu switch since thats a clear optimization option, but I’m unsure what the performance unlocks for vllm optimization would be.
I’m a bit of a greenbeard in this area though, so I could use advice on hardware choice optimizations really matter for a vllm-radiance deployment on qwen3.8-next-flash or quantized glm5.3-flash like I am currently targeting.
Realizing now the importance of caching and how thats usually done with system ram. Also seeming to find that a pcie 4.0 host might still be able to support a pcie 5.0 gpu-gpu fabric. So now I’m considering a cheaper pcie 4.0 threadripper base system with 256 gb system ram. Likely need a new housing solution or run a rack.
For an 8-GPU TP8 setup, focus less on the CPU and more on the PCIe topology. If the GPUs can communicate directly through a Gen5 switch without routing P2P traffic through the host, that should matter much more for tensor-parallel inference than having an expensive CPU.
The important thing is to benchmark the actual topology rather than assume x16 per GPU will automatically translate into better vLLM performance. Check GPU-to-GPU bandwidth/latency, P2P support, NUMA effects, and how the workload scales from 4 → 8 GPUs. Also make sure the motherboard and BIOS expose the required PCIe lanes and ACS/IOMMU settings correctly.
For ~200 GB models, memory capacity and inter-GPU communication can easily become the bottleneck, so I’d validate the switch configuration with NCCL P2P tests before spending more on the host. A cheap CPU is reasonable if it doesn’t become the bottleneck for tokenization, scheduling, or data movement.
When you scale past 4 GPUs your 2 biggest problems become power and cooling.
Get a case with power distribution board designed for this specific deployment scenario like The Rig 3.0 or you are going to spend weeks cooking things, chasing signal quality problems that are actually grounding issues, etc.
Depending on what your software is doing, that dictates the hardware design. With tp8 blackwells on large moe models, the dual turin 160 lane approach actually beat 2x cpayne switches in pure performance. Come to the discord and hang around the hardware channel to see all the unhinged builds people have made.
Also RDNA4 specifics here, last I checked, RCCL straight up doesn’t work through host PCIe so you absolutely need a switch just to init. Also last I checked, TP=8 on RDNA4 also did not work, but I’m not sure how far Wendell made it on that particular issue at this point. I’m only running four of them on my pleb tier AI rig so I can’t really weigh in.
Some good news, I got a response from Utran offering to sell their 8 card pcie 5.0 16x switch backplane P5-SW144-GPU-I-R12 for $3000. Might be a good option and would save on MICO adapters.
I’m leaning towards that replacing the backplane in a MM-A515-CPW. Utran might have a better solution for an enclosure already though so I’m waiting to hear back.
@Jordan.wells Agreed, thats the goal for sure. Thanks for the heads up on checking to make sure the motherboard exposes ACS/IOMMU settings. Also the idea of testing switch config with NCCL P2P tests on a cheaper host sounds look a good idea, I’ll probably plan an initial cheap host and upgrade if testing reveals the bottleneck.
@thr3e Great point on grounding, I used to build electrical enclosures for manufacturing equipment and have been bitten by that in the past. If I do need to go custom route, I’ll make sure gounding planes are connected right (like housing paint stripped below metal standoffs for board screws). Hoping I won’t have to go too crazy with needing a ferrules or copper tape for ghetto faraday caging. I joined the discord too, will do some searches there to find similar builds.
I use lmcache to offload to ram. I dont like thrashing NVME because its expensive and wears out. Buy if you need TB’s of cache there is really no way to avoid that I guess. in nvidia space lmcache (or mooncake) offloading to nvme with nixl can be externalized to external storage if you have enough mellanox rdma bandwidth to swing around.
Ah right, didn’t realize the read/write ops of a cache would wear out a nvme drive quickly.
Then sounds like for caching on a budget I’d be looking at a sp3 mobo + rome cpu and 256gb of ram to support ~8 concurrent sessions hot cached, then some nvme drives as cold cache.
Also I’m considering a utran B8144M or B8144Q. I missed the fact that Utran just sells exactly what I need, if the price makes sense I’ll probably bite. Otherwise modifying a old miner case like a octominer 8x or 12x would be where I’m looking.
Its been a while since the last update but I’ve been between researching this project quite a lot. Even did long research sessions comparing options like a gaudi 2 server, 2x m5 ultra 256gb, a old dgx 8x a100 40gb server, and 8x Tenstorent blackhole p150. I still think r9700 represents the best value, primarily for now and future community software support.
I got some quotes back, and its not looking good for a 8 card pcie 5.0 server. Most options 1.5-2x in price this year compared to last on ebay and aliexpress, the switch board alone runs $4500 from Utran and Maicun. Seems like the cheapest barebones server option with a switch I found was ~5.5k on ebay, and then the host needs expensive ddr5 ram and newer xenon chips adding an extra 5.5k.
Even exploring 8 card pcie 4.0 16x or 4.0 8x switch options were expensive, totaling around ~$3k best case for either option.
So now I’m convinced the best value would be a used z20, but it presents many challenges:
Each gpu bay for 2 cards has only 2x cpu style 8 pin power connector rated at 300w max each. Each r9700 has a 12v-2x6 connector and a 300w draw. I’ll need 8 sets of adapters like this.
Pcie switch leafs per gpu pair, seemingly using host pcie and the cpu as the joining switch. Latency and configuration could get tricky. The RCCL not working on the host PCIe issue @Strawberry raised worries me here too.
Airflow may be questionable depending on spacing in the 2 card bay, Might need to do some gasketing or fan upgrades if thermals get spicy during summer.
But all things considered, this z20 approach will allow me to have a true 8 card pcie 4.0 16x host for a total of ~$3000 including cpu, 256gb ram, and 4tb m.2 storage. Think I’ll go for that and crush my dreams of a pcie 5.0 host. Also because of the wind tunnel design, this sounds even more like a jet engine than a normal 4u gpu server, so later sound dampening where its installed will be a need.
Also because of the wind tunnel design, this sounds even more like a jet engine than a normal 4u gpu server, so later sound dampening where its installed will be a need.