TL;DR: Pricing out small on-prem LLM deployments for office clients (vLLM/continuous batching, ~20 tok/s/user target). My bandwidth-bound math says two of these options wildly outperform a DGX Spark for the same money, but I know the calc is idealized and real serving software doesn’t always hit it. Two things I specifically don’t trust: (1) dual-GPU PCIe scaling, (2) whether Mac Studio’s continuous-batching software is anywhere near mature enough to hit what the raw bandwidth numbers promise. Hoping someone’s actually run these and can tell me where I’m wrong.
What I’m trying to do
Sizing hardware for small-office multi-user assistants (10-30 concurrent people, everyday office work + some coding). Budget lands around $8-9k USD equivalent. Options on the table:
- 1x NVIDIA DGX Spark (128GB unified memory, 273GB/s bandwidth)
- 2x AMD Radeon PRO R9700 (32GB GDDR6 each, 640GB/s each, ~$1,300/card), via vLLM --pipeline-parallel-size 2 (no NVLink between them)
- 1x Mac Studio, M3 Ultra, 256GB unified memory, 819GB/s bandwidth (~$6k config)
The math I’m running
aggregate_tok/s = bandwidth / (active_params_bytes * bytes_per_param)
concurrency = aggregate_tok/s / target_tok/s_per_user
For a ~35B MoE model (3B active, ~20GB weights, 48 layers, 8 KV heads, 128 head dim):
Spark: 273e9 / (3e9*0.57) ≈ 159 tok/s → ~7 concurrent users
Dual R9700*: 1280e9 / (3e9*0.57) ≈ 746 tok/s → ~37 concurrent users
Mac Studio: 819e9 / (3e9*0.57) ≈ 478 tok/s → ~23 concurrent users
The dual R9700 number assumes a naive linear combination of the two cards’ bandwidth. I know pipeline parallelism doesn’t give clean linear scaling in practice. There’s supposed to be pipeline-bubble overhead and vLLM’s own docs recommend pipeline parallelism over tensor parallelism specifically when there’s no NVLink, for exactly this multi-user-throughput use case. But I can’t find a real benchmark for this pairing, especially on AMD/ROCm rather than NVIDIA.
The number that really doesn’t sit right with me is the Mac Studio one at bigger model sizes. Running the same formula against a ~120B-class model (5.1B active params, ~65GB weights) on the Mac Studio’s 819GB/s and 256GB memory gives:
aggregate = 819e9 / (5.1e9*0.54) ≈ 296 tok/s → ~14 concurrent users
14 concurrent users running a genuinely large (120B-class) model, on a single desktop box that isn’t even primarily marketed as an AI server? That’s the hardest number to believe out of this whole exercise.
Why I’m suspicious specifically of the Mac number
The formula assumes vLLM-grade continuous batching, i.e. that the serving software can actually keep the bandwidth saturated across many concurrent users. That’s true and battle-tested for vLLM on CUDA. On Apple Silicon I can’t find anything that assures clarity: the most advanced continuous-batching project I could find for MLX reports something like 4.3x aggregate throughput at 16 concurrent requests, nowhere near the ~16x linear scaling my formula assumes. If that gap is representative, the real Mac Studio number could be way below 14, not just “a bit lower.”
What I actually want to know
-
Has anyone run vLLM (or llama.cpp) with
--pipeline-parallel-size 2across two PCIe-only GPUs for concurrent multi-user serving (not single-stream)? What fraction of the naive linear-sum throughput did you actually get? -
Specifically for AMD/ROCm multi-GPU, is the R9700 (very new card) actually stable for this yet, or is ROCm multi-GPU still rough around the edges compared to CUDA?
-
Has anyone run mlx-lm, llama.cpp (Metal,
--cont-batching), vLLM-metal, or vllm-mlx on an M3 Ultra Mac Studio under genuine multi-user concurrent load (not single-stream benchmarks)? What tok/s per user did you actually get at, say, 10+ simultaneous requests? -
Does anyone have real numbers running a 100B+ class model specifically on Mac (not just “it loads and runs” single-stream demos, which I’ve seen plenty of)?
-
Is continuous batching on Mac (any backend) actually production-viable today for a multi-user office deployment, or is this still “works great for one person, falls over past a handful of concurrent users” territory?
-
Is my ~20 tok/s/user target reasonable for “feels responsive,” or am I over/under-shooting what real users actually need, in a way that would change any of this?
Happy to share more of my numbers/assumptions if useful, trying to get this right before I put real money behind a recommendation to clients.
Also very grateful that you took time out of your day to help me with my problem.