Got the full 355B GLM-4.7 (not Flash) serving in NVFP4 across two DGX Spark GB10s at
▎ 64K context, ~17.5 tok/s on vLLM (TP=2). Four walls to get there: a Ray session
▎ collision (–no-ray), a weight_loader arity bug (one-line fix, now a PR upstream),
▎ the KV-memory squeeze (batch limits + gpu-memory-utilization 0.90, no SSD offload
▎ needed — it fits in unified memory), and it being a reasoning model.
▎
▎ - Full technical recipe (NVIDIA DGX Spark forum):
▎ https://forums.developer.nvidia.com/t/full-glm-4-7-355b-nvfp4-at-64k-context-on-2x-d
▎ gx-spark-gb10-working-recipe-vllm-tp-2/375690
▎ - The story / write-up:
▎ vLLM is a Beast on Dual Sparks - Full 355B GLM-4.7 Proves It
▎ - Upstream fix (PR): Add GLM-4.7-NVFP4 KVCacheScaleParameter arity fix + full-GLM 64K example by vintageaiswarm-ray · Pull Request #307 · eugr/spark-vllm-docker · GitHub
▎
▎ Happy to answer questions on the dual-Spark setup.
1 Like