Followed the documentation on primitive-ai/Qwen3.8-Flash-Next-NVFP4 · Hugging Face Here is my full report Claude Artifact . Basic 81- 88 tokens per second on a single session. Scales to a throughput of 341 tokens per second when running 6 concurrent client. Interesting piece is the prompt preprocessing is GPU compute bound at 12000 tokens per second even when context is scaled up to 100k. This model rules but you need to have 100GB spare system memory to hold the 51B n-gram table.
This is without MTP activated. I am working on benchmarking with MTP currently.
Finished running a setup with MTP active on a single card. I had to make some sacrifices, max context is 200k down from 268k and max kv cache is 300k down from 404k but in return it boosts token performance around 68%. Here is the write up on the same system. Claude Artifact I ended up having a dead end trying to get KV Cache to offload onto system memory because I have room to spare but its not supported by Vllm. Llama.cpp has a PR in the works for this.
With MTP - 120-173 tok/s in real workloads.
Prefill took a hit to performance by about 11-12% now only prefilling at 11,000 tok/s
