Qwen3.8-Flash-Next on 2× NVIDIA CMP 170HX — llama.cpp GGUF vs. vLLM INT4 W4A16
I did a direct before/after benchmark of Qwen3.8-Flash-Next on my dual NVIDIA CMP 170HX setup.
The interesting result is not decode performance — which is basically unchanged — but prompt processing / prefill performance. Moving from the early llama.cpp GGUF implementation to the new Ampere-targeted vLLM W4A16 build improved long-context prefill by up to 10.3× at 262k context.
Hardware
-
2× NVIDIA CMP 170HX
-
GPU architecture: GA100 / Ampere
-
64 GiB VRAM per GPU
-
Total physical VRAM: 128 GiB
-
PCIe: Gen2 x8 per GPU
-
GPU P2P: working / OK
-
System RAM: 131 GB
-
NVMe cache/storage available
-
Linux
BEFORE — llama.cpp / GGUF
Model
unsloth/Qwen3.8-Flash-Next-GGUF
Quant:
UD-IQ4_XS
Model size:
93.7 GB, 3 GGUF shards
Hugging Face:
https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main/UD-IQ4_XS
Backend:
llama.cpp / CUDA
Architecture name inside llama.cpp:
qwen4exp
Qwen3.8-Flash-Next support came from:
llama.cpp PR #27742 — “model: add Qwen3.8-Flash-Next (qwen4exp)”
Merged commit:
6c84c7d5d
llama.cpp:
https://github.com/ggml-org/llama.cpp
This GGUF is confirmed by the Unsloth repository as UD-IQ4_XS, 93.7 GB. The qwen4exp support was added to llama.cpp through PR #27742.
AFTER — vLLM / native INT4 W4A16
Model
VnimanieAI/Qwen3.8-Flash-Next-W4A16
Hugging Face:
https://huggingface.co/VnimanieAI/Qwen3.8-Flash-Next-W4A16
Quantization:
INT4 W4A16
More precisely:
-
compressed-tensors
-
INT4 weights
-
FP16/BF16 activations
-
symmetric
-
group size 128
-
Marlin kernels
-
Routed MoE experts: INT4
-
QSA attention projections: INT4
-
Shared experts: BF16
-
MTP head: BF16
-
PLE tables: BF16 / CPU offloaded
-
MoE router: BF16
-
QSA indexer: BF16
-
GDN / linear attention: BF16
The model authors specifically built this quantization for Ampere GPUs that cannot run the official FP8 checkpoint.
Backend:
vLLM qwen4_exp development build
Docker image used:
vllm/vllm-openai:qwen38-flash-next-x86_64-cu130
Configuration:
-
TP2
-
Expert Parallelism enabled
-
PLE CPU offload enabled
-
VLLM_PLE_CPU_OFFLOAD=1
-
MTP speculative decoding
-
MTP depth = 4
-
Full native 262,144 token context
-
torch.compile / Inductor disabled
-
decode-only CUDA graphs
Relevant launch options:
--tensor-parallel-size 2
--enable-expert-parallel
--max-model-len 262144
--speculative-config '{"method":"mtp","num_speculative_tokens":4}'
--compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY"}'
The repository explicitly recommends Expert Parallelism, PLE CPU offload, MTP4 and decode-only CUDA graphs for this model family.
Before vs. After — Prompt Processing / Prefill
These measurements were performed cache-free, and the token counts came from the actual API tokenizer rather than estimated text length.
| Context |
llama.cpp UD-IQ4_XS |
vLLM W4A16 |
Improvement |
| 8k |
727 tok/s |
2,476 tok/s |
3.41× |
| 16k |
702 tok/s |
2,536 tok/s |
3.61× |
| 32k |
618 tok/s |
2,545 tok/s |
4.12× |
| 65k |
499 tok/s |
2,510 tok/s |
5.03× |
| 98k |
413 tok/s |
2,475 tok/s |
5.99× |
| 131k |
353 tok/s |
2,443 tok/s |
6.92× |
| 196k |
273 tok/s |
2,336 tok/s |
8.56× |
| 262k |
222 tok/s |
2,295 tok/s |
10.34× |
The most striking result is the native 262k context:
222 tok/s → 2,295 tok/s
That is a 10.34× increase in prompt-processing throughput.
It also scales much better with context length.
The old llama.cpp run dropped from:
727 → 222 tok/s
between 8k and 262k, a reduction of roughly 69.5%.
The new vLLM W4A16 stack only drops from:
2,476 → 2,295 tok/s
over the same range, or roughly 7.3%.
Peak measured prefill was:
2,545 tok/s @ 32k
while full native 262k still manages:
2,295 tok/s
MTP4 Decode — 256 Output Tokens
llama.cpp / previous run
| Run |
Decode |
| Run 1 |
90.6 tok/s |
| Run 2 |
132.2 tok/s |
| Run 3 |
108.5 tok/s |
| Average |
110.4 tok/s |
vLLM W4A16 / current run
| Run |
Decode |
| Run 1 |
112.7 tok/s |
| Run 2 |
111.8 tok/s |
| Run 3 |
109.6 tok/s |
| Average |
111.3 tok/s |
Decode comparison
110.4 → 111.3 tok/s
Only about:
+0.8%
So decode performance is essentially unchanged.
That is actually useful information, because it makes the prefill result much more interesting: this does not look like a general 3–10× GPU performance change.
The huge improvement is specifically in the prompt-processing / long-context execution path.
Summary
| Metric |
Before |
After |
| Model |
Unsloth GGUF UD-IQ4_XS |
VnimanieAI W4A16 |
| Backend |
llama.cpp |
vLLM |
| Quantization |
UD-IQ4_XS GGUF |
INT4 W4A16 G128 |
| Parallelism |
dual-GPU llama.cpp |
TP2 + EP |
| PLE |
normal GGUF execution |
CPU offload |
| Speculative decoding |
MTP |
MTP4 |
| 8k Prefill |
727 tok/s |
2,476 tok/s |
| 32k Prefill |
618 tok/s |
2,545 tok/s |
| 131k Prefill |
353 tok/s |
2,443 tok/s |
| 262k Prefill |
222 tok/s |
2,295 tok/s |
| 256-token Decode |
110.4 tok/s avg. |
111.3 tok/s avg. |
| 262k Prefill Gain |
— |
10.34× |
The current VnimanieAI INT4 W4A16 + vLLM TP2/EP + PLE-offload + MTP4 stack is therefore an easy winner on these two CMP 170HX cards.
Decode remains around 111 tok/s, but long-context prefill has gone from being the biggest weakness of the system to one of its strongest points.
A 2× CMP 170HX GA100 setup doing ~2,300 tok/s prompt processing at the full native 262k context while still decoding at ~111 tok/s is far better than I expected from these cards.
The complete cache-free benchmark sweep took:
9 minutes 23 seconds.
I was considering a similar thing… cmp170hx with the unlocked memory… I’m wondering how it does with the nerfed pcie bus in multi-card configurations. That’s -very- tempting knowing that one of those cards is nearly on par with two 9700s, but can they scale higher? I wonder if anyone has tested that specifically.
so they are best at fixing them) and take it apart, add the caps, and then get a water cooling kit.
I let Codex made it unlock under Unraid.
