You might remember our Z8 Fury G6i from our other videos and setup guides – be sure to check out our full review:
We’re back again and now we’vet got:
Quick how-to on setting up Qwen 3.8 Flash Next on this system but with 1M context! (1.5M aggregate context) and over 200t/s!
Be sure to check out our full review and other content featuring the HP Z8 Fury G6i
Setup Summary
Host: 2x RTX PRO 6000 Blackwell (96 GB), 125 GB RAM, 48 cores Intel Xeon X658X
Model: Qwen/Qwen3.8-Flash-Next-FP8 (125B MoE / 6B active, 51B n-gram table, 4B MTP)
Engine: vLLM vllm/vllm-openai:qwen38-flash-next (0.1.dev20073)
Status: SERVING on port 8000, model qwen3.8-flash-next, 1M context, MTP enabled
Architecture decisions
| Component | Choice | Why |
|---|---|---|
| Checkpoint | FP8 (187 GB) | BF16 is 335 GB won’t fit 2x96GB |
| n-gram table | VLLM_PLE_CPU_OFFLOAD=1 |
51B params moved to host RAM; GPU holds only ~67 GiB weights |
| MTP | num_speculative_tokens: 3 |
2x decode speedup; acceptance 98-100% |
| Context | 1M via YaRN | Fits at 0.95 gpu-mem-util (95.5 GiB/GPU)* |
| TP | 2 | 2 GPUs |
Update: More YaRN Testing
I ran the count task under three rope configs at the same contexts:
| Config | 128K | 240K | 500K |
|---|---|---|---|
| Native 262K | 50 | 42 | — |
| YaRN f2 (524K) | — | 30 | 30 |
| YaRN f4 (1M) | 30 | 30 | 25 |
So I built a test harness that operates on a large python codebase to test to 900k context. The results are a bit mixed imho.
There shouldn’t be any prose-periodicity exploit, which is why I thought maybe this common test was benchmaxxed (turns out it seems so). So 5× repeats at true context depth:
| Context | codefind | codeorder | codecount (of 25) |
|---|---|---|---|
| 128K | 5/5 | 5/5 | 10 |
| 240K | 5/5 | 5/5 | 10 |
| 500K | 5/5 | 5/5 | 7 |
| 750K | 5/5 | 5/5 | 7 |
| 900K | 5/5 | 5/5 | 6 |
Finding a specific function (codefind) and listing functions in file order (codeorder) are bulletproof to 900K – I wasn’t expecting that given the same sort of test with prose/text was not quite as good. But this is exactly the operations you’d want for “navigate a huge codebase.”
However, counting definitions still degrades (10 vs 6), which shows the same aggregation weakness we see in testing against prose/text (instead of code).
So, with YaRN, No degradation to 900K needle, KV, multihop, varchange, codefind, codeorder all 100%.
However, aggregation/counting: degrades with context and but the slope depends heavily on YaRN. Under factor=4 it’s 30 to 15 (prose) / 10 to 6 (code); under native it starts at 50. So the model’s native aggregation at 128K to 256K is much better than the YaRN-scaling number suggests.
Net: if you’re using this for programming at large contexts, factor=2 up to ~500K is the sweet spot because you’ll keep near-native behavior for the tasks that matter (find/track) and lose the YaRN counting penalty, but side effects of going to 1M isn’t that bad.
You do get some degredation that’s undesirable moving beyond the native context window.
This is an interesting finding and means, I think, running this model @ 500k with a YaRN scale factor of 2 is more indicated for accuracy.
MTP acceptance: 98-100%, mean acceptance length ~4.0 (near-perfect drafting).
I also ran some needle/haystack and variable recall benchmarks:
Bottom Line: If you’re going to run YaRN run scale_factor=2 for coding tasks.
Notes
- Host is currently (595.71.05 / CUDA 13.2, Docker 29.6.1).
- 1M context is the VRAM ceiling: 95.5/96 GiB per GPU. No real headroom for larger.
- Long-context verified at 55K tokens (2.8s prefill, correct answer).
Here’s my startup script (I’m a little surprised this was so easy? and reachable on this system?)
#!/bin/bash
# Launch Qwen3.8-Flash-Next-FP8 on bighp (2x RTX PRO 6000 Blackwell 96GB)
# Usage: ./launch_qwen38.sh [baseline|1m] [mtp|nomtp]
set -euo pipefail
MODE="${1:-baseline}" # baseline (262K) or 1m
MTP="${2:-mtp}" # mtp or nomtp
NAME="vllm-qwen38-fpn"
MODEL_DIR="/home/w/models/Qwen3.8-Flash-Next-FP8"
PORT=8000
# Common args
ARGS=(
--model /models/Qwen3.8-Flash-Next-FP8
--tensor-parallel-size 2
--gpu-memory-utilization 0.90
--max-num-seqs 256
--enable-prefix-caching
--no-enable-flashinfer-autotune
--enable-auto-tool-choice
--tool-call-parser qwen3_xml
--reasoning-parser qwen3
--served-model-name qwen3.8-flash-next
--port "$PORT"
)
if [ "$MODE" = "1m" ]; then
ARGS+=(
--max-model-len 1000000
--gpu-memory-utilization 0.95
--hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'
)
else
ARGS+=(--max-model-len 262144)
fi
if [ "$MTP" = "mtp" ]; then
ARGS+=(
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
)
fi
echo "=== Launching vLLM: mode=$MODE mtp=$MTP ==="
echo "docker run -d --name $NAME ... ${ARGS[*]}"
# Remove old container if present
docker rm -f "$NAME" >/dev/null 2>&1 || true
docker run -d --name "$NAME" --restart unless-stopped \
--ipc=host --gpus all \
--shm-size 16g \
-e VLLM_PLE_CPU_OFFLOAD=1 \
-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
-v "$MODEL_DIR":/models/Qwen3.8-Flash-Next-FP8:ro \
-p "$PORT":8000 \
vllm/vllm-openai:qwen38-flash-next \
"${ARGS[@]}"
echo "Container $NAME started. Follow logs: docker logs -f $NAME"