Got deepseek-v4.1-flash running here through 0xSero/deepseek-v4.1-flash-4x-rtx-pro-6000 sglang’s runtime. Very impressed, here are the aggregate stats for my real traffic (same workload: cc, codex, agent evals) vs previous deployment of deepseek-v4-flash-0731:
| Metric | DSV4-FLASH-0731 | DSV4.1-FLASH |
|---|---|---|
| Duration | ||
| Wall-clock observation span | 531.70 h | 47.55 h |
| Sum of request-activity windows | 511.73 h | 38.43 h |
| Observed launches | 16 | 8 (1 running) |
| Completed non-health generations | 256,325 | 10,871 |
| Health-check generations | 6,303 | 2,661 |
| POST /v1/messages | 175,668 | 8,651 |
| POST /v1/chat/completions | 62,436 | 1,768 |
| POST /v1/responses | 18,662 | 550 |
| POST /v1/messages/count_tokens | 2,356 | 86 |
| Prompt size, tokens | ||
| Average full prompt | 111,499 | 113,199 |
| Median prompt | 102,047 | 108,197 |
| p95 prompt | 264,293 | 251,338 |
| Largest prompt | 473,107 | 524,177 |
| Average cached prompt tokens | 107,688 | 89,025 |
| Average uncached prompt tokens | 3,811 | 24,174 |
| Latency, seconds — p50 / p95 / p99 | ||
| Queue / admission | 0.64 / 2.08 / 27.14 | 0.84 / 78.98 / 204.46 |
| TTFT, engine-side | 1.21 / 7.09 / 42.00 | 1.38 / 107.15 / 232.27 |
| Prefill | 0.27 / 2.27 / 18.09 | 0.30 / 38.05 / 48.43 |
| Decode | 5.04 / 49.89 / 152.32 | 3.23 / 86.35 / 251.02 |
| E2E, engine-side | 6.47 / 58.60 / 161.35 | 5.37 / 186.02 / 335.03 |
| Throughput, average | ||
| Per-request decode, tok/s | 48.8 | 102.0 |
| Batch generation, tok/s | 137.4 | 191.9 |
| Prefill input, tok/s | 2,444 | 5,955 |
| DSPARK(5) acceptance length / rate | N/A — not enabled | 3.71 / 0.542 |
| Tokens | ||
| Input tokens | 28,579,867,197 | 1,230,587,082 |
| Output tokens | 148,540,907 | 8,652,596 |
| Of which reported reasoning | 87,042,866 | 5,690,092 |
| Average output per generation | 580 | 796 |
| Cache & health | ||
| Token-weighted cache hit | 96.58% | 78.64% |
| Zero-hit generations | 11,578 (4.52%) | 2,127 (19.57%) |
| HTTP 200 / 503 / 400, all routes | 303,043 / 40 / 37 | 21,438 / 26 / 0 |
| Other HTTP statuses | 404: 54, 500: 10 | 0 |
| HTTP 2xx rate, all routes | 99.953% | 99.879% |
| HTTP 2xx rate, inference routes only | 99.982% | 100.000% |
| Allocator OOM-warning log lines | 0 | 1,470 |
OxSero’s DSV4.1-flash recipe is mostly as is, with exception to engram OFFLOAD_MODE=ram and sglang’s RAM HiCache enabled at 256GB (total offloaded at 468.8GiB) and swa caching patches per Retain reusable SWA request windows in RAM HiCache · ambientlight/sglang@28cedd2 · GitHub, old dsv4-flash-0731 per https://hub.docker.com/r/ambientlight/sglang-sm120-mxfp4?tag=2026.08.0-cu130-sm120a, launch params diff per:
| Metric | DSV4-FLASH-0731 | DSV4.1-FLASH |
|---|---|---|
| Docker container | dsv4 |
dsv41 |
| Served model name | deepseek-v4-flash |
deepseek-v4-flash |
| Tensor / expert parallelism | TP4 / EP1 | TP4 / EP4 |
| Context limit, tokens | 1,048,576 | 524,288 |
| Static memory fraction | 0.85 | 0.87 |
| Maximum running requests | 16 | 8 |
| Chunked prefill size, tokens | 4,096 | 2,048 |
Maximum prefill tokens (--max-prefill-tokens) |
16,384 | 16,384 |
| Prefill/decode interval | 4 | 4 |
KV capacity sizing (--max-total-tokens) |
Automatic; unset | Automatic; unset |
| Observed main KV capacity, logical tokens⁴ | 5,677,312 | 3,367,936 |
| Observed GPU SWA pool, slots per rank⁴ | 567,552 | 26,112 |
| KV cache dtype | fp8_e4m3 |
fp8_e4m3 |
| KV page size, tokens | 256 | 256 |
| Prefix-cache eviction policy | LRU | LRU + SWA boundary retention |
| RAM HiCache budget | Disabled | 256 GB host-total; FULL + SWA |
| HiCache write / I/O / layout | N/A | write_through / kernel / page_first |
| NVMe KV-cache tier | Disabled | Disabled |
| Engram offload mode | N/A | ram; separate NVMe row cache disabled |
| Speculative decoding | Disabled | DSPARK, block size 5 |
| CUDA graph batch-size limit | 16 | 8 (decode) |
| CUDA expandable segments | Enabled | Disabled |
| Attention backend | dsv4 |
dsv4 |
| MoE runner backend | flashinfer_mxfp4 |
flashinfer_mxfp4 |
| Reasoning / tool-call parser | deepseek-v4 / deepseekv4 |
deepseek-v41 / deepseekv41 |
| Default temperature / top-p | 1.0 / 0.95 | 1.0 / 0.95 |