I built Crucible LLM — a terminal-based benchmarking suite that stress-tests any OpenAI-compatible inference server and measures what a real user actually experiences. One static Rust binary. No Python. No venv. No dependencies.
Why it exists
Most benchmarks shout “xx.x tokens/sec!” and stop. But when 32 users hit that server at once, each one gets 2 tokens/sec. A number that looks fast in the aggregate can feel painfully slow per user. Crucible measures that gap — the difference between a server’s peak throughput and what each individual connection actually gets — as well as how competent the LLM is at real tasks.
What it does
- One static binary. A single ~15 MB Rust executable. No Python, no venv, no system libraries — SQLite and GPU telemetry are compiled in. Point it at any
/v1endpoint and go. - TUI-first, 60 Hz dashboard. Six keyboard-driven views (Live Monitor, Concurrency, NIAH, GPU & Power, History, Config) with real-time graphs and per-engine metrics. Headless/CLI mode for CI and scripting.
- Seven benchmark engines. Speed → Concurrency → Long-context (NIAH) → Reasoning → Structured output → Energy → Flat Out, each individually selectable.
- The “practical sweet spot.” The concurrency sweep finds the highest user count where each person still gets ≥ 30 t/s — the comfortable hosting number, kept distinct from the raw peak.
- Reasoning-model aware. Separates the chain-of-thought from the answer, judges on the answer, and reports over-thinkers honestly (
truncated_by_reasoning) instead of failing them. - Deeper-context NIAH. Its own needle-in-a-haystack matrix, runtime-configurable and probing RoPE/YaRN past 128k — and now settable right in the TUI, not just the CLI.
- Multi-vendor GPU monitoring. NVIDIA, AMD, and Intel, with live power / utilization / VRAM / temperature and energy-per-token (Linux).
- Accurate energy. Joules/token measured over the full run — full load window, idle baseline subtracted, real token denominator — with an honest
N/Ainstead of a silent 0. - Flat Out = true max throughput. A 60-second run at the concurrency peak — the headline number to quote when comparing setups — distinct from the practical sweet spot.
- The Matrix Report. One key (
m) or one flag (--matrix-report) produces a ready-to-paste[COMMUNITY-SUBMISSION]block for the Local LLM Matrix. - Run-log access. Press
l(or--show-logheadless) to view, copy, or save the full run log, now with a[CONFIG]block, per-engine wall-times, and a shareable run-end[DIGEST]. - Any OpenAI-compatible server. vLLM, llama.cpp, LM Studio, Unsloth, SGLang, Ollama, TGI.
- Three visual themes. Cyberpunk, Vampire, Monochrome Pastel.
- Headless/CI-ready. Pure JSON, meaningful exit codes, export to JSON / Markdown / CSV.
See it in action
The Live Monitor: a real-time throughput graph with an auto-scaling y-axis, engine-transition markers, key metrics, and a live event log.
Try it
Grab the latest pre-compiled binary from the Releases page — Linux, macOS, and Windows are all built:
chmod +x crucible-llm-v0.1.9-linux-x86_64
./crucible-llm-v0.1.9-linux-x86_64
Or build from source (stable Rust toolchain):
git clone https://github.com/MadGoatHaz/crucible-llm
cd crucible-llm
cargo build --release
./target/release/crucible-llm
On first run, the TUI asks you to pick a theme, then walks you through entering your server URL and picking a model.
The v0.1.5 → v0.1.9 arc
Last month I wrote about making reasoning honest — separating the chain-of-thought from the answer, judging on the answer, and reporting over-thinkers without failing them. That was the foundation. The four releases since have been about the rest of the numbers: making everything else honest too, and making the results easy to share. Every number is now accounted for.
v0.1.6 — Energy, done right. The energy figures were lying to me. The J/token number was roughly 25× too high, and when I dug in there were three compounding reasons: a phantom GPU power reading that slipped past the sanity check, a denominator that only counted a fraction of the real tokens, and a 6-minute sample window on a 19-minute run (plus an estimated CPU power floor that was never labeled as an estimate). All of it is fixed now. The energy integral runs over the full load window, the denominator counts the real prompt + completion tokens, a generic power-sanity clamp catches phantom readings, and the idle baseline is subtracted so you see the load-attributable energy. Flat Out also finally runs at the true max-throughput point — the peak of the concurrency sweep — which is a different animal from the “practical sweet spot” (the comfortable per-user hosting number), and the two are no longer conflated. And the NIAH ladders are now settable in the TUI, not just on the command line.
v0.1.7 — Sharing made easy. The headline addition for the community is the Matrix Report. After a run, press m in the TUI (or --matrix-report headless) and you get a ready-to-paste [COMMUNITY-SUBMISSION] block for the Local LLM Matrix. Crucible auto-fills the things it can measure — hardware, VRAM tier, model, and speed — and you fill in the things only you know — backend, quant, use case, and notes. Copy it to your clipboard (OSC 52) or save it as a .md, and it’s on the Matrix. This release also folds in the corrections from 0.1.6: I’d been hardcoding a per-GPU TDP table so the power clamp knew, say, “an A4000 is 200 W” — the kind of assumption a general benchmarking tool shouldn’t make — so I removed it. The clamp is now generic: ≤1.5× the card’s queried TDP when it’s available, and when the query fails the reading stands but is flagged unvalidated — a label of uncertainty, not a correction. The energy denominator was completed to include all the concurrency legs, and the efficiency panel reports an honest N/A when no tokens were generated, never a silent 0. And this is the release where I started shipping an actual downloadable binary, with a CI workflow that builds Linux, macOS, and Windows on every release — so you no longer have to compile it yourself.
v0.1.8 — The run log, made easy to grab. I made the full run log easy to view and share. Press l in the TUI (any time, no completion gate) to open a full-screen, scrollable log viewer — c copies it to the clipboard, s saves it to a file, Esc closes — and --show-log headless prints the whole log to stdout. The log is now self-describing, too: a [CONFIG] block at the top records the resolved inputs (url, model, mode, tokens, iterations, the concurrency and NIAH ladders, engines, hardware), each ✓ engine headline carries its wall-clock duration, and a compact run-end [DIGEST] block is the thing to paste when you share a run. The Matrix Report also got sharper here: a multi-GPU pool’s hardware string is now unambiguous (8x NVIDIA RTX A4000 (16 GB each, 128 GB total) + 118 GB RAM — per-card and pool total), and for pure-reasoning (“CoT-only”) models the speed falls back to an honest blended decode rate with an explanatory notes line — never a fabricated answer-phase number.
v0.1.9 — Every number accounted for. This one is a measurement-integrity pass. The theme: a number means what it says. The biggest fix is no more phantom rates. A sub-10 ms answer window (say, a 2-token answer delivered in one batch) used to divide by a clamped 1 ms sliver and mint a fake ~2000 t/s — the old “C2 avg 799 t/s on a 27B model.” Now an unmeasurable window reads n/a and is excluded from the averages, instead of a fabricated number. Flat Out’s token counts use one method so the summary and the per-stream lines reconcile; stream complete lines are self-consistent ({n} tokens in {total}s (decode {rate} t/s; TTFB {ttfb}s)); there’s exactly one started line per engine and no duplicate complete lines; the [DIGEST] is a genuinely shareable block (version, config, wall-times, failures, totals, hardware); and the [HW] line is unambiguous for a multi-GPU pool (8× NVIDIA RTX A4000, plus the vLLM version and driver/CUDA). Honest by design: an unmeasurable window is n/a, never a made-up number.
The through-line across all four: I’d rather report an honest n/a than a confident wrong number. That’s the whole point of a benchmark.
Thanks
A big thank-you to @harshithmohan and @strawberry
Feedback — your help is the real test
This is where I need you. Two things would be great:
- Try the Matrix Report. Do a run, press
m, fill in your backend / quant / use case, and post your rig to the Local LLM Matrix. The more honest, consistently-measured entries there are, the more useful it gets — and if anything in the auto-filled block looks off for your setup, tell me. I’d rather fix a quirk than ship a bad number. - Sanity-check the numbers you care about. Run v0.1.9 against a model you know, and check the energy / J-per-token and Flat Out figures against what you’d expect from your hardware. Those are the two I rewrote, and a second pair of eyes on real rigs is exactly what I want.
If a number looks wrong, post it — the [DIGEST] block is the quickest thing to paste. I read every one.
Changelog
| Version | Date | Highlights |
|---|---|---|
| 0.1.9 | 2026-10-20 | Measurement-integrity pass: no phantom rates (sub-10 ms → n/a, excluded from averages); Flat Out token counts reconcile; self-consistent stream complete lines; one started / no duplicate complete per engine; richer shareable [DIGEST]; unambiguous multi-GPU [HW] (+ vLLM / driver / CUDA) |
| 0.1.8 | 2026-10-19 | Run-log access (l TUI / --show-log headless); run-end [DIGEST] block; [CONFIG] block at start; per-engine wall-times; unambiguous multi-GPU hardware string; honest blended rate for pure-CoT models |
| 0.1.7 | 2026-10-18 | Matrix Report (m / --matrix-report) → [COMMUNITY-SUBMISSION] for the Local LLM Matrix; no card-specific TDP (generic clamp + unvalidated flag); energy denominator completes all concurrency legs; honest N/A (never a silent 0); CI ships Linux / macOS / Windows binaries |
| 0.1.6 | 2026-10-18 | Energy J/token ~25× fix (full load window, real prompt + completion denominator, idle-baseline subtraction, generic power clamp); Flat Out = true max-throughput (distinct from the practical sweet spot); NIAH ladders settable in the TUI |
| 0.1.5 | 2026-10-12 | Hybrid reasoning (budget = correctness, disable_thinking best-effort, truncated_by_reasoning); runtime-configurable NIAH (2k → 1M); SSE keep-alive routing fix; (approx.) token honesty |
| 0.1.4 | 2026-10-11 | Full remediation cycle: metric integrity, reasoning-model correctness, engine selection, traceability, complete exports |
| 0.1.3 | 2026-10-05 | GPU & Power Monitor (View 4); multi-GPU; energy tracking + cost estimation; NVML power clamp; CPU power (RAPL / Super I/O) |
| 0.1.2 | 2026-10-04 | AMD + Intel GPU monitoring; vendor-agnostic backend; auto-detection; NVIDIA by default |
| 0.1.1 | 2026-10-03 | Three visual themes; Flat Out engine; authoritative token counting; multi-backend; practical sweet spot (30 t/s); methodology transparency |
| 0.1.0 | 2026-10-01 | Initial release: core speed / concurrency / capability engines, TUI dashboard, any OpenAI-compatible server. Single static Rust binary, zero deps |
Contribute
The repo is at github.com/MadGoatHaz/crucible-llm. If you find a bug, have a feature idea, or just want to mess with the engines, the issues and forks are open. A star is always appreciated, too.

