Crucible LLM :: Bench your LLM and server. Is it fast enough… for everyone?

I built Crucible LLM — a terminal-based benchmarking suite that stress-tests any OpenAI-compatible inference server and measures what a real user actually experiences. One static Rust binary. No Python. No venv. No dependencies.

Why it exists

Most benchmarks shout “xx.x tokens/sec!” and stop. But when 32 users hit that server at once, each one gets 2 tokens/sec. A number that looks fast in the aggregate can feel painfully slow per user. Crucible measures that gap — the difference between a server’s peak throughput and what each individual connection actually gets — as well as how competent the LLM is at real tasks.

What it does

  • One static binary. A single ~15 MB Rust executable. No Python, no venv, no system libraries — SQLite and GPU telemetry are compiled in. Point it at any /v1 endpoint and go.
  • TUI-first, 60 Hz dashboard. Six keyboard-driven views (Live Monitor, Concurrency, NIAH, GPU & Power, History, Config) with real-time graphs and per-engine metrics. Headless/CLI mode for CI and scripting.
  • Seven benchmark engines. Speed → Concurrency → Long-context (NIAH) → Reasoning → Structured output → Energy → Flat Out, each individually selectable.
  • The “practical sweet spot.” The concurrency sweep finds the highest user count where each person still gets ≥ 30 t/s — the comfortable hosting number, kept distinct from the raw peak.
  • Reasoning-model aware. Separates the chain-of-thought from the answer, judges on the answer, and reports over-thinkers honestly (truncated_by_reasoning) instead of failing them.
  • Deeper-context NIAH. Its own needle-in-a-haystack matrix, runtime-configurable and probing RoPE/YaRN past 128k — and now settable right in the TUI, not just the CLI.
  • Multi-vendor GPU monitoring. NVIDIA, AMD, and Intel, with live power / utilization / VRAM / temperature and energy-per-token (Linux).
  • Accurate energy. Joules/token measured over the full run — full load window, idle baseline subtracted, real token denominator — with an honest N/A instead of a silent 0.
  • Flat Out = true max throughput. A 60-second run at the concurrency peak — the headline number to quote when comparing setups — distinct from the practical sweet spot.
  • The Matrix Report. One key (m) or one flag (--matrix-report) produces a ready-to-paste [COMMUNITY-SUBMISSION] block for the Local LLM Matrix.
  • Run-log access. Press l (or --show-log headless) to view, copy, or save the full run log, now with a [CONFIG] block, per-engine wall-times, and a shareable run-end [DIGEST].
  • Any OpenAI-compatible server. vLLM, llama.cpp, LM Studio, Unsloth, SGLang, Ollama, TGI.
  • Three visual themes. Cyberpunk, Vampire, Monochrome Pastel.
  • Headless/CI-ready. Pure JSON, meaningful exit codes, export to JSON / Markdown / CSV.

See it in action

The Live Monitor: a real-time throughput graph with an auto-scaling y-axis, engine-transition markers, key metrics, and a live event log.

Try it

Grab the latest pre-compiled binary from the Releases page — Linux, macOS, and Windows are all built:

chmod +x crucible-llm-v0.1.9-linux-x86_64
./crucible-llm-v0.1.9-linux-x86_64

Or build from source (stable Rust toolchain):

git clone https://github.com/MadGoatHaz/crucible-llm
cd crucible-llm
cargo build --release
./target/release/crucible-llm

On first run, the TUI asks you to pick a theme, then walks you through entering your server URL and picking a model.

The v0.1.5 → v0.1.9 arc

Last month I wrote about making reasoning honest — separating the chain-of-thought from the answer, judging on the answer, and reporting over-thinkers without failing them. That was the foundation. The four releases since have been about the rest of the numbers: making everything else honest too, and making the results easy to share. Every number is now accounted for.

v0.1.6 — Energy, done right. The energy figures were lying to me. The J/token number was roughly 25× too high, and when I dug in there were three compounding reasons: a phantom GPU power reading that slipped past the sanity check, a denominator that only counted a fraction of the real tokens, and a 6-minute sample window on a 19-minute run (plus an estimated CPU power floor that was never labeled as an estimate). All of it is fixed now. The energy integral runs over the full load window, the denominator counts the real prompt + completion tokens, a generic power-sanity clamp catches phantom readings, and the idle baseline is subtracted so you see the load-attributable energy. Flat Out also finally runs at the true max-throughput point — the peak of the concurrency sweep — which is a different animal from the “practical sweet spot” (the comfortable per-user hosting number), and the two are no longer conflated. And the NIAH ladders are now settable in the TUI, not just on the command line.

v0.1.7 — Sharing made easy. The headline addition for the community is the Matrix Report. After a run, press m in the TUI (or --matrix-report headless) and you get a ready-to-paste [COMMUNITY-SUBMISSION] block for the Local LLM Matrix. Crucible auto-fills the things it can measure — hardware, VRAM tier, model, and speed — and you fill in the things only you know — backend, quant, use case, and notes. Copy it to your clipboard (OSC 52) or save it as a .md, and it’s on the Matrix. This release also folds in the corrections from 0.1.6: I’d been hardcoding a per-GPU TDP table so the power clamp knew, say, “an A4000 is 200 W” — the kind of assumption a general benchmarking tool shouldn’t make — so I removed it. The clamp is now generic: ≤1.5× the card’s queried TDP when it’s available, and when the query fails the reading stands but is flagged unvalidated — a label of uncertainty, not a correction. The energy denominator was completed to include all the concurrency legs, and the efficiency panel reports an honest N/A when no tokens were generated, never a silent 0. And this is the release where I started shipping an actual downloadable binary, with a CI workflow that builds Linux, macOS, and Windows on every release — so you no longer have to compile it yourself.

v0.1.8 — The run log, made easy to grab. I made the full run log easy to view and share. Press l in the TUI (any time, no completion gate) to open a full-screen, scrollable log viewer — c copies it to the clipboard, s saves it to a file, Esc closes — and --show-log headless prints the whole log to stdout. The log is now self-describing, too: a [CONFIG] block at the top records the resolved inputs (url, model, mode, tokens, iterations, the concurrency and NIAH ladders, engines, hardware), each ✓ engine headline carries its wall-clock duration, and a compact run-end [DIGEST] block is the thing to paste when you share a run. The Matrix Report also got sharper here: a multi-GPU pool’s hardware string is now unambiguous (8x NVIDIA RTX A4000 (16 GB each, 128 GB total) + 118 GB RAM — per-card and pool total), and for pure-reasoning (“CoT-only”) models the speed falls back to an honest blended decode rate with an explanatory notes line — never a fabricated answer-phase number.

v0.1.9 — Every number accounted for. This one is a measurement-integrity pass. The theme: a number means what it says. The biggest fix is no more phantom rates. A sub-10 ms answer window (say, a 2-token answer delivered in one batch) used to divide by a clamped 1 ms sliver and mint a fake ~2000 t/s — the old “C2 avg 799 t/s on a 27B model.” Now an unmeasurable window reads n/a and is excluded from the averages, instead of a fabricated number. Flat Out’s token counts use one method so the summary and the per-stream lines reconcile; stream complete lines are self-consistent ({n} tokens in {total}s (decode {rate} t/s; TTFB {ttfb}s)); there’s exactly one started line per engine and no duplicate complete lines; the [DIGEST] is a genuinely shareable block (version, config, wall-times, failures, totals, hardware); and the [HW] line is unambiguous for a multi-GPU pool (8× NVIDIA RTX A4000, plus the vLLM version and driver/CUDA). Honest by design: an unmeasurable window is n/a, never a made-up number.

The through-line across all four: I’d rather report an honest n/a than a confident wrong number. That’s the whole point of a benchmark.

Thanks

A big thank-you to @harshithmohan and @strawberry

Feedback — your help is the real test

This is where I need you. Two things would be great:

  1. Try the Matrix Report. Do a run, press m, fill in your backend / quant / use case, and post your rig to the Local LLM Matrix. The more honest, consistently-measured entries there are, the more useful it gets — and if anything in the auto-filled block looks off for your setup, tell me. I’d rather fix a quirk than ship a bad number.
  2. Sanity-check the numbers you care about. Run v0.1.9 against a model you know, and check the energy / J-per-token and Flat Out figures against what you’d expect from your hardware. Those are the two I rewrote, and a second pair of eyes on real rigs is exactly what I want.

If a number looks wrong, post it — the [DIGEST] block is the quickest thing to paste. I read every one.

Changelog

Version Date Highlights
0.1.9 2026-10-20 Measurement-integrity pass: no phantom rates (sub-10 ms → n/a, excluded from averages); Flat Out token counts reconcile; self-consistent stream complete lines; one started / no duplicate complete per engine; richer shareable [DIGEST]; unambiguous multi-GPU [HW] (+ vLLM / driver / CUDA)
0.1.8 2026-10-19 Run-log access (l TUI / --show-log headless); run-end [DIGEST] block; [CONFIG] block at start; per-engine wall-times; unambiguous multi-GPU hardware string; honest blended rate for pure-CoT models
0.1.7 2026-10-18 Matrix Report (m / --matrix-report) → [COMMUNITY-SUBMISSION] for the Local LLM Matrix; no card-specific TDP (generic clamp + unvalidated flag); energy denominator completes all concurrency legs; honest N/A (never a silent 0); CI ships Linux / macOS / Windows binaries
0.1.6 2026-10-18 Energy J/token ~25× fix (full load window, real prompt + completion denominator, idle-baseline subtraction, generic power clamp); Flat Out = true max-throughput (distinct from the practical sweet spot); NIAH ladders settable in the TUI
0.1.5 2026-10-12 Hybrid reasoning (budget = correctness, disable_thinking best-effort, truncated_by_reasoning); runtime-configurable NIAH (2k → 1M); SSE keep-alive routing fix; (approx.) token honesty
0.1.4 2026-10-11 Full remediation cycle: metric integrity, reasoning-model correctness, engine selection, traceability, complete exports
0.1.3 2026-10-05 GPU & Power Monitor (View 4); multi-GPU; energy tracking + cost estimation; NVML power clamp; CPU power (RAPL / Super I/O)
0.1.2 2026-10-04 AMD + Intel GPU monitoring; vendor-agnostic backend; auto-detection; NVIDIA by default
0.1.1 2026-10-03 Three visual themes; Flat Out engine; authoritative token counting; multi-backend; practical sweet spot (30 t/s); methodology transparency
0.1.0 2026-10-01 Initial release: core speed / concurrency / capability engines, TUI dashboard, any OpenAI-compatible server. Single static Rust binary, zero deps

Contribute

The repo is at github.com/MadGoatHaz/crucible-llm. If you find a bug, have a feature idea, or just want to mess with the engines, the issues and forks are open. A star is always appreciated, too.

2 Likes

You might pull some ideas from GitHub - local-inference-lab/llm-inference-bench: LLM inference decode throughput benchmark with Rich TUI dashboard. Measures token generation speed across concurrency levels and context lengths. Supports SGLang and vLLM engines. · GitHub this is the blackwell performance community gold standard right now

1 Like

Very interesting, similar in many ways. Thanks for the heads up. It’s good to know what people are using.

Although I think Crucible aims to and offers more insight into not just the performance overall, but the LLM itself as well.

Thanks again :+1:

Am I missing something? I downloaded it, entered the URL, chose a model, hit enter, the model loaded on the target, but it’s not running any tests. I am using llama.cpp but according to the quick little readme here, it shouldn’t matter. I tried this, pointing it at several different machines I have. I also just tried pointing it directly at LM Studio and it seems to behave the same. A tiny blip in the GPU usage and then nothing.

I was looking forward to really pushing some tokens and running some benchmarks. I actually have a project I’m working on right now to figure out what the best model/quant is for use on my rig with Qwen 3.8 Flash Next, and this would be conveniently, immediately, ironically useful.

1 Like

You are correct in what you’re seeing and I can confirm I’ve replicated it on my side - it is for sure a deadlock and i’m working on a change to the engines now. I should have a the new release with a couple new merges as well released here within the hour or so. Thanks for the feedback! - I’ll update the main thread and this post when they’re up.

1 Like

The idea of measuring per-user performance instead of only aggregate tokens/sec is really useful. I especially like the “practical sweet spot” concept, because a server can look great in a benchmark while individual users are already getting poor response times. The TUI and CI/JSON output also make it seem practical for both testing and ongoing monitoring.

1 Like

I’m sitting here hitting F5 every hour on the Github page watching the commits roll in like :eyes:. I know it’s a volunteer project, but I’m very excited.

1 Like

I’m so sorry, I fell asleep at the desk last night and today was a nightmare - BUT, I’m wrapping up the release right now… I won’t give a time frame as I blew it last time saying that … Soon, very very soon (tonight). I will update here when It’s pushed. LOTS of changes I wanted to complete that all got included in this release.

Crucible LLM v0.1.1 — Big Update - Pushed and should be up. Any and all feedback VERY welcome. THANKS!

1 Like

I’m glad I upgraded the power supply recently…

Crucible LLM v0.1.2 — GPU Monitoring for Everyone - Pushed Again: Any and all feedback VERY welcome. THANKS!

If you could - I would love to know how Crucible reports the power for your AMD cards. I don’t have any AMD cards to test on (nor Intel for that matter atm).

It reports only the first card? This feature is very new. I can say, what it’s reporting is definitely correct. The 9700 spikes over 400 watts on NIAH test because it’s pure prefill. I was monitoring with LACT and running the benchmarks from my dedicated Linux desktop remotely. It only sees one card, other than that this is VERY nice.

1 Like

– That is INCREDIBLY helpful. Thank you. I was wondering if it was going to only grab one card, this confirms that - Also, are you using 0.1.3? - I have been working on adding a new power tab this evening - I wonder if the cards are reported in the power tab or if it’s still pulling GPU0?

Thanks a million either way!

I just downloaded a fresh copy of it because as mentioned in the notes, it needs to run on the machine with the cards in it in order for that feature to work. And I noticed that it’s on 0.1.3 versus the 0.1.1 I’ve been using all day. Only one GPU shows up on the Power tab, but all of the metrics appear to be correct, which is very promising.

I’m also wondering how it calculates the cost per million tokens. Is there a way to provide a power cost? That figure does appear to be a little high… It also appears to be incrementing, like it’s supposed to be a flat figure. Unless it only says dollar per million tokens and it actually means cost for this session? There’s also the difference of input versus output tokens… Anyway, I really like what you’ve been doing with this thing. Keep it up. This is so fantastic. This is already incredibly useful. As I mentioned before, I’ve been using this all day to benchmark different models to figure out which ones can properly locate data in their context, which ones are the most effective for the use cases that I’m going to be subjecting them to… I have a pile of hardware lying around and most of these are doing double duty, like the 5090 in my daily driver, and so on. Being able to just point an executable at an API endpoint and have it return a report that says, this loaded configuration can do X, Y, and Z, and it can’t do A, B, and C is so incredibly helpful.

I’m also curious about the needle in a haystack test. So far, not a single model that I’ve loaded has been able to return anything at all on any position in the 128K context. I also would like to be able to test beyond that because I’ve been using Qwen with the maximum context set, and I’d like to see how it does with rope/yarn.

1 Like

— Crucible LLM v0.1.3 — GPU & Power Monitor Released

Lots of changes and polishing - Thank you @Strawberry for the feedback and testing!

Thank you for sharing your tool.

May you consider for this aspect a dispersion behaviour, either in tk/s or more praticaly in latency ?

This is a really good idea! - Can you elaborate on what you’d like to see / how you could benefit from such information? It would help me design / configure the data to be the most helpful to the end user.

Thank for the feedback / suggestions!

With 4x9700?

I am in fact using 4x 9700 pro cards. Mostly qwen 3.8 flash next, qwen 3.8 27b models and their derivatives. I recently switched to Radiance which is a real game changer for these cards.

Hey, so I have one more question / suggestion / discussion item. I asked some colleagues to run the needle in a haystack check on their models as a sanity check on mine. They were getting some very interesting results and it was failing almost 100% of the time no matter what their settings were. So, they had their models go digging around in the codebase, which I’ll admit I probably should have done myself, but didn’t.

To run the needle in a haystack check, do you have to disable thinking? It looks like it caps the output tokens at 64. Which means if you have a thinky model, the thinking will run out the cap and it will fail the test, even if the return would have been perfect otherwise. If this really is the case, it would go quite a long way to explaining the oddness I’ve noticed.

They also had some comments about the prompt that’s being used in A and F. It’s only 45 tokens, so the reported speed for the pre-fill is entirely inaccurate and doesn’t scale.

If I get any other feedback, I’ll be sure to share it here. And of course, as this is a volunteer project, you have no obligation to go fix it, but I thought you might want to know.

1 Like