Pow3rtool and nvfp4 and oh my

High-Performance Computing Machine Learning, LLMs, & AI
1

Qwenfight in Pow3rtool:

https://pow3rtool.github.io/Qwenception/#dataset=sp06&baseline=h200-triton-bf16-bf16kv&runs=b200-triton-bf16-bf16kv%2Csm120-triton-bf16-bf16kv%2Csm121-triton-bf16-bf16kv&range=all&ties=argmax

Spawned from this discussion:

which has led to us doing analysis like this with our “breakout clone test”

^ this is where you can download the differently-configured breakout-clone-output files and bang on the tires a bit.

More yummy context:

and
context:

I think we’ve lost a bit of the conversation about quantization and output quality. We spend plenty of time comparing how much VRAM a model needs and how many tokens per second it produces. I want to know how much of the model’s useful behavior survived getting it onto my hardware. The Factory Must Grow.

That’s what thr3e and I have been digging into across these threads. My dual-Spark testing initially looked pretty good against the larger deployment. Expanding the tests exposed consistency problems that the first handful of prompts had missed. thr3e took this considerably further, isolating changes to weights, KV cache, attention kernels and execution configuration. Some differences were reproducible with the same weights and inputs. You cannot explain those away by saying “LLMs are random.”

A decent harness can paper over some of this quantization squirliness. The model calls a tool incorrectly, gets an error, reads the error and tries again. Tests fail, so it edits the code and reruns them. Eventually it finishes, and the benchmark records a success. It is cool it can recover, but it’s nice to know we went wrong somewhere. It also means the final score can conceal quite a lot of stumbling around. Agent evaluations increasingly track steps, tool calls and cost alongside completion, because getting there in three calls and getting there in thirty are different results. The extra calls alone don’t establish that quantization caused the problem, but they give us something else to measure.

For understanding the deviations themselves, I think Pow3rTool’s Qwenception explorer is a better tool. Pick a BF16 baseline and comparison runs, then watch what each configuration wanted to emit given the same recorded history. Where their next-token choices disagree, you can follow captured alternate continuations. It puts the deviation in front of you: a different argument, a different command, a tool call that no longer closes properly. It calls the tool for the wrong interface (oof). You can inspect the point where the paths separate, before a harness has spent another dozen calls persuading the model to recover.

BF16 isn’t automatically correct, either. We’re using it as a numerical reference. A deviation might be harmless, might improve the answer, or might send the model off configuring the wrong interface. The explorer helps us investigate that distinction. Its alternate tool calls weren’t executed, and a continuation that hits the length limit is marked inconclusive. I’m interested in understanding what changed and then testing whether it matters to the work we actually do.

That’s also why I’m excited about the recent GLM-5.3 NVFP4 work. There are improvements happening in both the checkpoints and the inference engines. Quantization-aware distillation lets the model learn to compensate for the reduced precision. Separately, Local Inference Lab’s lossless scale compression saves storage without changing the reconstructed scale values. On the full GLM-5.3 checkpoint, that saves about 22 GB of weight files. These are different mechanisms with different implications for quality, despite all appearing somewhere under “NVFP4.”

Local Inference Lab

The community deserves substantial credit for making this useful on local hardware. These guys popped up on my radar afterI finished the video, and are where I’ve turned my attention to now. Local Inference Lab’s Blackwell stack includes the kernels, runtime integration and PCIe work needed to turn a downloadable checkpoint into a usable system. NVIDIA’s GLM-5.3-Flash-FP4-Local instructions explicitly acknowledge cooperation with LIL and direct users to community containers while upstream support catches up.

The shout-out from nvidia is weirdly tucked into the deployment instructions, but the dependency is quite literal.

I want more of this: better quants, faster engines, and enough visibility to tell what we traded to get there. A harness successfully cleaning up after a model is worth measuring. So is how often it needed the mop for cleanup…

9 Likes
2

We should add CMP170hx 8gb and a100 to the Pow3rTool’s Qwenception explorer running at INT4/INT8. Thus compare older hardware to newer Blackwell and Hopper cards.

3

I would be super, super, super curious about AWQ and Autoround quants (IE w4a16) vs. ‘regular’ fp8 / fp4 quants with your pow3rtool.

Very cool and super interesting work.

4

The more I learn, the more it feels like the wild west. Ive got a 4090, 2x B70, and 2x R9700, with a 256g M5 Ultra on the way, and I cant find anything near a guiding principle for what to run on this mess besides “qwen” at anything above Q4, and keep kv at Q8 or more. And only that bit because of a general “this number is smaller when doing it this way, or this go fassszszzt!”.

So, looking forward to this, would love to know what knobs to turn, even if vicariously, for better results. Even if I have to ask hermes to decipher it and ELI5.

Wanting to get turnstone up on my laptop, but dont want to give hermes SSH lol. Looking forward to the video :slight_smile:

2 Likes
5

Great video wendell.

Venturing online, I’ll see a lot of discussions either geared towards guiding a beginner, or about a new model that came out, and they all largely follow the same formula.

Speed is king, closely followed by people’s interesting desire to run the model at the closest to full floating point precision. Whether the model is actually good, it’s toolset and harness constrain it properly, and what types of outputs the model produces, we’ll never really know, since it’ll be dissolved down into some “take”.

It can be quite frustrating as a person learning, to figure out the why of things, why is 4bit “bad”, why never quantize cache, and other mechanical questions. We take it all as hearsay from some random anonymized internet person.

The constraints of my hardware kind of pushed me to figure out all of this, from a bottom up approach. What’s the smallest model I can run, because I need space for context. What’s truly the difference between 4bit and 8bit quants, how does quantizing context affect the model, and what’s the mechanics behind that. Lots of deep down questions about this “black box” that are largely ignored in lieu of “what’s the best”.

Running the tests myself providing me insight that no random comment on reddit ever could. And the interesting part was, it was all kind of wrong. 3bit is fine, and now 2bit is showing promise? Quantizing your context, those issues won’t show up for a long time, if ever, and guess what, a ( changed into a [ will fail a gate, or a -l mistaken for -L in a cmd will fail, and then the model can just rewrite it. That maybe whatever flavor of the month when it comes to harnesses and inference engine forks isn’t actually the “best”.

I’m rambling, but basically, thanks for shedding light on output quality. That’s truly what matters. We’ll all get there in the end, by sharing ideas and what we’ve built with one another.

(proof-reading this, it sounds like ai speak, i interact with llm’s too dang much lol)

2 Likes
6

Deepinfer :slight_smile:

7

wendell:

I want to know how much of the model’s useful behavior survived getting it onto my hardware. The Factory Must Grow.

@wendell this is exactly what I’m trying to do as well. I’ve been able to get bit-wise token exactness on my rig if I account for what I call the deployment and prompt tuples.

Deployment tuple

  1. Model configuration

    • Model checkpoint and artifact hash.
    • Tokenizer and chat template versions.
    • Numerical formats: weight format and quantization, activation precision, and KV-cache format.

    I highly recommend pinning the model, tokenizer, and chat template to a specific Hugging Face commit or verifying the files by hash. One little push to the chat template or tokenizer, or a small “training” update to the weights, and you have a different model.

  2. Hardware configuration

    • GPU models.
    • GPU count.
    • Parallel configuration, including how the model is distributed across devices.

    DON’T trust the model families. Blackwell is marketing. The model number is the silicon. Trust the silicon.

  3. Inference engine configuration

    • Software stack: inference engine, driver, compute runtime, and numerical library versions.
    • Computation settings: attention backend and kernel selection policy.
    • Batching settings: batching policy and batch invariance configuration, with its verified scope.

    Batch variance is an interesting one. vLLM has a beta out for batch invariance, and they are working to optimize it now. Also see the Thinking Machines batch-invariant operations repo for implementations and tests.

The deployment tuple fixes the hardware and software conditions under which the math runs. Once the deployment is held still, you need to account for the prompt tuple: the input and conditions for each request sent to the model.

Prompt tuple

  1. The actual prompt and built context

    • System instructions and user input.
    • Conversation history.
    • Retrieved context and tool results.

    Preserve the complete serialized input.

  2. Generation settings

    • Sampling controls: temperature, top-p, and top-k.
    • Random seed, when applicable.
    • Output token limit and stopping conditions.

  3. Execution state

    • Relevant cache state.
    • Batch conditions during generation, when they can affect the result.
    • Kernel selections during execution with those inputs and batch conditions.

Holding both tuples still gives me a reproducible baseline for the model’s tool-call output. Here’s the sequence I use:

  1. Verify exact replay with both tuples fixed.

  2. Repeat that check across 20 to 50 seeds in a chosen set.

    • For each seed, repeat the same sequence of prompts from the same starting state, including the same supplied context and tool results. Use the same seed initialization or reset procedure each time.
    • Check that repeated runs within each seed reproduce the same token sequence and entropy trace. Different seeds may produce different token sequences, and once their histories diverge, later entropies may differ too. Variation between seeds is allowed. Repeatability within a seed is the test.

  3. Change one deployment or request setting and rerun the same seed set. Measure tool-call validity, retries, and completion. Changing output is not automatically breaking the agent.

I have spent the last year wrangling these variables on my rig and have built my own harness specifically to account for all of them such that I can feed my agent the trace of a previous run and reproduce it to bit-wise exactness. Why you ask? My goal is to take Anthropic’s Jacobian Lens and use it to analyze the model under the above conditions so I can run counterfactual diagnostic runs.

8

Hi @wendell , you provide us some great topics, but at the same time I feel the interesting part the videos content is dropping quality since there is less and less valuable chunk of information to digest anymore. No offense intended of course, just my two cents worth, you can leave on the counter.
Since you go the IA road, I think you have too much to do and to digest whitch might impact the level of informantion distilled in the content… pun intended ofc.
Taking the subject of this video as exemple : it is really interesting and could end in a paper work.
But at the same time you don’t go into enough detail, even when it comes to presentig your work. And it’s a shame because it seems an insane amount of work !
I’m all in for a following video going deeper into piping.

Overwise do some one carry a litterature survey on this subject ?
Because I think there are lots of good insight in papers on how to measure and monitor LLM answer and trajectory as you already have the walkthrougth.
There are probably dealing with a broader subject such as keep model on track with very large context and we have seen lots of improvement this year.
I think the tools used to help quantify that, can be used to quantify our strong formulation of the problem on quants and implementation impact.

I have no clue on tools used yet, but to give my contribution I think it is also possible to contructs some thougths experiment with the LLM to help assess its capacity to hold on.
These are a different road than what you all used and they might allready existe or be unhelpfull I don’t lookup yet.
Assuming model can handle context length better than it can preserve consistency of prediction, then we might use the former as a tracking of inconsistency of context.
1 - inject information in dud load of context (but beside what model can handle) and retrieve it at different steps in chat.
2 - inject information, and ask to replace it time to time while bursting it with dud load of context each time, and verify.
3 - load a BIG context in one shot and incrementally see when the model can’t handle simple fact retrival in this flow of words.
And how about tracking the thinking consistency (like do A to make B in order to have C) in long sequence ?

Anyway before rushing into results discussion, we also need references, with the proper model and implementation we can land.
I don’t know how mutch an LLM can be repetable when set to Temp=1 and so. With MoE there is also the question of expert routes.
Additionnaly if it is too difficult to follow and measure the model coherence, we can juste inject error and track divergence.
There we could also use part of the model or an other one to get what token injection is a probable one and ones that are unprobable. The two scenarios are interresting to follow !

@Todd_W_Bucy I never heard of this tool, awesome. Might be to mutch at a first state but can be usefull later on to test hypothesis and scenarios.

9

@wendell Get two more Sparks and load up DS v4.1F 1019 with its native quant. It is a different world compared to these small dense models.

10

He has 4x 6k’s and can already run ds41F, the native release, not some weird quant. In fact the performance on the 6k is vastly better than the spark.

A dense model was selected because MoE have additional complexity and challenges to analyze. This specific Qwen was used because of its widespread popularity across the local inference userbase.

Regardless of weights though, the point of the exercise is that depending on the underlying hardware platform (among a whole lot of other factors) even running the same weights with the same config results in different output with measurably different quality.

3 Likes
11

This evaluation is a more in depth variation of what we have been doing over at the Nvidia dev forums since spring.

There we use TEB to compare backends, models and quants. It is a pretty good test to see if things are getting better or worse between iterations.