Part 2:
Last weekend I ran a broad statistical comparison centered on one dataset and 3 specific scenarios: What happens when you change the attention cuda kernel, what happens when you lobotomize KV cache, and what happens when you compare the base model with two 8bit and two 4bit quants.
The output was centered on probability distributions that were severe enough to result in token flips, top-1 change output. Some of these were absolutely fascinating when viewed in depth, so for part 2 of my evil plan to take over^H^H^H^H^H^H drag the local LLM sins out into the open, the methodology is going to shift.
Rather than stay high level and capture 3% of the logits, I am now going to capture 100% of the logits for the most impactful areas of the workstream: during tool calls. Gentlemen, we need to go deeper…
Inferenception. A stream within a stream.
I built a small visualizer for my massive hypercube of test case logit captures. It shows a parallel stream of output tokens from some number (2-5) comparable runtimes. And when they differ? We branch and follow both.
The only requirement across runs is the dictionary be the same (so I’m staying within the qwen3.x model family) but can be anything.
Low level cuda kernel and NCCL path differences, driver differences, vllm container runtimes, different GPUs, different combinations of multiple GPUs, attention runtimes, different caching, different model quantizations, and in fact… even different models. 3.6 vs 3.8 anyone?
When the token flip happens, we do not stop and yank the wrong model back to the teacher. We let it continue. This forked multiverse of token output shows us where it went after the error and how it diverged!
Network Qwengineering
Lets go all the way down to unstructured tensor space, see a real failed tool call, a token flip caused by the difference in attention back end / cuda kernel:
In this example of a single token flip, the model executes a tool call targeting an interface on a Cisco router: GigabitEthernet0/0/1.201
Flash attention 2 gets it wrong. It targets GigabitEthernet0/1/4 instead.
Then, it runs the wrong command AGAIN in two diverging tool calls:
The correct command (trying to find the owner of a mac address) is show mac address table. FA2 tries to show run its way out of the mess that token flip has gotten it into.
In this next example, the LLM tried to configure a description on an interface. The FA2 token flip failed to execute that task at all.
Remember. this is simply changing one configuration option in VLLM to select TRT/FA2/FI. Same gpu, same os/drivers/software/vllm/prompt/cache/weights/activations… And this is REPEATABLE between runs, bit-identical logit captures!
If a simple runtime difference in cuda kernels caused this to happen in production, it could result in a critical network outage. The positioning is so impressively bad I could not have hoped for a better example of why precision measurement and testing matters!
Enter the Tensorverse!
We see token flips in many scenarios. This is comparing Tensor Parallelism vs single GPU:
At TP1 we get an acceptable tool call, at TP2 it fails, at TP4 it succeeds again. WTF?! (This is USUALLY NCCL’s fault when you debug even further and capture the nccl graphs…)
Across the 5-weight quant-off from the weekend:
We have BF16, FP8, INT8, and W4A16 all getting it right. Only NVFP4 fails this tool call >_>
Wrap up
Short post today due to work.
We have a growing pile of test case captures in a variety of prompts. Most of my lab include network automation so the corpus will improve as I identify and scrub additional workstream sessions out of turnstone.
So far I have detailed 100% captures with forking realities with:
- Different weights
- Different models
- Different KV cache quantization
- Different tensor parallelism
- Different NCCL settings
- Different attention backends
- Different cards (6k vs 5090) in SM120 family
- and much much more.
I am working to package up some of the testing tools and dataset into a distributable package people can run on their rigs and report results, as well as a vast run using a couple rented hopper/blackwell/etc. GPUs