Mostly yes, a lot of things were glossed over as I tried to make it both approachable as well as technically useful… That’s why while I have a huge KLD write up aside from the post it was not a central piece of the data.
Our harness does normalize the captured logits. It converts them to float64 and applies log_softmax, then calculates directional (D_{KL}(P_{BF16}|P_{candidate})), reverse KL, and Jensen–Shannon divergence. We retain per-token values and summarize them within individual output ranges rather than presenting one global average.
BF16 is a numerical-fidelity reference, not an oracle or correctness label. A quantized model can absolutely diverge from BF16 and produce a semantically better answer. Repeating the identical deterministic run tests reproducibility, but does not make BF16 correct; correctness requires labelled answers, executable tool-call checks, or semantic grading across varied workloads.
Our Top-1 percentage also is not a percentage difference between logits. It is the percentage of evaluated output positions where the candidate’s argmax token ID differs from BF16’s:
Top-1 disagreement = changed winner positions / evaluated positions. Top-1 and KL describe different things. KL measures movement of the whole distribution, including changes that leave the winner unchanged. Top-1 disagreement is a discontinuous winner test: an extremely small KL change can flip a 50.1/49.9 decision, while a much larger KL change can leave a dominant winner unchanged.
Top-1 is directly relevant to greedy decoding, but a teacher-forced Top-1 flip is still a counterfactual root, not automatically a different complete answer. That is why we additionally branch from selected flip positions and inspect whether the alternate continuation recovers, changes meaning, or produces malformed or incorrect tool calls.
I will be the first to point out using a random 3% token distribution (part 1) to measure divergence is not a correct overall methodology, but i needed a big picture view. That’s why part 2 went straight down the rabbit hole to what-does-a-top1-flip-mean and why would it matter to you in a specific use case.
There is a clear impact, to both end user perception of a model’s output as well as measured correctness in benchmark results. Just wait for what’s coming next…
The initial H200 runs mostly finished last night, and B200 runs finishing this morning sometime.