Chance Rates in Disguise: When a Benchmark Quietly Measures Nothing
One of our models scored 0.235 on MMLU. For a 32B reasoning model, that is
chance. It looked like a finding about the model. It was a finding about our harness.
The tell
We logged every extracted answer across 200 questions. The distribution came back:
A: 152 D: 33 B: 5 C: 2 (blank: 8)
A four-option benchmark should land near 25% per option. Seventy-six percent of the
answers were a single letter. That is not a model that prefers A. That is an
answer extractor with nothing to extract, falling back to the first
option every time.
The cause
The raw responses explained it immediately. They were truncated mid-sentence at roughly
600 characters:
"Okay, so I have this multiple-choice question about abstract algebra, and I need to figure out the correct answer. Let me read it again: ... First, let's recall what a free abelian group is"
The model was reasoning correctly — coherent, on-topic, working the problem. It
simply never reached an answer, because the reasoning trace consumed the entire generation
budget. The request we were sending omitted the field that disables thinking for benchmark
work. With thinking left on, the model thinks until it is cut off, and the harness then
scores the cutoff.
The proof
With that one field set, the same model on the same benchmark produced responses of
approximately one character — just the answer letter — and an
extracted distribution that is finally balanced:
D: 4 C: 2 B: 2 A: 2
Accuracy on that sample: 0.60. That is a measurement. The 0.235 was arithmetic on a
default.
The rule we took from it
Check the answer distribution before you trust a score. If one option
dominates, you are measuring your extractor, not your model. It costs one line of code and
it is the difference between a result and a plausible-looking artifact. This check was
available all day and we did not run it until the end.
A related signature: empty or near-empty responses are not the same failure.
Here only 4% of responses were empty. The model produced plenty of text — it just
never produced an answer. “No output” and “output that never concludes” have different
causes and need different fixes.
What neither engine can do
We tested both serving engines for logprobs. Neither returns them, through any variant
we tried, both answering HTTP 200. That has a hard consequence: every loglikelihood-based
benchmark — MMLU, HellaSwag, ARC, WinoGrande, TruthfulQA — cannot be
run through either engine at all.
So the two questions have to be separated:
- What do the weights know? Run a published benchmark against the
checkpoint directly, with no serving engine in the path. - How well does the serving configuration do? Measure the engine:
quantisation, memory, latency, long-context behaviour, batch yield.
Those are different questions and they need different tools. We spent a day trying to
answer both with one hand-built suite, and got artifacts instead of answers.
Latency numbers are cache-confounded
A measurement worth publishing because it invalidates several of our own earlier
numbers. Repeating a prompt against a warm cache:
~8K tokens: cold 18.3s -> warm 2.3s ~32K tokens: cold 60.4s -> warm 3.1s
Up to roughly 20x. An earlier comparison of the two engines — one appearing to be
several times faster than the other — was almost certainly warm-versus-cold rather
than a real engine difference. Every latency figure needs a cold/warm annotation,
and engine comparisons have to be cold against cold.
What we would do differently
Four times in one session we drew a conclusion from a mid-run snapshot and revised it
when the next observation arrived. Each revision came from checking rather than assuming,
which is how it should work — but the underlying error was consistent: building on an
unverified assumption about where the data lives and what a default does.
The benchmark was never broken. Everyone else’s numbers are fine. Our request was
incomplete — and a default we never overrode turned a modelling question into
arithmetic on the first option.