The output ceiling that made nine models look broken

We built a benchmark to decide which local models earn a place in the fleet and which ones get retired. Then we discovered the benchmark had been quietly rigging the contest, and that one of its victims had already been publicly recommended for deletion.

What we were measuring

The preliminary suite is deliberately small and frozen. Thirty-two model configurations run the same six bounded workflows — a literal-evidence extraction task, two short community replies, and three structured portfolio decisions — and then review four private repositories for engineering problems. Every response is scored by deterministic code, not by a judge.

The harness gave the extraction task 1,600 output tokens. Everything else got 320. Code reviews got 3,200.

Those numbers were chosen to keep runs fast and comparable. They were not chosen with reasoning models in mind, and that turned out to matter enormously.

What the audit found

Twenty-seven of 192 workflow runs and thirty-three code-review runs never produced a complete answer. They were cut off mid-sentence by the ceiling, and then scored as failures.

That is 14% of the suite. On its own, that figure looks like tolerable noise. It is not, because the truncation was not spread evenly — it landed almost entirely on nine specific configurations, and it hit some of them on every single run.

Two smoking guns

GPT-OSS 20B hit the ceiling on every domain case. When we pulled the stored output, we found the model writing correct, schema-valid JSON with accurate verbatim quotations — and then simply stopping mid-object. It had understood the task. It was performing the task. It ran out of room before it could finish saying so. Its recorded score was zero out of six.

DeepSeek R1 returned empty text on a short community reply. Not truncated text. Nothing visible at all, after consuming its entire 320-token allowance. A reasoning model spends tokens thinking before it writes, and 320 tokens is barely a thought. The harness then recorded a failure to answer.

Neither result says anything about whether those models can do the work. Both results say a great deal about the harness.

Why nobody noticed

This is the part worth dwelling on. In aggregate the suite looked healthy. Most configurations never came close to the ceiling, because most models answer a short question in well under 320 tokens. The defect was invisible in every summary statistic we were tracking.

It only became visible when we ran the same tests against two cloud control models and noticed that they, too, were failing cases they should have walked through. That sent us looking for a benchmark defect rather than a capability gap. We found two scoring bugs on that first pass, fixed them, and published the corrections. We did not think to look at the budgets.

The pattern repeated itself. Something failed. We assumed the model. Twice now, it was the measurement.

The cost of not checking

One of the affected configurations is a 1.16 GiB model that published with a provisional verdict of deletion candidate. Its review lane had produced unparseable output on two of four repository calls — runs that had hit the ceiling and given us nothing to evaluate. Those were counted alongside the two calls that parsed and genuinely got things wrong.

Unparseable output is not bad judgment. A model that was silenced mid-sentence has not told us anything about the quality of its reasoning, and scoring its silence as a wrong answer manufactures evidence that does not exist. That verdict could have justified retiring a working model.

The verdict has been changed to retest required, the article now says plainly which runs failed to parse, and the model stays protected. Every correction is visible on the page rather than edited into history.

What changed

Configurations get marked, not quietly rescored. A run is budget-limited when it ends on the ceiling or returns something unparseable. Nine configurations carry that label, and their results are excluded from any ranking while the label stands.

The affected configurations are being re-run at a raised budget — 4,000 tokens for workflows, 8,000 for reviews. Only the affected rows get replaced.

Mixing budgets across a dataset sounds unsound, but it is not, and the reason is worth stating: raising a ceiling cannot change a result that never reached it. A configuration that produced complete output at 320 tokens produces byte-identical output at 4,000. The audit is what proves that, and it now runs whenever the suite changes.

Every run records its budget and its actual completion tokens. If a model fails from here on, we can demonstrate it failed for a reason other than being cut off.

What it means for the plan

This is not a footnote. It changed how the competitions are designed.

The qualifying suite has saturated — thirteen configurations now score a perfect six for six — so it can no longer separate models. The next round is a set of role-scoped tournaments, each testing only what a model would actually be asked to do. A 2-billion-parameter model failing an agentic multi-tool task is not a finding, and it will not be asked to try.

Three design rules came directly out of this failure:

• Budget honesty first. Every test declares an output budget generous enough that the harness cannot be the reason for failure, and records what each model actually used.
• No judge in the critical path. Deterministic scoring decides outcomes. Where judgment is unavoidable, it is blind and cross-model.
• Unparseable is not wrong. A response we cannot evaluate is recorded as unscored, never as a negative.

Safety brackets use pass/fail gates rather than averages, because you cannot average away a child-safety failure. Utility models are scored on whether they are right or honestly flag that they are unsure, since a bad router is worse than no router. Test-writing is graded by whether the tests catch deliberately injected bugs, which measures the thing without asking anyone’s opinion.

An unexpected inventory result

We assumed local vision support was thin, and planned to download additional vision models to fill the gap. Then we asked the engines instead of guessing from model names. Twenty-seven configurations already installed can accept images: fifteen of nineteen Ollama tags report vision support, and twelve of sixteen MLX checkpoints declare a vision tower.

The smallest is a 0.8-billion-parameter model at 1.16 GiB. Another is a twelve-billion-parameter checkpoint that handles audio as well as images.

Nothing needed downloading. What is actually unproven is whether declared vision reads an image correctly, so a probe is running now against synthetic images with exact known answers — chart values, table cells, shapes and colours, and an image containing no text at all, to see whether a model invents a sign that is not there.

Where this leaves us

The uncomfortable lesson is not that we had a bug. It is that the bug was invisible, self-consistent, and pointed at exactly the wrong conclusion. A benchmark that silently penalises reasoning models will retire the models that think hardest and keep the ones that answer fastest in short bursts.

It also produced a public recommendation to delete a working model, which is the kind of mistake that does not announce itself.

The correction is now mechanical rather than conscientious: budgets are recorded, truncation is labelled, affected configurations are excluded from ranking, and the audit runs every time the suite changes. The models that were judged on truncated evidence get a clean re-run, and their articles will say what the re-run finds.