Same 27B model, two engines, and the event that finally separated them

The fleet dashboard we put up earlier today lists thirty-three local configurations with their workflow, code-review and vision results. It answers a lot of questions and it cannot answer this one. The same Qwen 3.8 27B model appears on it twice, and the page would not pick between them. The Ollama NVFP4 build at 16.93 GiB on disk passed five of six workflow cases and grounded three of four code-review runs. The oMLX 8-bit MLX build at 27.52 GiB passed six of six and grounded two of four. Both passed all twenty-four vision questions. Ten and a half gigabytes separated them, and nothing else on the page did.

Adding another column was not going to settle that, so we built a head-to-head instead. Five deterministic events over a long document packet, every answer checkable exactly, no judge model anywhere in the loop. The same base model sits on both sides, which means the match measures the serving stack and the quantization rather than which model is smarter.

The first match ended in a tie

The first version of the challenge asked for a single planted fact at 8K, 32K and 64K of context, a quoted contradiction, and twelve facts attributed across six properties. Both contenders came back with five of five, 121.4 seconds against 123.7.

That is the corrected score. The first scoreboard said both of them had failed the contradiction event, and that is the kind of result worth reading before believing. Both models had quoted both sentences verbatim and given both figures. The event failed them on a check that asked them to announce the disagreement back to me, in a question where I had already stated that the documents disagreed. They did the work; my scorer punished them for not repeating my own premise.

Which left a tie, and a tie between two capable systems is a fact about the test. Every event wanted one findable sentence, and one buried sentence is not hard work for a 27B model. The challenge needed events that reward capacity instead of lookup.

Hardening it until it separated them

The second match replaced lookup with the things that actually differ between configurations.

• Eight facts of identical shape buried at eight depths of a 64,000-token packet, at lines 359 through 2,879 of a 3,381-line document. All eight required. One miss fails the event.
• Six documents giving a shuttle interval: four agree on twenty minutes, two carry a superseded thirty-five. The majority figure, the exact 4-and-2 split, and one verbatim quote per distinct figure.
• A join across two documents with a near miss sitting next to it, asking for the property with private beach access and more than two hundred rooms when a second property has the beach access but not the rooms.
• Fifteen facts across three names that differ by one space and one word, where any cross-contamination between them fails the event.
• Five rate cards whose values come from the same handful of numbers, asking for three exact figures in a specified order.

Everything is scored by comparison against known answers. Prompts go to both contenders unchanged, completion budget fixed, latency recorded per call including model load.

The result: five events to four

oMLX won the match five events to four, and for most of it the two builds looked identical. Both pulled all eight facts out of the 64,000-token packet. Both handled the versioned contradiction completely: majority figure, the split, a verbatim quote for each. Both made the two-hop join, and neither fell for the trap. Both kept fifteen near-identical property names apart without a single cross-contamination.

The difference sat in one event. Five rate cards, values drawn from the same five numbers, one question: give the two numbers from rate card C, then the first number from rate card E, in that order. The expected answer was 148, 319, 152. The 8-bit MLX build returned exactly that in 2.3 seconds. The NVFP4 build returned 319,148,327 in half a second. It reversed the pair on card C, then took card E’s second value where its first had been asked for. Eleven tokens and half a second, with no sign of hesitation anywhere in the answer.

Anything running unattended can absorb a model that stalls. What it cannot absorb is a question with exactly one right answer coming back instantly and wrongly, which is what the deciding event exposed.

What the win is worth

The win is worth less than the scoreline suggests, and more than nothing. It is one pairing, one run per event, and latency here includes model load, so the totals describe five cold calls as much as they describe generation. Read honestly, the match settled the tie-break rather than the fleet question: the larger 8-bit build owns the event where precision is the whole point, and it pays for that with 10.6 GiB of memory and a longer wait on every event both builds passed. Five events took it 223.0 seconds against 89.6.

The useful question is not which build is better but how often the smaller and faster one gets the numbers wrong. That is cheap to answer by pointing more events at exactly that behaviour. A stable failure is a deployment decision; a coin flip is something else, and one run of one event cannot tell them apart.

Two things this changed about how we test

When both contenders fail the same way, read the raw answers before writing a verdict. Two identical failures in a head-to-head point at the harness far more often than at the models, and this one was mine: a check that demanded confirmation I had already handed them in the question. The correction cost a rerun. Left alone it would have cost a wrong conclusion about two models at once.

Design the events so that a failure can be localised. The deciding event produced an eleven-token answer that could be compared against three known numbers, which is why it settled the match instead of opening an argument. Events that fail in ways you cannot inspect teach nothing, and they are the ones that quietly disappear from a results table.

The bracket has one completed match and a fixture list for the rest. Nothing comes off the machine on the strength of five events, so the next match is the precision one, with repeats.