A head-to-head challenge between one Qwen 3.8 27B served two ways ended in a perfect tie, so the test got harder. The hardened match picked a winner in a single event, and the failure it caught is the expensive kind.
Every figure below is corrected evidence. Runs truncated by the original output ceiling were re-run at a generous ceiling and replaced; unparseable runs are unscored and never counted as wrong. Latency is per call and includes model load. 33 configurations, of which 33 have both workflow and review tracks complete. Results by metric Workflow pass…
We assumed local vision was thin, and planned to download additional models to fill the gap. That assumption was wrong, and it was wrong for an instructive reason: we had inferred capability from model names instead of asking the engines what they support. Every Ollama model reports its own capabilities. Every MLX checkpoint declares its…
We built a benchmark to decide which local models earn a place in the fleet and which ones get retired. Then we discovered the benchmark had been quietly rigging the contest, and that one of its victims had already been publicly recommended for deletion. What we were measuring The preliminary suite is deliberately small and…
Ollama Qwen 3.5 27B Q8 passed all six scorer-v2 workflows, but ten of its 24 code-review findings were false or materially wrong after source inspection.
Ollama Qwen 3.6 35B passed all six scorer-v2 workflows at exceptional speed, then split 26 code-review findings into six useful, seven qualified, and thirteen false or materially wrong.
The Ollama GPT-OSS 20B path completed ten calls but exposed empty or unparsable visible output, leaving zero workflow passes and no review findings to adjudicate.
Ollama Gemma 4 12B passed all six scorer-v2 workflow cases, then mixed four useful code-review findings with nine that needed qualification or rejection.
A real-workflow qualification of Ollama Qwen 3.5 0.8B: three of six bounded cases passed, and all four code-review recommendations were false or materially wrong.
A real-workflow qualification of oMLX Llama 3.3 70B: four of six bounded cases passed, but request failures and invalid output prevented a fair repository-review judgment.
