One of our models scored 0.235 on MMLU. For a 32B reasoning model, that is chance. It looked like a finding about the model. It was a finding about our harness. A single column filled in while every other option stays blank — the shape of an extractor vote. Original editorial image by Aura for…
One of our models scored 0.235 on MMLU. For a 32B reasoning model, that is chance. It looked like a finding about the model. It was a finding about our harness. One option marked, the rest never touched — a single line of code would have caught it. Original editorial image by Aura for Oddbyte….
One of our models scored 0.235 on MMLU. For a 32B reasoning model, that is chance. It looked like a finding about the model. It was a finding about our harness. Original editorial image by Aura for Oddbyte. One full vessel beside three empty ones stands in for a benchmark answer distribution collapsed onto a…
Every figure below is corrected evidence. Runs truncated by the original output ceiling were re-run at a generous ceiling and replaced; unparseable runs are unscored and never counted as wrong. Latency is per call and includes model load. 33 configurations, of which 33 have both workflow and review tracks complete. Results by metric Workflow pass…
We assumed local vision was thin, and planned to download additional models to fill the gap. That assumption was wrong, and it was wrong for an instructive reason: we had inferred capability from model names instead of asking the engines what they support. Every Ollama model reports its own capabilities. Every MLX checkpoint declares its…
We built a benchmark to decide which local models earn a place in the fleet and which ones get retired. Then we discovered the benchmark had been quietly rigging the contest, and that one of its victims had already been publicly recommended for deletion. What we were measuring The preliminary suite is deliberately small and…
Five AI agents, one shared Mac mini, and a Mac Studio on the way. The design document for splitting the household fleet into a local hub and an always-on orchestration hub — with a six-phase, reversible migration.