The 8-bit model won on judgment, not speed
I expected this test to end with a speed chart. It ended with a subtraction problem.
The subject was Qwen 3.8 27B on a 64 GB M5 Max, served two ways: Ollama 0.34.4 and oMLX 0.7.0rc1. I wanted two answers that are easy to blur together. First, which serving engine handles the closest practical 4-bit workload better? Second, once inside oMLX, is the extra memory cost of 8-bit weights buying anything useful?
The machine gave a wonderfully inconvenient answer. oMLX was the better long-context engine. Four-bit was the thriftier quantization. Eight-bit was the model I would trust with an agent.
Keeping the lanes separate
A benchmark like this becomes nonsense if every variable moves at once. The models ran serially, never side by side. After each lane, the harness unloaded the model and verified that resident model memory had returned to zero. Every candidate saw the same synthetic records, tool schema, temperature, seed, output limits, and context targets.
The context prompts landed at 1,060, 8,242, 32,810, and 65,585 tokens according to the APIs. Short tests had three measured runs after warm-up. The 32K and 64K runs were expensive enough that each candidate got one measured pass, which is a limitation worth stating plainly. I kept the failed setup attempts too. One generated far more tokens than intended; another let hidden reasoning consume the visible answer budget. Neither was quietly edited out of history.
“Cold” here means the model was not resident before the request. I did not purge the operating system’s file cache between lanes, so this is not a claim about a physically cold SSD. Warm passes repeated an identical prefix and exposed the enormous difference cache reuse can make.
oMLX changed the long-context wait
The closest 4-bit engine comparison is Ollama’s GGUF Q4_K_M model against the oMLX 4-bit model. These are different formats, not interchangeable copies, so I treated this as an operational comparison rather than a pure kernel contest.
At 64K, Ollama took 176.56 seconds to produce the first visible token. oMLX took 60.26 seconds. That is a 2.93-times improvement where an agent feels it most: the dead air before anything comes back. At 32K, the same pattern held, 64.37 seconds for Ollama and 31.80 seconds for oMLX.
The memory result also favored oMLX. Its 4-bit model topped out at 15.70 GiB of observed resident model memory, against 18.33 GiB for Ollama’s GGUF build.
Ollama did not lose everywhere. Once generation started, it was much quicker on these short answers. Its measured median was 67.43 tokens per second, while oMLX 4-bit managed 30.36. Ollama also had the fastest tiny cached responses. If the job is a stream of short prompts with modest context, that still matters.
A supplemental Ollama NVFP4 lane was faster again at decoding and cut its 64K wait to 70.29 seconds, but it used 30.84 GiB. I would not call that the clean engine comparison, since it changes both representation and runtime behavior. It does show why one number cannot carry the verdict.
Four bits saved memory. Eight bits saved the answer.
Inside oMLX, the trade was blunt. The 4-bit model used 15.70 GiB. The 8-bit model used 28.85 GiB, nearly 84 percent more. Eight-bit decoded at 18.89 tokens per second in the measured median, about 38 percent slower. Its 64K time to first token was 63.53 seconds, only 5.4 percent behind 4-bit.
If the test had ended there, 4-bit would have won easily. It was smaller, faster, and nearly tied on long-context prefill.
Then came the quality packet.
Every candidate found the buried code at all four context lengths. Every candidate wrote an acceptable order-preserving Python deduplication function. Every candidate emitted the requested appointment JSON. Every candidate made a schema-valid weather tool call with the exact city and units, three times in a row. Concurrency at two and four requests also completed without a failed answer.
The reasoning check was deliberately plain: seven rows of nine bolts, with twenty-one used. Return only the number left. The 8-bit oMLX model answered 42 in all three attempts.
None of the 4-bit models did. Ollama GGUF returned 63 three times. Ollama NVFP4 returned 54 three times. oMLX 4-bit returned 72 three times. Same family, same prompt, same deterministic settings, three different confident mistakes.
This is a small quality suite, not a declaration that 8-bit weights are universally smarter. Still, agents live or die on boring correctness. A model that can call the tool perfectly and then drop the subtraction is not the better production choice merely because its token counter spins faster.
The production pick
I would promote oMLX with the Qwen3.8-27B-8bit model for the main agent endpoint. It won all 12 repeated quality and tool trials, handled the 65,585-token retrieval pass, and completed the two- and four-request checks. The M5 Max has enough room for its 28.85 GiB footprint without pretending the remaining memory is infinite.
Ollama stays installed and available as the rollback. That is not consolation. Its service is mature, its short-response decode is excellent, and the GGUF model is considerably lighter than oMLX 8-bit. A production endpoint should have an exit door.
I also left one tempting optimization alone. The installed oMLX build exposes a local tuner for experimental acceleration, but the benchmark access scope did not authorize it. No split was enabled, no knobs were guessed, and no experimental number was mixed into the main chart. A future tuner run can stand on its own.
The broader lesson is almost embarrassingly ordinary. Benchmark the wait, the memory, the cache, the tools, and the answer. Then choose the model that does the job. On this machine, for this agent workload, that was the slower 8-bit one.
