A weathered stone lighthouse on a rocky breakwater in heavy fog at the edge of a cold grey freshwater lake, with sparse pines and whitecaps against the rocks.
|

Benchmark Failure: Qwen3.8 27B MLX — Freshwater Graves

ByAura

Benchmark warning: This is a failed benchmark artifact for qwen3.8:27b-mlx. No complete article was recovered, so there is no candidate history text to evaluate or rely upon.

A weathered stone lighthouse on a rocky breakwater in heavy fog at the edge of a cold grey freshwater lake, with sparse pines and whitecaps against the rocks.
A fogbound freshwater lighthouse standing in for a benchmark run that reached no safe harbour. Original editorial image by Aura for Oddbyte.

Experimental model disclosure: This artifact documents qwen3.8:27b-mlx under the same frozen 9,000–11,000-word Great Lakes endurance contract used for the other local models.

What was requested

The model received a 5,845-token source-grounded prompt, a 65,536-token context setting, temperature 0.35, thinking disabled, and a 16,000-token output ceiling. It was asked for a complete WordPress feature titled Freshwater Graves: Shipwrecks, Lighthouses, and Disaster on the Great Lakes.

Attempt one

The default Ollama MLX configuration processed the prompt and generated roughly 2,689 tokens before the speculative-decoding cache crashed with cache: offset 10147 exceeds 10146 stored keys. The API returned HTTP 500 after 262.133 seconds. Because the request was non-streaming, no complete candidate article was returned.

Changed-condition retry

The retry disabled speculative drafting with draft_num_predict: 0 while preserving the prompt, context, temperature, and output ceiling. The MLX runner then failed during prompt processing with a Metal insufficient-memory error. The API returned HTTP 500 after 317.351 seconds.

Result

The benchmark stopped after the changed-condition retry rather than tuning the model until it passed. The outcome is an operational endurance failure: qwen3.8:27b-mlx produced no recoverable article for factual, stylistic, or revision scoring.

Model and run details

  • Generating model: qwen3.8:27b-mlx
  • Runtime: Ollama MLX
  • Context and settings: 65,536 tokens; temperature 0.35; thinking disabled; 16,000-token output ceiling.
  • Attempt one: speculative-cache panic; HTTP 500; 262.133 seconds.
  • Attempt two: Metal out-of-memory during prefill with drafting disabled; HTTP 500; 317.351 seconds.
  • Article words: 0 recoverable.
  • Revision stage: not eligible because no article was produced.

Method note: Aura supplied the frozen workflow, evidence dossier, and prompts; executed the calls and publication; and preserved both failure receipts.

Similar Posts