Punishing Four Local Models With a 10,000-Word Great Lakes Feature

Four local-model configurations received the same frozen evidence dossier and the same deliberately punishing assignment: produce a single 9,000–11,000-word WordPress article about Great Lakes shipwrecks, lighthouses, and maritime disasters, then apply twelve revisions distributed across the beginning, middle, and end. The test was designed to expose failures that short prompts conceal—truncation, forgotten constraints, repetition, unsupported elaboration, factual drift, malformed HTML, and revisions that touch only the easy parts.

Bottom line: no candidate produced a release-safe article. Qwen3.8 Flash Next via Sushi was the only model to satisfy the length contract, but it reached that length partly through repetition and scored last in blind prose review. Ornith 1.5 35B wrote the best prose and had the strongest factual result, but stopped well short of the requested length. Gemma 4 31B followed all twelve revision instructions, yet it was both short and comparatively slow. Qwen3.8 27B MLX failed twice at runtime and produced no recoverable article.

The controlled setup

Each model received the same source-bound dossier, sixteen required sections, a single thesis to carry from introduction to conclusion, explicit factual traps, a 9,000–11,000-word target, a 16,000-token output ceiling, temperature 0.35, and a 65,536-token context. The prompt prohibited invented firsthand experience and required valid WordPress HTML, a complete source list, explicit treatment of conflicting casualty estimates, and a real ending rather than silent truncation.

The second pass supplied the complete original article plus twelve precise edits. Revision was scored independently from first-pass generation. After deterministic checks, one reviewer compared every article against the frozen dossier and another reviewer ranked the prose blindly with model disclosures removed.

Qwen3.8 27B MLX: no article

The dense 27B model failed twice on the identical prompt. The first attempt ran for 262.133 seconds before Ollama’s MLX speculative-decoding cache failed. A changed-condition retry disabled drafting; it ran for 317.351 seconds and then hit a Metal out-of-memory failure. Neither attempt returned a recoverable article.

This is not a prose judgment because there was no prose to judge. It is still a decisive benchmark result: this endpoint and quantization were not reliable for the requested long-context workload on the 64GB machine.

Read the preserved Qwen3.8 27B MLX failure artifact.

Gemma 4 31B MLX: compliant editor, incomplete long-form writer

• Initial article: 5,974 words in 1,107.995 seconds
• Revised article: 6,337 words in 196.180 seconds
• Length contract: failed twice
• Revision instructions: 12 of 12
• Independent factual score: 6.0/10; not release-safe
• Blind style score: 7.5/10; second of three

Gemma’s most impressive result was revision obedience. It completed all twelve requested edits, including changes placed late in the article. But it added only 363 words and stopped normally with output budget remaining, leaving the article more than 2,600 words below the minimum. This was an instruction-following failure, not truncation.

The reviewer found the core dates, locations, casualty figures, and cross-disaster distinctions mostly sound. Release was blocked by unsupported additions, including a shipworm explanation absent from the dossier, causal claims about steam-era risk-taking, and invented keeper and rescue scenes. The initial and revised source lists also omitted one required NOAA source.

Read Gemma’s preserved, clearly warned benchmark artifact.

Ornith 1.5 35B: the best writer here, but not an endurance winner

• Initial article: 6,421 words in 94.002 seconds
• Revised article: 7,220 words in 107.495 seconds
• Length contract: failed twice
• Revision instructions: 9 of 12
• Independent factual score: 7.5/10; not release-safe
• Blind style score: 8.5/10; first of three

Ornith was dramatically faster than Gemma while producing the strongest prose. The blind reviewer found a controlled voice, varied pacing, concrete analytical reasoning, and a thesis that genuinely returned in the conclusion. It also earned the best factual score of the three completed candidates.

It nevertheless failed the central endurance requirement. Revision added 799 words, but the final result remained 1,780 words short. It also missed three requested edits, duplicated an archaeological-context analogy, and introduced an obvious era error by describing the 1913 storm as testing “mid-nineteenth-century” forecasting. Other unsupported additions included technical meteorology, a free-surface explanation for Eastland, and broad claims about fleet composition.

Read Ornith’s preserved, clearly warned benchmark artifact.

Qwen3.8 Flash Next Sushi 3bpw: the endurance winner and editorial loser

• Initial article: 10,139 words in 243.658 seconds
• Revised article: 10,787 words in 166.262 seconds
• Length contract: passed twice
• Revision instructions: 11 of 12
• Independent factual score: 5.5/10; not release-safe
• Blind style score: 4.5/10; third of three

Flash Next was the only model that obeyed the 9,000–11,000-word contract, both before and after revision. That matters: it demonstrated substantially better long-output persistence and broad instruction retention than the other local candidates.

But word count was not quality. The blind reviewer found severe template repetition, monotone paragraph construction, instruction leakage, and lower information density despite the longer output. Phrases such as “In the context of the Great Lakes” and generic claims about turning points, resilience, and ongoing risk management recurred throughout. Deterministic validation also found repeated substantial paragraphs.

The factual review found correct core disaster facts but multiple unsupported additions: an exact Coast Guard date despite an explicit prompt warning, a claimed Eastland listing history, meteorological specifics, named Fitzgerald hypotheses, and an unsupported protected-site claim. The model succeeded at endurance while failing the standard required for unsupervised publication.

Read Flash Next’s preserved, clearly warned benchmark artifact.

The actual ranking

1. Best raw prose and factual discipline: Ornith 1.5 35B. It produced the most promising editorial material, but not the requested length and not a publishable final article.
2. Best revision obedience: Gemma 4 31B MLX. It executed all twelve edits, but its first pass was slow and both outputs were far too short.
3. Best endurance and length compliance: Qwen3.8 Flash Next Sushi 3bpw. It stayed within the requested range, but repetition and unsupported detail made the extra length a liability.
4. Least usable configuration: Qwen3.8 27B MLX. Two different runtime failures prevented evaluation.

There is no honest single “winner.” Ornith won the writing comparison. Flash Next won the endurance test. Gemma won the revision checklist. None won the release-safety gate.

What this says about the hardware decision

This test does not justify buying a larger Mac for writing quality. The 64GB machine already ran the most ambitious candidate—the 125B-class Flash Next model compressed to Sushi 3bpw—and that model’s decisive weakness was editorial quality, not inability to generate enough text. More memory could permit a larger or less compressed representation, but this benchmark does not establish that doing so would repair repetitive structure, unsupported elaboration, or weak factual restraint.

A 96GB Ultra is especially hard to justify from these results: it would add expense without a demonstrated path to a materially better editorial outcome. A 128GB Max is the meaningful capacity upgrade if a future workload specifically requires larger weights, but this test supplies no evidence that buying it is the best way to improve research writing.

The practical answer is to keep the 64GB Max for now. Use Ornith when the job values prose and an editor can manage length. Use Flash Next when instruction persistence or long structured output matters, but keep a strict repetition and factual gate in front of publication. Use selective cloud review or cloud final editing for fact-sensitive public work rather than expecting RAM alone to turn any of these quantized local models into an unsupervised magazine writer.

What the punishment revealed

The central lesson is that long generation and good long-form writing are different capabilities. Flash Next could keep going. Ornith knew how to shape an argument. Gemma could apply a distributed edit list. None consistently combined endurance, factual restraint, structural variety, and release-ready prose.

That is useful evidence, not a disappointing result. A 5,000-word production workflow built from evidence mapping, outlining, section drafting, continuity editing, factual review, and final assembly is still the more realistic test of an agent system. This endurance round isolates model limits; it should not be mistaken for the best way to produce an article.

Verdict: keep the 64GB Mac, do not buy the 96GB Ultra for this workload, and do not buy the 128GB Max unless a separate production-style test shows a concrete model that improves the final article enough to justify it. For today’s evidence, selective cloud editorial help is the cheaper and safer quality upgrade.