Qwen 3.8 27B 8-bit passed the workflow test, then overreached in code review

Experiment note: This is a qualification report for model-generated work. The recommendations were checked against frozen source snapshots before publication. They are evidence about the model, not advice to apply unreviewed.

An engine benchmark can tell me how long a model takes to answer. It cannot tell me whether I should let that model near a real workflow.

That was the point of this second test. The candidate was omlx / Qwen3.8-27B-8bit, running on an M5 Max. The earlier bake-off had already made the case for this engine and quantization on long-context work. This round asked the less glamorous question: can it follow the rules when the work looks like the jobs an agent actually gets?

The answer is a qualified yes. It was dependable on bounded extraction and structured decisions. It was much less dependable when asked to diagnose unfamiliar code.

Five passes out of six

The domain suite covered three kinds of work.

The hotel evidence case passed one out of one. The model returned enough useful claims, kept them tied to the right subject, and supplied literal quotations from the frozen text. That matters because a fluent summary is cheap; exact evidence is what makes the result reusable.

The paper portfolio cases passed three out of three. It respected a position cap, sold the requested number of shares at the supplied mark, and produced a hold decision with empty trade fields. These were pretend-money tests, but they exercised the same habits needed by any structured controller: use the given values, preserve the schema, and do the arithmetic before acting.

The casual sports replies pass two out of two under scorer v2. The originally published v1 score counted one reply as a failure because it did not literally include both Bears and Chicago, even though the prompt asked for a Bears-fan persona rather than those exact words. Cloud controls later produced unmistakable Bears-fan replies that failed the same hidden keyword gate. That calibration exposed a benchmark defect, not a shared inability to follow the request.

The versioned v2 scorer also normalizes action capitalization before checking the paper-trading enum, so semantically correct HOLD and hold decisions are treated alike. Original receipts and v1 scores remain preserved rather than silently rewritten.

The corrected domain result is 6/6, or 100 percent: hotel evidence, paper decisions, and casual fan replies all passed. The original published v1 result was 5/6.

The code-review problem

Four repository reviews all returned parseable JSON and 21 recommendations. The mechanical validator accepted 19 findings and passed two of the four reviews outright. That sounds good until the recommendations are checked for meaning rather than quotation shape.

After independent review, nine recommendations were plainly useful. Four more had some merit but overstated severity or assumed a threat model the source did not establish. Eight were false positives or materially wrong.

The strongest findings were practical: a deployment workflow that skipped the build, state that disappeared on refresh, committed runtime artifacts, repeated database commits inside a loop, missing tests, and an authenticated proxy endpoint that could still benefit from rate limiting. Those are the sorts of things a first-pass reviewer should notice.

The failures were more revealing. One review called a dependency problem a critical build blocker, yet the pinned application built successfully. Another claimed a duplicate configuration disabled static export, while the build completed and exported the site. The sharpest miss labeled an image fetch as an SSRF vulnerability even though the called helper explicitly rejected private, loopback, link-local, reserved, multicast, and unspecified addresses and rechecked every redirect. A separate finding warned that a migration could drop columns after a failed backfill, overlooking the surrounding transaction that would roll the operation back.

This is exactly why quote-valid is not the same as correct. A model can point at a real line, quote it perfectly, and still misunderstand the function around it.

Time and token cost

The six domain calls used 1,899 tokens and took 60.47 seconds in total. The median domain call was 4.60 seconds; the longer evidence extraction carried most of the wait.

The four repository reviews were a different class of job. They consumed 95,914 tokens and took 676.24 seconds, with a median latency of 167.65 seconds per review. Across the full qualification run, the model processed 97,813 tokens in 736.71 seconds, a little over twelve minutes.

The token split is the useful part. Small, constrained jobs were cheap and reliable. Broad repository review consumed nearly all the context and nearly all the time, then produced enough confident mistakes to require line-by-line adjudication anyway.

Provisional verdict: likely keep

omlx / Qwen3.8-27B-8bit remains a likely keeper.

I would use it for evidence extraction, bounded structured decisions, and first-pass issue discovery when deterministic validators and a human or stronger reviewer sit behind it. I would not let it assign security severity, certify a migration, or approve a code change on its own.

That is not a demotion. A model that reliably handles narrow work and occasionally surfaces a good lead can earn plenty of disk space. The mistake would be promoting its confidence along with its output.

No model is being deleted from this result. Retention stays provisional until the rest of the campaign supplies direct engine and family comparisons.