Local Hotel Source Selection: 28-Model Benchmark
.ob-table{border-collapse:collapse;width:100%;font-size:14px}.ob-table th,.ob-table td{border:1px solid #d8dee4;padding:7px;vertical-align:top}.ob-table th{background:#f6f8fa;text-align:left}.ok{background:#e9f8ee}.warn{background:#fff8dc}.bad{background:#fdecec}.mono{font-family:ui-monospace,SFMono-Regular,Menlo,monospace}.note{border-left:4px solid #4b74c6;padding:10px 14px;background:#f5f8ff}.metric{display:inline-block;padding:7px 10px;margin:3px;background:#f2f4f7;border-radius:5px}pre{white-space:pre-wrap;overflow-wrap:anywhere;font-size:12px}details{margin:10px 0}.small{font-size:13px;color:#4b5563}
Scope: This is a source-selection benchmark, not an end-to-end browsing benchmark. Every model received the same frozen 16-result candidate set produced by deterministic search and had to choose the property-specific official and guest-review corpus. That isolates the judgment task a reliable acquisition controller should delegate to a model; it does not claim these models can operate a browser reliably.
We tested all 28 installed general-purpose text-model tags available on the Mac mini and the remote worker: 19 local and 9 remote. Coding, vision, and embedding models were excluded. Each physical host ran one model at a time; local and remote queues ran concurrently.
Result grid
| Model | Machine | Status | Score / 100 | Wall seconds | What it found |
|---|---|---|---|---|---|
| gemma4:e4b | remote-10.7.6.39 | accepted | 100.00 | 14.10 | Exact expected corpus |
| qwen3:8b | local-mac-mini | accepted | 100.00 | 22.60 | Exact expected corpus |
| muse-glimmer:30b | remote-10.7.6.39 | accepted | 100.00 | 23.45 | Exact expected corpus |
| qwen3.8:latest | remote-10.7.6.39 | accepted | 100.00 | 25.69 | Exact expected corpus |
| edtorre/muse-glimmer-agent:latest | remote-10.7.6.39 | accepted | 100.00 | 26.21 | Exact expected corpus |
| qwen3.6:latest | remote-10.7.6.39 | accepted | 100.00 | 27.54 | Exact expected corpus |
| qwen3.5:35b | remote-10.7.6.39 | accepted | 100.00 | 28.31 | Exact expected corpus |
| qwen3:14b | local-mac-mini | accepted | 100.00 | 47.28 | Exact expected corpus |
| gemma3:12b | local-mac-mini | accepted | 100.00 | 47.58 | Exact expected corpus |
| gemma4:31b | remote-10.7.6.39 | accepted | 100.00 | 59.84 | Exact expected corpus |
| mistral-small:24b | local-mac-mini | accepted | 100.00 | 72.18 | Exact expected corpus |
| qwen3.6:latest | local-mac-mini | accepted | 100.00 | 91.95 | Exact expected corpus |
| qwen3.6:27b | local-mac-mini | accepted | 100.00 | 105.83 | Exact expected corpus |
| qwen3.5:9b | remote-10.7.6.39 | accepted | 95.00 | 11.81 | missed C04 |
| qwen3.5:9b-64k | local-mac-mini | accepted | 95.00 | 31.63 | missed C04 |
| qwen3.5:9b | local-mac-mini | accepted | 95.00 | 33.54 | missed C04 |
| mo-shakib/clearwriter:latest | local-mac-mini | accepted | 94.86 | 16.38 | missed C06 |
| nemotron-3.5-lightning:30b | remote-10.7.6.39 | accepted | 94.00 | 36.50 | missed C06, C14 |
| nemotron-3.5-lightning:30b-mlx | local-mac-mini | request_error | 0.00 | 0.04 | Request failed |
| muse-glimmer:30b-mlx | local-mac-mini | request_error | 0.00 | 0.06 | Request failed |
| gemma4:e2b-mlx | local-mac-mini | invalid_output | 0.00 | 5.48 | Wrong output contract |
| gemma4:e4b-mlx | local-mac-mini | invalid_output | 0.00 | 10.23 | Wrong output contract |
| gemma4:26b-16k | local-mac-mini | invalid_output | 0.00 | 19.58 | Wrong output contract |
| gemma4:26b-mlx | local-mac-mini | invalid_output | 0.00 | 19.92 | Wrong output contract |
| gpt-oss:20b | local-mac-mini | invalid_output | 0.00 | 20.47 | Wrong output contract |
| gemma4:12b-mlx | local-mac-mini | invalid_output | 0.00 | 41.60 | Wrong output contract |
| gemma4:12b-mlx-chat | local-mac-mini | invalid_output | 0.00 | 44.01 | Wrong output contract |
| qwen3.8:27b-mlx | local-mac-mini | invalid_output | 0.00 | 62.28 | Wrong output contract |
Headline findings
- 13 models produced the exact expected corpus.
- The fastest exact result overall was gemma4:e4b on remote-10.7.6.39 at 14.10 seconds.
- The fastest exact local result was qwen3:8b at 22.60 seconds.
- The current qwen3.5:9b family correctly chose the canonical page and all five guest sources, but consistently omitted official location page C04, scoring 95.
- Several Gemma4 MLX tags returned sensible-looking selections under their own invented schemas. They are scored zero because downstream automation needs the agreed contract, not merely readable prose.
- The remote Nemotron result was schema-valid but selected one non-preferred source; the two local 30B MLX requests failed at the Ollama server before producing output.
Same-model local versus remote speed
We verified identical Ollama digests for qwen3.5:9b and qwen3.6:latest, then ran three paired, simultaneous, cold trials per model with keep_alive=0. Each host ran only one model at a time.
| Exact model | Local median | Remote median | Remote speedup | Local samples | Remote samples |
|---|---|---|---|---|---|
| qwen3.5:9b | 31.57 | 10.53 | 3.00× | 33.66, 31.57, 31.55 | 12.62, 10.51, 10.53 |
| qwen3.6:latest | 119.71 | 29.17 | 4.10× | 94.36, 122.21, 119.71 | 31.94, 29.17, 27.67 |
These are observed end-to-end wall times for this prompt, including cold loading. They are not a general hardware benchmark.
What this changes
A larger model is not required for the bounded selection decision. The 8B local Qwen produced the exact corpus in 22.60 seconds, while the existing 9B Qwen missed one useful official page. The correct process remains deterministic search and rendering plus a narrow model selection call. The benchmark does not rehabilitate the original broad 15-turn web agent, which mixed searching, URL guessing, rendering, budgeting, and artifact writing.
