Hotel Source Selection Model Benchmark: Full Results

ByAURA

.ob-table{border-collapse:collapse;width:100%;font-size:14px}.ob-table th,.ob-table td{border:1px solid #d8dee4;padding:7px;vertical-align:top}.ob-table th{background:#f6f8fa;text-align:left}.ok{background:#e9f8ee}.warn{background:#fff8dc}.bad{background:#fdecec}.mono{font-family:ui-monospace,SFMono-Regular,Menlo,monospace}.note{border-left:4px solid #4b74c6;padding:10px 14px;background:#f5f8ff}.metric{display:inline-block;padding:7px 10px;margin:3px;background:#f2f4f7;border-radius:5px}pre{white-space:pre-wrap;overflow-wrap:anywhere;font-size:12px}details{margin:10px 0}.small{font-size:13px;color:#4b5563}

← Back to the summary grid

Scope warning: These models did not browse the web. They selected sources from the same frozen candidate catalog. This report measures identity resolution, source-quality judgment, contract compliance, and speed—the model-owned portion of the proposed deterministic source-acquisition controller.

Method

  • One hotel: DoubleTree by Hilton Hotel Chicago – North Shore Conference Center in Skokie.
  • One frozen set of 16 deterministic search candidates.
  • Temperature 0, seed 42, 8,192-token context, 1,800-token output limit, structured JSON requested.
  • Cold execution with keep_alive=0.
  • One sequential model queue per physical host; the two hosts ran concurrently.
  • No manual corrections and no retry of malformed outputs.

Expected corpus and score

{
  "canonical_official_id": "C01",
  "supplemental_official_ids": [
    "C02",
    "C03",
    "C04"
  ],
  "independent_or_guest_ids": [
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejected_ids": [
    "C05",
    "C07",
    "C10",
    "C11",
    "C13",
    "C15",
    "C16"
  ]
}

Scoring: schema 10; canonical official page 20; supplemental-official recall 15; guest-source precision 15; guest-source recall 15; rejection recall 15; correct ready status 10; invented IDs incur a 20-point penalty. Wrong schemas score zero because they cannot be consumed safely.

Candidate catalog

ID Publisher Title URL Search-result description
C01 Hilton DoubleTree by Hilton Hotel Chicago – North Shore Conference Center source Official property result for the named Skokie hotel; modern Skokie hotel near O'Hare with rooftop ballroom.
C02 Hilton Doubletree Hotel in Skokie IL | North Shore Conference Hotel source Property-specific hotel information and amenities including on-site dining, event space, and services.
C03 Hilton Skokie IL Hotel Rooms | DoubleTree Chicago North Shore source Property-specific rooms page for the Skokie DoubleTree.
C04 Hilton Chicago North Shore Hotels | DoubleTree Chicago – Skokie, IL source Property-specific location page for the DoubleTree in Skokie.
C05 Hilton DoubleTree Hotels in Chicago, IL – Find Hotels – Hilton source Generic regional index listing multiple DoubleTree properties around Chicago.
C06 Yelp DoubleTree by Hilton Chicago – North Shore Conference Center – Reviews source Property-specific business page with guest reviews and the 9599 Skokie Boulevard address.
C07 Instagram DoubleTree Chicago North Shore Conference Center on Instagram source Property social-media account; promotional posts rather than an independent hotel-review corpus.
C08 Tripadvisor DoubleTree by Hilton Hotel Chicago – North Shore Conference Center – Reviews source Property-level page with approximately 1,540 traveler reviews and photos.
C09 Booking.com Verified Hotel Reviews of DoubleTree Chicago North Shore source Property-specific page containing hundreds of verified guest reviews.
C10 Booking.com DoubleTree Chicago North Shore – Booking Page source Property booking and availability page; a review-specific page is separately available as C09.
C11 Tripadvisor One individual traveler review of DoubleTree Chicago North Shore source A single old user-review permalink; the property-level Tripadvisor corpus is C08.
C12 Expedia DoubleTree Chicago North Shore Reviews, Deals and Photos source Property-level listing with guest-review material and current property overview.
C13 Expedia-AARP DoubleTree Chicago North Shore Hotel Reviews source Syndicated Expedia-family mirror of the same property, duplicating C12.
C14 Hotels.com DoubleTree Chicago North Shore Hotel Reviews, Rooms and Prices source Property-specific page with traveler reviews and property information.
C15 WeddingWire DoubleTree Chicago North Shore Wedding Venue Reviews source Event and wedding-venue reviews, not a general hotel-stay review corpus.
C16 HotelPlanner DoubleTree Chicago North Shore Group Booking Page source Third-party group-booking page with thin review material and booking emphasis.
Exact benchmark prompt 
You are selecting a frozen hotel source corpus from deterministic search results. Use only candidate IDs listed below. Do not browse, invent URLs, extract hotel facts, or write prose outside the required JSON.

SUBJECT
subject_id: doubletree-chicago-north-shore-skokie
canonical_name: DoubleTree by Hilton Hotel Chicago - North Shore Conference Center
location: 9599 Skokie Boulevard, Skokie, Illinois 60077-1314
official_domain_hint: hilton.com

CONTRACT
- Select exactly one canonical official property page.
- Select no more than 3 supplemental official property-specific pages.
- Select 5 independent or guest-review property-level pages when credible.
- At least 3 credible independent or guest-review pages are required for ready_for_archive.
- Prefer a property-level review corpus over an individual review permalink.
- Reject generic regional indexes, social-media pages, duplicate/syndicated mirrors, booking-only duplicates when a review-specific page exists, event-only reviews, and thin booking pages.
- Include every unselected candidate in rejections with a concise reason.
- Every candidate ID must appear exactly once across selected fields or rejections.
- Return strict JSON matching the supplied schema.

CANDIDATES
C01 | Hilton | DoubleTree by Hilton Hotel Chicago - North Shore Conference Center | https://www.hilton.com/en/hotels/chiccdt-doubletree-chicago-north-shore-conference-center/ | Official property result for the named Skokie hotel; modern Skokie hotel near O'Hare with rooftop ballroom.
C02 | Hilton | Doubletree Hotel in Skokie IL | North Shore Conference Hotel | https://www.hilton.com/en/hotels/chiccdt-doubletree-chicago-north-shore-conference-center/hotel-info/ | Property-specific hotel information and amenities including on-site dining, event space, and services.
C03 | Hilton | Skokie IL Hotel Rooms | DoubleTree Chicago North Shore | https://www.hilton.com/en/hotels/chiccdt-doubletree-chicago-north-shore-conference-center/rooms/ | Property-specific rooms page for the Skokie DoubleTree.
C04 | Hilton | Chicago North Shore Hotels | DoubleTree Chicago - Skokie, IL | https://www.hilton.com/en/hotels/chiccdt-doubletree-chicago-north-shore-conference-center/hotel-location/ | Property-specific location page for the DoubleTree in Skokie.
C05 | Hilton | DoubleTree Hotels in Chicago, IL - Find Hotels - Hilton | https://www.hilton.com/en/locations/usa/illinois/chicago/doubletree-by-hilton/ | Generic regional index listing multiple DoubleTree properties around Chicago.
C06 | Yelp | DoubleTree by Hilton Chicago - North Shore Conference Center - Reviews | https://www.yelp.com/biz/doubletree-by-hilton-chicago-north-shore-conference-center-skokie | Property-specific business page with guest reviews and the 9599 Skokie Boulevard address.
C07 | Instagram | DoubleTree Chicago North Shore Conference Center on Instagram | https://www.instagram.com/doubletree_chicagonorthshore/ | Property social-media account; promotional posts rather than an independent hotel-review corpus.
C08 | Tripadvisor | DoubleTree by Hilton Hotel Chicago - North Shore Conference Center - Reviews | https://www.tripadvisor.com/Hotel_Review-g36707-d217509-Reviews-Doubletree_By_Hilton_Hotel_Chicago_North_Shore_Conference_Center-Skokie_Illinois.html | Property-level page with approximately 1,540 traveler reviews and photos.
C09 | Booking.com | Verified Hotel Reviews of DoubleTree Chicago North Shore | https://www.booking.com/reviews/us/hotel/doubletree-and-conference-center-chicago-north-shore.html | Property-specific page containing hundreds of verified guest reviews.
C10 | Booking.com | DoubleTree Chicago North Shore - Booking Page | https://www.booking.com/hotel/us/doubletree-and-conference-center-chicago-north-shore.html | Property booking and availability page; a review-specific page is separately available as C09.
C11 | Tripadvisor | One individual traveler review of DoubleTree Chicago North Shore | https://www.tripadvisor.com/ShowUserReviews-g36707-d217509-r279651664-DoubleTree_by_Hilton_Chicago_North_Shore-Skokie_Illinois.html | A single old user-review permalink; the property-level Tripadvisor corpus is C08.
C12 | Expedia | DoubleTree Chicago North Shore Reviews, Deals and Photos | https://www.expedia.com/Chicago-Hotels-DoubleTree-By-Hilton-Chicago-North-Shore-Conference-Center.h15560.Hotel-Information | Property-level listing with guest-review material and current property overview.
C13 | Expedia-AARP | DoubleTree Chicago North Shore Hotel Reviews | https://www.expedia-aarp.com/Skokie-Hotels-DoubleTree-By-Hilton-Chicago-North-Shore-Conference-Center.h15560.Hotel-Reviews | Syndicated Expedia-family mirror of the same property, duplicating C12.
C14 | Hotels.com | DoubleTree Chicago North Shore Hotel Reviews, Rooms and Prices | https://www.hotels.com/ho113070/doubletree-by-hilton-chicago-north-shore-conference-center-skokie-united-states-of-america/ | Property-specific page with traveler reviews and property information.
C15 | WeddingWire | DoubleTree Chicago North Shore Wedding Venue Reviews | https://www.weddingwire.com/reviews/doubletree-by-hilton-hotel-chicago-north-shore-conference-center-skokie/a4841b01d861d086.html | Event and wedding-venue reviews, not a general hotel-stay review corpus.
C16 | HotelPlanner | DoubleTree Chicago North Shore Group Booking Page | https://www.hotelplanner.com/Hotels/129052/Reservations-Doubletree-by-Hilton-Chicago-North-Shore-Skokie-9599-Skokie-Blvd-60077 | Third-party group-booking page with thin review material and booking emphasis.

Paired speed trials

Exact model Local median Remote median Remote speedup Local samples Remote samples
qwen3.5:9b 31.57 10.53 3.00× 33.66, 31.57, 31.55 12.62, 10.51, 10.53
qwen3.6:latest 119.71 29.17 4.10× 94.36, 122.21, 119.71 31.94, 29.17, 27.67

Per-model findings and raw output

1. gemma4:e4b — remote-10.7.6.39

Status accepted Score 100.00
Wall 14.10 s Ollama total 14.04 s
Load 9.27 s Tokens 1489 in / 422 out
Assessment Exact expected corpus
Exact model output 
{
  "canonical_official_id": "C01",
  "supplemental_official_ids": [
    "C02",
    "C03",
    "C04"
  ],
  "independent_or_guest_ids": [
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": [
    {
      "candidate_id": "C05",
      "reason": "Generic regional index listing multiple DoubleTree properties around Chicago."
    },
    {
      "candidate_id": "C07",
      "reason": "Property social-media account; promotional posts rather than an independent hotel-review corpus."
    },
    {
      "candidate_id": "C10",
      "reason": "Property booking and availability page; a review-specific page is separately available as C09."
    },
    {
      "candidate_id": "C11",
      "reason": "A single old user-review permalink; the property-level Tripadvisor corpus is C08."
    },
    {
      "candidate_id": "C13",
      "reason": "Syndicated Expedia-family mirror of the same property, duplicating C12."
    },
    {
      "candidate_id": "C15",
      "reason": "Event and wedding-venue reviews, not a general hotel-stay review corpus."
    },
    {
      "candidate_id": "C16",
      "reason": "Third-party group-booking page with thin review material and booking emphasis."
    }
  ],
  "corpus_status": "ready_for_archive",
  "confidence": 1.0
}
Deterministic scoring receipt 
{
  "score": 100.0,
  "schema_valid": true,
  "invented_ids": [],
  "canonical_correct": true,
  "components": {
    "schema": 10.0,
    "canonical": 20.0,
    "supplemental_recall": 15.0,
    "guest_precision": 15.0,
    "guest_recall": 15.0,
    "rejection_recall": 15.0,
    "status": 10.0
  },
  "penalty": 0.0,
  "selected": {
    "canonical_official_id": "C01",
    "supplemental_official_ids": [
      "C02",
      "C03",
      "C04"
    ],
    "independent_or_guest_ids": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ],
    "rejected_ids": [
      "C05",
      "C07",
      "C10",
      "C11",
      "C13",
      "C15",
      "C16"
    ]
  }
}

2. qwen3:8b — local-mac-mini

Status accepted Score 100.00
Wall 22.60 s Ollama total 22.58 s
Load 3.47 s Tokens 1353 in / 251 out
Assessment Exact expected corpus
Exact model output 
{
  "canonical_official_id": "C01",
  "supplemental_official_ids": [
    "C02",
    "C03",
    "C04"
  ],
  "independent_or_guest_ids": [
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": [
    {
      "candidate_id": "C05",
      "reason": "Generic regional index listing multiple properties"
    },
    {
      "candidate_id": "C07",
      "reason": "Social-media account with promotional content"
    },
    {
      "candidate_id": "C10",
      "reason": "Booking page without review-specific content"
    },
    {
      "candidate_id": "C11",
      "reason": "Single individual review permalink"
    },
    {
      "candidate_id": "C13",
      "reason": "Syndicated mirror of C12"
    },
    {
      "candidate_id": "C15",
      "reason": "Event-specific reviews, not general hotel reviews"
    },
    {
      "candidate_id": "C16",
      "reason": "Thin review material and group-booking focus"
    }
  ],
  "corpus_status": "ready_for_archive",
  "confidence": 1
}
Deterministic scoring receipt 
{
  "score": 100.0,
  "schema_valid": true,
  "invented_ids": [],
  "canonical_correct": true,
  "components": {
    "schema": 10.0,
    "canonical": 20.0,
    "supplemental_recall": 15.0,
    "guest_precision": 15.0,
    "guest_recall": 15.0,
    "rejection_recall": 15.0,
    "status": 10.0
  },
  "penalty": 0.0,
  "selected": {
    "canonical_official_id": "C01",
    "supplemental_official_ids": [
      "C02",
      "C03",
      "C04"
    ],
    "independent_or_guest_ids": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ],
    "rejected_ids": [
      "C05",
      "C07",
      "C10",
      "C11",
      "C13",
      "C15",
      "C16"
    ]
  }
}

3. muse-glimmer:30b — remote-10.7.6.39

Status accepted Score 100.00
Wall 23.45 s Ollama total 23.42 s
Load 13.89 s Tokens 1343 in / 237 out
Assessment Exact expected corpus
Exact model output 
{
  "canonical_official_id": "C01",
  "supplemental_official_ids": [
    "C02",
    "C03",
    "C04"
  ],
  "independent_or_guest_ids": [
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": [
    {
      "candidate_id": "C05",
      "reason": "Generic regional index listing multiple DoubleTree properties around Chicago."
    },
    {
      "candidate_id": "C07",
      "reason": "Property social-media account; promotional posts rather than an independent hotel-review corpus."
    },
    {
      "candidate_id": "C10",
      "reason": "Property booking and availability page; a review-specific page is separately available as C09."
    },
    {
      "candidate_id": "C11",
      "reason": "A single old user-review permalink; the property-level Tripadvisor corpus is C08."
    },
    {
      "candidate_id": "C13",
      "reason": "Syndicated Expedia-family mirror of the same property, duplicating C12."
    },
    {
      "candidate_id": "C15",
      "reason": "Event and wedding-venue reviews, not a general hotel-stay review corpus."
    },
    {
      "candidate_id": "C16",
      "reason": "Third-party group-booking page with thin review material and booking emphasis."
    }
  ],
  "corpus_status": "ready_for_archive",
  "confidence": 0.9
}
Deterministic scoring receipt 
{
  "score": 100.0,
  "schema_valid": true,
  "invented_ids": [],
  "canonical_correct": true,
  "components": {
    "schema": 10.0,
    "canonical": 20.0,
    "supplemental_recall": 15.0,
    "guest_precision": 15.0,
    "guest_recall": 15.0,
    "rejection_recall": 15.0,
    "status": 10.0
  },
  "penalty": 0.0,
  "selected": {
    "canonical_official_id": "C01",
    "supplemental_official_ids": [
      "C02",
      "C03",
      "C04"
    ],
    "independent_or_guest_ids": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ],
    "rejected_ids": [
      "C05",
      "C07",
      "C10",
      "C11",
      "C13",
      "C15",
      "C16"
    ]
  }
}

4. qwen3.8:latest — remote-10.7.6.39

Status accepted Score 100.00
Wall 25.69 s Ollama total 25.67 s
Load 15.66 s Tokens 1367 in / 401 out
Assessment Exact expected corpus
Exact model output 
{
  "canonical_official_id": "C01",
  "supplemental_official_ids": [
    "C02",
    "C03",
    "C04"
  ],
  "independent_or_guest_ids": [
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": [
    {
      "candidate_id": "C05",
      "reason": "Generic regional index listing multiple properties, not property-specific."
    },
    {
      "candidate_id": "C07",
      "reason": "Social-media page; promotional content rather than an independent review corpus."
    },
    {
      "candidate_id": "C10",
      "reason": "Booking-only page; a review-specific page (C09) exists for the same property."
    },
    {
      "candidate_id": "C11",
      "reason": "Individual review permalink; property-level corpus (C08) is preferred."
    },
    {
      "candidate_id": "C13",
      "reason": "Syndicated mirror of C12 (Expedia-AARP), duplicating the same property data."
    },
    {
      "candidate_id": "C15",
      "reason": "Event-specific (wedding) reviews, not a general hotel-stay review corpus."
    },
    {
      "candidate_id": "C16",
      "reason": "Thin group-booking page with minimal review material."
    }
  ],
  "corpus_status": "ready_for_archive",
  "confidence": 0.98
}
Deterministic scoring receipt 
{
  "score": 100.0,
  "schema_valid": true,
  "invented_ids": [],
  "canonical_correct": true,
  "components": {
    "schema": 10.0,
    "canonical": 20.0,
    "supplemental_recall": 15.0,
    "guest_precision": 15.0,
    "guest_recall": 15.0,
    "rejection_recall": 15.0,
    "status": 10.0
  },
  "penalty": 0.0,
  "selected": {
    "canonical_official_id": "C01",
    "supplemental_official_ids": [
      "C02",
      "C03",
      "C04"
    ],
    "independent_or_guest_ids": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ],
    "rejected_ids": [
      "C05",
      "C07",
      "C10",
      "C11",
      "C13",
      "C15",
      "C16"
    ]
  }
}

5. edtorre/muse-glimmer-agent:latest — remote-10.7.6.39

Status accepted Score 100.00
Wall 26.21 s Ollama total 26.18 s
Load 15.33 s Tokens 1370 in / 272 out
Assessment Exact expected corpus
Exact model output 
{
  "canonical_official_id": "C01",
  "supplemental_official_ids": [
    "C02",
    "C03",
    "C04"
  ],
  "independent_or_guest_ids": [
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": [
    {
      "candidate_id": "C05",
      "reason": "Generic regional index listing multiple DoubleTree properties around Chicago"
    },
    {
      "candidate_id": "C07",
      "reason": "Property social-media account; promotional posts rather than an independent hotel-review corpus"
    },
    {
      "candidate_id": "C10",
      "reason": "Property booking and availability page; a review-specific page is separately available as C09"
    },
    {
      "candidate_id": "C11",
      "reason": "A single old user-review permalink; the property-level Tripadvisor corpus is C08"
    },
    {
      "candidate_id": "C13",
      "reason": "Syndicated Expedia-family mirror of the same property, duplicating C12"
    },
    {
      "candidate_id": "C15",
      "reason": "Event and wedding-venue reviews, not a general hotel-stay review corpus"
    },
    {
      "candidate_id": "C16",
      "reason": "Third-party group-booking page with thin review material and booking emphasis"
    }
  ],
  "corpus_status": "ready_for_archive",
  "confidence": 0.9
}
Deterministic scoring receipt 
{
  "score": 100.0,
  "schema_valid": true,
  "invented_ids": [],
  "canonical_correct": true,
  "components": {
    "schema": 10.0,
    "canonical": 20.0,
    "supplemental_recall": 15.0,
    "guest_precision": 15.0,
    "guest_recall": 15.0,
    "rejection_recall": 15.0,
    "status": 10.0
  },
  "penalty": 0.0,
  "selected": {
    "canonical_official_id": "C01",
    "supplemental_official_ids": [
      "C02",
      "C03",
      "C04"
    ],
    "independent_or_guest_ids": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ],
    "rejected_ids": [
      "C05",
      "C07",
      "C10",
      "C11",
      "C13",
      "C15",
      "C16"
    ]
  }
}

6. qwen3.6:latest — remote-10.7.6.39

Status accepted Score 100.00
Wall 27.54 s Ollama total 27.52 s
Load 19.18 s Tokens 1367 in / 389 out
Assessment Exact expected corpus
Exact model output 
{
  "canonical_official_id": "C01",
  "supplemental_official_ids": [
    "C02",
    "C03",
    "C04"
  ],
  "independent_or_guest_ids": [
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": [
    {
      "candidate_id": "C05",
      "reason": "Generic regional index listing multiple properties, not property-specific."
    },
    {
      "candidate_id": "C07",
      "reason": "Social-media page; promotional content rather than independent review corpus."
    },
    {
      "candidate_id": "C10",
      "reason": "Booking-only duplicate; a dedicated review page (C09) is available."
    },
    {
      "candidate_id": "C11",
      "reason": "Individual review permalink; property-level corpus (C08) is preferred."
    },
    {
      "candidate_id": "C13",
      "reason": "Syndicated mirror duplicating C12."
    },
    {
      "candidate_id": "C15",
      "reason": "Event/wedding-venue specific reviews, not general hotel-stay corpus."
    },
    {
      "candidate_id": "C16",
      "reason": "Thin review material with booking emphasis; not a credible review corpus."
    }
  ],
  "corpus_status": "ready_for_archive",
  "confidence": 0.98
}
Deterministic scoring receipt 
{
  "score": 100.0,
  "schema_valid": true,
  "invented_ids": [],
  "canonical_correct": true,
  "components": {
    "schema": 10.0,
    "canonical": 20.0,
    "supplemental_recall": 15.0,
    "guest_precision": 15.0,
    "guest_recall": 15.0,
    "rejection_recall": 15.0,
    "status": 10.0
  },
  "penalty": 0.0,
  "selected": {
    "canonical_official_id": "C01",
    "supplemental_official_ids": [
      "C02",
      "C03",
      "C04"
    ],
    "independent_or_guest_ids": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ],
    "rejected_ids": [
      "C05",
      "C07",
      "C10",
      "C11",
      "C13",
      "C15",
      "C16"
    ]
  }
}

7. qwen3.5:35b — remote-10.7.6.39

Status accepted Score 100.00
Wall 28.31 s Ollama total 28.29 s
Load 20.22 s Tokens 1367 in / 368 out
Assessment Exact expected corpus
Exact model output 
{
  "canonical_official_id": "C01",
  "supplemental_official_ids": [
    "C02",
    "C03",
    "C04"
  ],
  "independent_or_guest_ids": [
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": [
    {
      "candidate_id": "C05",
      "reason": "Generic regional index listing multiple properties, not property-specific."
    },
    {
      "candidate_id": "C07",
      "reason": "Social-media page with promotional content, not a review corpus."
    },
    {
      "candidate_id": "C10",
      "reason": "Booking-only page; a dedicated review-specific page (C09) exists."
    },
    {
      "candidate_id": "C11",
      "reason": "Individual review permalink; property-level corpus (C08) is preferred."
    },
    {
      "candidate_id": "C13",
      "reason": "Syndicated mirror duplicating C12 content."
    },
    {
      "candidate_id": "C15",
      "reason": "Event/wedding-venue specific reviews, not general hotel stay corpus."
    },
    {
      "candidate_id": "C16",
      "reason": "Third-party group booking page with thin review material and booking emphasis."
    }
  ],
  "corpus_status": "ready_for_archive",
  "confidence": 0.95
}
Deterministic scoring receipt 
{
  "score": 100.0,
  "schema_valid": true,
  "invented_ids": [],
  "canonical_correct": true,
  "components": {
    "schema": 10.0,
    "canonical": 20.0,
    "supplemental_recall": 15.0,
    "guest_precision": 15.0,
    "guest_recall": 15.0,
    "rejection_recall": 15.0,
    "status": 10.0
  },
  "penalty": 0.0,
  "selected": {
    "canonical_official_id": "C01",
    "supplemental_official_ids": [
      "C02",
      "C03",
      "C04"
    ],
    "independent_or_guest_ids": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ],
    "rejected_ids": [
      "C05",
      "C07",
      "C10",
      "C11",
      "C13",
      "C15",
      "C16"
    ]
  }
}

8. qwen3:14b — local-mac-mini

Status accepted Score 100.00
Wall 47.28 s Ollama total 47.25 s
Load 6.35 s Tokens 1353 in / 334 out
Assessment Exact expected corpus
Exact model output 
{
  "canonical_official_id": "C01",
  "supplemental_official_ids": [
    "C02",
    "C03",
    "C04"
  ],
  "independent_or_guest_ids": [
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": [
    {
      "candidate_id": "C05",
      "reason": "Generic regional index listing multiple properties; not property-specific."
    },
    {
      "candidate_id": "C07",
      "reason": "Social-media page with promotional content; not an independent hotel-review corpus."
    },
    {
      "candidate_id": "C10",
      "reason": "Booking page; a review-specific page (C09) is available and preferred."
    },
    {
      "candidate_id": "C11",
      "reason": "Single individual review permalink; the property-level corpus (C08) is preferred."
    },
    {
      "candidate_id": "C13",
      "reason": "Syndicated mirror of C12; duplicate content and not preferred."
    },
    {
      "candidate_id": "C15",
      "reason": "Event and wedding-venue reviews; not a general hotel-stay review corpus."
    },
    {
      "candidate_id": "C16",
      "reason": "Thin review material with booking emphasis; not a credible review corpus."
    }
  ],
  "corpus_status": "ready_for_archive",
  "confidence": 1
}
Deterministic scoring receipt 
{
  "score": 100.0,
  "schema_valid": true,
  "invented_ids": [],
  "canonical_correct": true,
  "components": {
    "schema": 10.0,
    "canonical": 20.0,
    "supplemental_recall": 15.0,
    "guest_precision": 15.0,
    "guest_recall": 15.0,
    "rejection_recall": 15.0,
    "status": 10.0
  },
  "penalty": 0.0,
  "selected": {
    "canonical_official_id": "C01",
    "supplemental_official_ids": [
      "C02",
      "C03",
      "C04"
    ],
    "independent_or_guest_ids": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ],
    "rejected_ids": [
      "C05",
      "C07",
      "C10",
      "C11",
      "C13",
      "C15",
      "C16"
    ]
  }
}

9. gemma3:12b — local-mac-mini

Status accepted Score 100.00
Wall 47.58 s Ollama total 47.56 s
Load 5.72 s Tokens 1491 in / 390 out
Assessment Exact expected corpus
Exact model output 
{
  "canonical_official_id": "C01",
  "supplemental_official_ids": [
    "C02",
    "C03",
    "C04"
  ],
  "independent_or_guest_ids": [
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": [
    {
      "candidate_id": "C05",
      "reason": "Generic regional index listing multiple properties."
    },
    {
      "candidate_id": "C07",
      "reason": "Property social-media account; promotional content."
    },
    {
      "candidate_id": "C10",
      "reason": "Property booking page; review-specific page exists (C09)."
    },
    {
      "candidate_id": "C11",
      "reason": "Single user-review permalink; property-level Tripadvisor corpus exists (C08)."
    },
    {
      "candidate_id": "C13",
      "reason": "Syndicated mirror of Expedia (C12)."
    },
    {
      "candidate_id": "C15",
      "reason": "Event and wedding-venue reviews, not general hotel stays."
    },
    {
      "candidate_id": "C16",
      "reason": "Third-party group-booking page with thin review material."
    }
  ],
  "corpus_status": "ready_for_archive",
  "confidence": 1.0
}
Deterministic scoring receipt 
{
  "score": 100.0,
  "schema_valid": true,
  "invented_ids": [],
  "canonical_correct": true,
  "components": {
    "schema": 10.0,
    "canonical": 20.0,
    "supplemental_recall": 15.0,
    "guest_precision": 15.0,
    "guest_recall": 15.0,
    "rejection_recall": 15.0,
    "status": 10.0
  },
  "penalty": 0.0,
  "selected": {
    "canonical_official_id": "C01",
    "supplemental_official_ids": [
      "C02",
      "C03",
      "C04"
    ],
    "independent_or_guest_ids": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ],
    "rejected_ids": [
      "C05",
      "C07",
      "C10",
      "C11",
      "C13",
      "C15",
      "C16"
    ]
  }
}

10. gemma4:31b — remote-10.7.6.39

Status accepted Score 100.00
Wall 59.84 s Ollama total 59.79 s
Load 19.29 s Tokens 1493 in / 370 out
Assessment Exact expected corpus
Exact model output 
{
  "canonical_official_id": "C01",
  "supplemental_official_ids": [
    "C02",
    "C03",
    "C04"
  ],
  "independent_or_guest_ids": [
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": [
    {
      "candidate_id": "C05",
      "reason": "Generic regional index listing multiple properties."
    },
    {
      "candidate_id": "C07",
      "reason": "Social-media page."
    },
    {
      "candidate_id": "C10",
      "reason": "Booking-only page; review-specific page C09 is preferred."
    },
    {
      "candidate_id": "C11",
      "reason": "Individual review permalink; property-level corpus C08 is preferred."
    },
    {
      "candidate_id": "C13",
      "reason": "Syndicated mirror of C12."
    },
    {
      "candidate_id": "C15",
      "reason": "Event-only reviews (wedding venue)."
    },
    {
      "candidate_id": "C16",
      "reason": "Thin booking page."
    }
  ],
  "corpus_status": "ready_for_archive",
  "confidence": 1.0
}
Deterministic scoring receipt 
{
  "score": 100.0,
  "schema_valid": true,
  "invented_ids": [],
  "canonical_correct": true,
  "components": {
    "schema": 10.0,
    "canonical": 20.0,
    "supplemental_recall": 15.0,
    "guest_precision": 15.0,
    "guest_recall": 15.0,
    "rejection_recall": 15.0,
    "status": 10.0
  },
  "penalty": 0.0,
  "selected": {
    "canonical_official_id": "C01",
    "supplemental_official_ids": [
      "C02",
      "C03",
      "C04"
    ],
    "independent_or_guest_ids": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ],
    "rejected_ids": [
      "C05",
      "C07",
      "C10",
      "C11",
      "C13",
      "C15",
      "C16"
    ]
  }
}

11. mistral-small:24b — local-mac-mini

Status accepted Score 100.00
Wall 72.18 s Ollama total 72.16 s
Load 9.03 s Tokens 1574 in / 296 out
Assessment Exact expected corpus
Exact model output 
{
  "canonical_official_id": "C01",
  "supplemental_official_ids": [
    "C02",
    "C03",
    "C04"
  ],
  "independent_or_guest_ids": [
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": [
    {
      "candidate_id": "C05",
      "reason": "Generic regional index"
    },
    {
      "candidate_id": "C07",
      "reason": "Social-media page"
    },
    {
      "candidate_id": "C10",
      "reason": "Thin booking page"
    },
    {
      "candidate_id": "C11",
      "reason": "Individual review permalink"
    },
    {
      "candidate_id": "C13",
      "reason": "Duplicate/syndicated mirror"
    },
    {
      "candidate_id": "C15",
      "reason": "Event-only reviews"
    },
    {
      "candidate_id": "C16",
      "reason": "Thin booking page"
    }
  ],
  "corpus_status": "ready_for_archive",
  "confidence": 1.0
}
Deterministic scoring receipt 
{
  "score": 100.0,
  "schema_valid": true,
  "invented_ids": [],
  "canonical_correct": true,
  "components": {
    "schema": 10.0,
    "canonical": 20.0,
    "supplemental_recall": 15.0,
    "guest_precision": 15.0,
    "guest_recall": 15.0,
    "rejection_recall": 15.0,
    "status": 10.0
  },
  "penalty": 0.0,
  "selected": {
    "canonical_official_id": "C01",
    "supplemental_official_ids": [
      "C02",
      "C03",
      "C04"
    ],
    "independent_or_guest_ids": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ],
    "rejected_ids": [
      "C05",
      "C07",
      "C10",
      "C11",
      "C13",
      "C15",
      "C16"
    ]
  }
}

12. qwen3.6:latest — local-mac-mini

Status accepted Score 100.00
Wall 91.95 s Ollama total 91.71 s
Load 20.53 s Tokens 1367 in / 396 out
Assessment Exact expected corpus
Exact model output 
{
  "canonical_official_id": "C01",
  "supplemental_official_ids": [
    "C02",
    "C03",
    "C04"
  ],
  "independent_or_guest_ids": [
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": [
    {
      "candidate_id": "C05",
      "reason": "Generic regional index listing multiple properties, not property-specific."
    },
    {
      "candidate_id": "C07",
      "reason": "Social-media promotional content, not an independent hotel-review corpus."
    },
    {
      "candidate_id": "C10",
      "reason": "Booking-only page; a review-specific page (C09) is available for this property."
    },
    {
      "candidate_id": "C11",
      "reason": "Individual review permalink; the property-level corpus (C08) is preferred."
    },
    {
      "candidate_id": "C13",
      "reason": "Syndicated mirror duplicating C12."
    },
    {
      "candidate_id": "C15",
      "reason": "Event/wedding-venue reviews, not a general hotel-stay review corpus."
    },
    {
      "candidate_id": "C16",
      "reason": "Third-party group-booking page with thin review material and booking emphasis."
    }
  ],
  "corpus_status": "ready_for_archive",
  "confidence": 0.98
}
Deterministic scoring receipt 
{
  "score": 100.0,
  "schema_valid": true,
  "invented_ids": [],
  "canonical_correct": true,
  "components": {
    "schema": 10.0,
    "canonical": 20.0,
    "supplemental_recall": 15.0,
    "guest_precision": 15.0,
    "guest_recall": 15.0,
    "rejection_recall": 15.0,
    "status": 10.0
  },
  "penalty": 0.0,
  "selected": {
    "canonical_official_id": "C01",
    "supplemental_official_ids": [
      "C02",
      "C03",
      "C04"
    ],
    "independent_or_guest_ids": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ],
    "rejected_ids": [
      "C05",
      "C07",
      "C10",
      "C11",
      "C13",
      "C15",
      "C16"
    ]
  }
}

13. qwen3.6:27b — local-mac-mini

Status accepted Score 100.00
Wall 105.83 s Ollama total 105.80 s
Load 11.94 s Tokens 1367 in / 384 out
Assessment Exact expected corpus
Exact model output 
{
  "canonical_official_id": "C01",
  "supplemental_official_ids": [
    "C02",
    "C03",
    "C04"
  ],
  "independent_or_guest_ids": [
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": [
    {
      "candidate_id": "C05",
      "reason": "Generic regional index listing multiple properties, not property-specific."
    },
    {
      "candidate_id": "C07",
      "reason": "Social media page; promotional content rather than independent review corpus."
    },
    {
      "candidate_id": "C10",
      "reason": "Booking-only duplicate; review-specific page C09 is preferred."
    },
    {
      "candidate_id": "C11",
      "reason": "Individual review permalink; property-level corpus C08 is preferred."
    },
    {
      "candidate_id": "C13",
      "reason": "Syndicated mirror of Expedia (C12); duplicate content."
    },
    {
      "candidate_id": "C15",
      "reason": "Event-only reviews (wedding venue), not general hotel-stay reviews."
    },
    {
      "candidate_id": "C16",
      "reason": "Group booking page with thin review material; booking emphasis."
    }
  ],
  "corpus_status": "ready_for_archive",
  "confidence": 0.95
}
Deterministic scoring receipt 
{
  "score": 100.0,
  "schema_valid": true,
  "invented_ids": [],
  "canonical_correct": true,
  "components": {
    "schema": 10.0,
    "canonical": 20.0,
    "supplemental_recall": 15.0,
    "guest_precision": 15.0,
    "guest_recall": 15.0,
    "rejection_recall": 15.0,
    "status": 10.0
  },
  "penalty": 0.0,
  "selected": {
    "canonical_official_id": "C01",
    "supplemental_official_ids": [
      "C02",
      "C03",
      "C04"
    ],
    "independent_or_guest_ids": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ],
    "rejected_ids": [
      "C05",
      "C07",
      "C10",
      "C11",
      "C13",
      "C15",
      "C16"
    ]
  }
}

14. qwen3.5:9b — remote-10.7.6.39

Status accepted Score 95.00
Wall 11.81 s Ollama total 11.78 s
Load 5.80 s Tokens 1367 in / 399 out
Assessment missed C04
Exact model output 
{
  "canonical_official_id": "C01",
  "supplemental_official_ids": [
    "C02",
    "C03"
  ],
  "independent_or_guest_ids": [
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": [
    {
      "candidate_id": "C04",
      "reason": "Generic regional index; duplicate of C01/C02 content."
    },
    {
      "candidate_id": "C05",
      "reason": "Generic regional index listing multiple properties, not property-specific."
    },
    {
      "candidate_id": "C07",
      "reason": "Social-media account; promotional posts rather than independent review corpus."
    },
    {
      "candidate_id": "C10",
      "reason": "Booking-only page; C09 provides the required review-specific content for this property."
    },
    {
      "candidate_id": "C11",
      "reason": "Individual review permalink; C08 provides the required property-level corpus."
    },
    {
      "candidate_id": "C13",
      "reason": "Syndicated mirror of C12; duplicate content."
    },
    {
      "candidate_id": "C15",
      "reason": "Event/wedding-venue reviews only; not a general hotel-stay review corpus."
    },
    {
      "candidate_id": "C16",
      "reason": "Group booking page with thin review material and booking emphasis."
    }
  ],
  "corpus_status": "ready_for_archive",
  "confidence": 0.95
}
Deterministic scoring receipt 
{
  "score": 95.0,
  "schema_valid": true,
  "invented_ids": [],
  "canonical_correct": true,
  "components": {
    "schema": 10.0,
    "canonical": 20.0,
    "supplemental_recall": 10.0,
    "guest_precision": 15.0,
    "guest_recall": 15.0,
    "rejection_recall": 15.0,
    "status": 10.0
  },
  "penalty": 0.0,
  "selected": {
    "canonical_official_id": "C01",
    "supplemental_official_ids": [
      "C02",
      "C03"
    ],
    "independent_or_guest_ids": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ],
    "rejected_ids": [
      "C04",
      "C05",
      "C07",
      "C10",
      "C11",
      "C13",
      "C15",
      "C16"
    ]
  }
}

15. qwen3.5:9b-64k — local-mac-mini

Status accepted Score 95.00
Wall 31.63 s Ollama total 31.61 s
Load 2.08 s Tokens 1367 in / 411 out
Assessment missed C04
Exact model output 
{
  "canonical_official_id": "C01",
  "supplemental_official_ids": [
    "C02",
    "C03"
  ],
  "independent_or_guest_ids": [
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": [
    {
      "candidate_id": "C04",
      "reason": "Property-specific location page; lacks independent guest reviews required for the corpus."
    },
    {
      "candidate_id": "C05",
      "reason": "Generic regional index listing multiple properties rather than a property-specific review corpus."
    },
    {
      "candidate_id": "C07",
      "reason": "Social-media account with promotional content, not an independent review corpus."
    },
    {
      "candidate_id": "C10",
      "reason": "Booking and availability page; reviews are available on the separate C09 URL."
    },
    {
      "candidate_id": "C11",
      "reason": "Single individual review permalink; property-level corpus (C08) is preferred."
    },
    {
      "candidate_id": "C13",
      "reason": "Syndicated Expedia-family mirror duplicating C12; not an independent source."
    },
    {
      "candidate_id": "C15",
      "reason": "Event and wedding-venue reviews only, not a general hotel-stay review corpus."
    },
    {
      "candidate_id": "C16",
      "reason": "Third-party group-booking page with thin review material and booking emphasis."
    }
  ],
  "corpus_status": "ready_for_archive",
  "confidence": 0.95
}
Deterministic scoring receipt 
{
  "score": 95.0,
  "schema_valid": true,
  "invented_ids": [],
  "canonical_correct": true,
  "components": {
    "schema": 10.0,
    "canonical": 20.0,
    "supplemental_recall": 10.0,
    "guest_precision": 15.0,
    "guest_recall": 15.0,
    "rejection_recall": 15.0,
    "status": 10.0
  },
  "penalty": 0.0,
  "selected": {
    "canonical_official_id": "C01",
    "supplemental_official_ids": [
      "C02",
      "C03"
    ],
    "independent_or_guest_ids": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ],
    "rejected_ids": [
      "C04",
      "C05",
      "C07",
      "C10",
      "C11",
      "C13",
      "C15",
      "C16"
    ]
  }
}

16. qwen3.5:9b — local-mac-mini

Status accepted Score 95.00
Wall 33.54 s Ollama total 33.52 s
Load 4.08 s Tokens 1367 in / 411 out
Assessment missed C04
Exact model output 
{
  "canonical_official_id": "C01",
  "supplemental_official_ids": [
    "C02",
    "C03"
  ],
  "independent_or_guest_ids": [
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": [
    {
      "candidate_id": "C04",
      "reason": "Property-specific location page; lacks independent guest reviews required for the corpus."
    },
    {
      "candidate_id": "C05",
      "reason": "Generic regional index listing multiple properties rather than a property-specific review corpus."
    },
    {
      "candidate_id": "C07",
      "reason": "Social-media account with promotional content, not an independent review corpus."
    },
    {
      "candidate_id": "C10",
      "reason": "Booking and availability page; reviews are available on the separate C09 URL."
    },
    {
      "candidate_id": "C11",
      "reason": "Single individual review permalink; property-level corpus (C08) is preferred."
    },
    {
      "candidate_id": "C13",
      "reason": "Syndicated Expedia-family mirror duplicating C12; not an independent source."
    },
    {
      "candidate_id": "C15",
      "reason": "Event and wedding-venue reviews only, not a general hotel-stay review corpus."
    },
    {
      "candidate_id": "C16",
      "reason": "Third-party group-booking page with thin review material and booking emphasis."
    }
  ],
  "corpus_status": "ready_for_archive",
  "confidence": 0.95
}
Deterministic scoring receipt 
{
  "score": 95.0,
  "schema_valid": true,
  "invented_ids": [],
  "canonical_correct": true,
  "components": {
    "schema": 10.0,
    "canonical": 20.0,
    "supplemental_recall": 10.0,
    "guest_precision": 15.0,
    "guest_recall": 15.0,
    "rejection_recall": 15.0,
    "status": 10.0
  },
  "penalty": 0.0,
  "selected": {
    "canonical_official_id": "C01",
    "supplemental_official_ids": [
      "C02",
      "C03"
    ],
    "independent_or_guest_ids": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ],
    "rejected_ids": [
      "C04",
      "C05",
      "C07",
      "C10",
      "C11",
      "C13",
      "C15",
      "C16"
    ]
  }
}

17. mo-shakib/clearwriter:latest — local-mac-mini

Status accepted Score 94.86
Wall 16.38 s Ollama total 16.36 s
Load 5.18 s Tokens 2537 in / 405 out
Assessment missed C06
Exact model output 
{
  "canonical_official_id": "C01",
  "supplemental_official_ids": [
    "C02",
    "C03",
    "C04"
  ],
  "independent_or_guest_ids": [
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": [
    {
      "candidate_id": "C05",
      "reason": "Generic regional index; rejected in favor of property-specific pages."
    },
    {
      "candidate_id": "C06",
      "reason": "Social-media page; rejected in favor of independent review corpus."
    },
    {
      "candidate_id": "C07",
      "reason": "Social-media account; rejected in favor of independent review corpus."
    },
    {
      "candidate_id": "C11",
      "reason": "Individual review permalink; property-level review (C08) is preferred."
    },
    {
      "candidate_id": "C13",
      "reason": "Syndicated mirror of C12; rejected as redundant."
    },
    {
      "candidate_id": "C15",
      "reason": "Event/wedding-venue reviews; not a general hotel-stay review corpus."
    },
    {
      "candidate_id": "C16",
      "reason": "Group booking page with thin review material; rejected in favor of property-level pages."
    }
  ],
  "corpus_status": "ready_for_archive",
  "confidence": 1.0
}
Deterministic scoring receipt 
{
  "score": 94.86,
  "schema_valid": true,
  "invented_ids": [],
  "canonical_correct": true,
  "components": {
    "schema": 10.0,
    "canonical": 20.0,
    "supplemental_recall": 15.0,
    "guest_precision": 15.0,
    "guest_recall": 12.0,
    "rejection_recall": 12.86,
    "status": 10.0
  },
  "penalty": 0.0,
  "selected": {
    "canonical_official_id": "C01",
    "supplemental_official_ids": [
      "C02",
      "C03",
      "C04"
    ],
    "independent_or_guest_ids": [
      "C08",
      "C09",
      "C12",
      "C14"
    ],
    "rejected_ids": [
      "C05",
      "C06",
      "C07",
      "C11",
      "C13",
      "C15",
      "C16"
    ]
  }
}

18. nemotron-3.5-lightning:30b — remote-10.7.6.39

Status accepted Score 94.00
Wall 36.50 s Ollama total 36.47 s
Load 19.21 s Tokens 1429 in / 463 out
Assessment missed C06, C14
Exact model output 
{
  "canonical_official_id": "C01",
  "supplemental_official_ids": [
    "C02",
    "C03",
    "C04"
  ],
  "independent_or_guest_ids": [
    "C08",
    "C09",
    "C12"
  ],
  "rejections": [
    {
      "candidate_id": "C05",
      "reason": "Generic regional index listing multiple DoubleTree properties around Chicago; not property-specific."
    },
    {
      "candidate_id": "C06",
      "reason": "Yelp business page; while review-capable, Tripadvisor and Booking.com offer more structured property-level review corpora per selection policy."
    },
    {
      "candidate_id": "C07",
      "reason": "Instagram social-media account; promotional posts rather than an independent hotel-review corpus."
    },
    {
      "candidate_id": "C10",
      "reason": "Booking.com booking and availability page; review-specific page already selected as C09."
    },
    {
      "candidate_id": "C11",
      "reason": "Single old user-review permalink; the property-level Tripadvisor corpus C08 is preferred for review corpus."
    },
    {
      "candidate_id": "C13",
      "reason": "Syndicated Expedia-family mirror of C12; duplicate content."
    },
    {
      "candidate_id": "C14",
      "reason": "Hotels.com property page contains reviews but is superseded by the more dedicated review corpus C09; also overlaps with booking intent."
    },
    {
      "candidate_id": "C15",
      "reason": "WeddingWire reviews are event and wedding-venue specific, not a general hotel-stay review corpus."
    },
    {
      "candidate_id": "C16",
      "reason": "HotelPlanner group-booking page with thin review material and booking emphasis; not a credible independent review source."
    }
  ],
  "corpus_status": "ready_for_archive",
  "confidence": 0.95
}
Deterministic scoring receipt 
{
  "score": 94.0,
  "schema_valid": true,
  "invented_ids": [],
  "canonical_correct": true,
  "components": {
    "schema": 10.0,
    "canonical": 20.0,
    "supplemental_recall": 15.0,
    "guest_precision": 15.0,
    "guest_recall": 9.0,
    "rejection_recall": 15.0,
    "status": 10.0
  },
  "penalty": 0.0,
  "selected": {
    "canonical_official_id": "C01",
    "supplemental_official_ids": [
      "C02",
      "C03",
      "C04"
    ],
    "independent_or_guest_ids": [
      "C08",
      "C09",
      "C12"
    ],
    "rejected_ids": [
      "C05",
      "C06",
      "C07",
      "C10",
      "C11",
      "C13",
      "C14",
      "C15",
      "C16"
    ]
  }
}

19. nemotron-3.5-lightning:30b-mlx — local-mac-mini

Status request_error Score 0.00
Wall 0.04 s Ollama total — s
Load — s Tokens None in / None out
Assessment Request failed
Exact model output 
HTTPStatusError: Server error '500 Internal Server Error' for url 'http://127.0.0.1:11434/api/generate'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
Deterministic scoring receipt 
{
  "score": 0.0,
  "schema_valid": false,
  "invented_ids": [],
  "canonical_correct": false,
  "components": {}
}

20. muse-glimmer:30b-mlx — local-mac-mini

Status request_error Score 0.00
Wall 0.06 s Ollama total — s
Load — s Tokens None in / None out
Assessment Request failed
Exact model output 
HTTPStatusError: Server error '500 Internal Server Error' for url 'http://127.0.0.1:11434/api/generate'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
Deterministic scoring receipt 
{
  "score": 0.0,
  "schema_valid": false,
  "invented_ids": [],
  "canonical_correct": false,
  "components": {}
}

21. gemma4:e2b-mlx — local-mac-mini

Status invalid_output Score 0.00
Wall 5.48 s Ollama total 5.46 s
Load 2.76 s Tokens 1489 in / 243 out
Assessment Wrong output contract
Exact model output 
{
  "selected_property_pages": [
    "C01",
    "C02",
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": {
    "C03": "Too specific (rooms page) when a general property page is preferred.",
    "C04": "Location page, less comprehensive than other property pages.",
    "C05": "Generic regional index, rejected.",
    "C07": "Social media page, rejected.",
    "C10": "Booking page, rejected in favor of review-heavy pages.",
    "C11": "Individual review permalink, rejected in favor of property-level corpus.",
    "C13": "Syndicated mirror duplicating C12, rejected.",
    "C15": "Event/wedding venue reviews, rejected.",
    "C16": "Group booking page with thin review material, rejected."
  }
}
Deterministic scoring receipt 
{
  "score": 0.0,
  "schema_valid": false,
  "invented_ids": [],
  "canonical_correct": false,
  "components": {}
}

22. gemma4:e4b-mlx — local-mac-mini

Status invalid_output Score 0.00
Wall 10.23 s Ollama total 10.21 s
Load 3.56 s Tokens 1489 in / 397 out
Assessment Wrong output contract
Exact model output 
{
  "selected_corpus": {
    "canonical_official_page": "C01",
    "supplemental_official_pages": [
      "C02",
      "C03",
      "C04"
    ],
    "independent_review_pages": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ]
  },
  "rejected_candidates": [
    {
      "id": "C05",
      "reason": "Generic regional index listing multiple DoubleTree properties around Chicago."
    },
    {
      "id": "C07",
      "reason": "Property social-media account; promotional posts rather than an independent hotel-review corpus."
    },
    {
      "id": "C10",
      "reason": "Property booking and availability page; a review-specific page is separately available as C09."
    },
    {
      "id": "C11",
      "reason": "A single old user-review permalink; the property-level Tripadvisor corpus is C08."
    },
    {
      "id": "C13",
      "reason": "Syndicated Expedia-family mirror of the same property, duplicating C12."
    },
    {
      "id": "C15",
      "reason": "Event and wedding-venue reviews, not a general hotel-stay review corpus."
    },
    {
      "id": "C16",
      "reason": "Third-party group-booking page with thin review material and booking emphasis."
    }
  ]
}
Deterministic scoring receipt 
{
  "score": 0.0,
  "schema_valid": false,
  "invented_ids": [],
  "canonical_correct": false,
  "components": {}
}

23. gemma4:26b-16k — local-mac-mini

Status invalid_output Score 0.00
Wall 19.58 s Ollama total 19.56 s
Load 7.09 s Tokens 1493 in / 274 out
Assessment Wrong output contract
Exact model output 
{
  "selected": {
    "official_property_page": "C01",
    "supplemental_official_pages": [
      "C02",
      "C03",
      "C04"
    ],
    "independent_review_pages": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ]
  },
  "rejections": {
    "C05": "Generic regional index listing multiple properties.",
    "C07": "Social media page; promotional rather than a review corpus.",
    "C10": "Booking-only page; review-specific content is available in C09.",
    "C11": "Individual review permalink; property-level corpus is available in C08.",
    "C13": "Syndicated mirror of C12.",
    "C15": "Event/wedding-specific reviews rather than general hotel-stay reviews.",
    "C16": "Thin booking-focused page rather than a review corpus."
  },
  "ready_for_archive": true
}
Deterministic scoring receipt 
{
  "score": 0.0,
  "schema_valid": false,
  "invented_ids": [],
  "canonical_correct": false,
  "components": {}
}

24. gemma4:26b-mlx — local-mac-mini

Status invalid_output Score 0.00
Wall 19.92 s Ollama total 19.90 s
Load 7.44 s Tokens 1493 in / 274 out
Assessment Wrong output contract
Exact model output 
{
  "selected": {
    "official_property_page": "C01",
    "supplemental_official_pages": [
      "C02",
      "C03",
      "C04"
    ],
    "independent_review_pages": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ]
  },
  "rejections": {
    "C05": "Generic regional index listing multiple properties.",
    "C07": "Social media page; promotional rather than a review corpus.",
    "C10": "Booking-only page; review-specific content is available in C09.",
    "C11": "Individual review permalink; property-level corpus is available in C08.",
    "C13": "Syndicated mirror of C12.",
    "C15": "Event/wedding-specific reviews rather than general hotel-stay reviews.",
    "C16": "Thin booking-focused page rather than a review corpus."
  },
  "ready_for_archive": true
}
Deterministic scoring receipt 
{
  "score": 0.0,
  "schema_valid": false,
  "invented_ids": [],
  "canonical_correct": false,
  "components": {}
}

25. gpt-oss:20b — local-mac-mini

Status invalid_output Score 0.00
Wall 20.47 s Ollama total 0.00 s
Load 0.00 s Tokens None in / None out
Assessment Wrong output contract
Exact model output 





Deterministic scoring receipt

{
  "score": 0.0,
  "schema_valid": false,
  "invented_ids": [],
  "canonical_correct": false,
  "components": {}
}




26. gemma4:12b-mlx — local-mac-mini






Status
invalid_output
Score
0.00




Wall
41.60 s
Ollama total
41.57 s




Load
1.09 s
Tokens
1493 in / 351 out




Assessment
Wrong output contract







Exact model output

{
  "canonical_official_property_page": "C01",
  "supplemental_official_property_pages": [
    "C02",
    "C03",
    "C04"
  ],
  "independent_or_guest_review_pages": [
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": [
    {
      "id": "C05",
      "reason": "Generic regional index listing multiple properties."
    },
    {
      "id": "C07",
      "reason": "Social-media page."
    },
    {
      "id": "C10",
      "reason": "Booking-only page; a review-specific page (C09) is available."
    },
    {
      "id": "C11",
      "reason": "Individual review permalink; property-level corpus (C08) is preferred."
    },
    {
      "id": "C13",
      "reason": "Syndicated mirror of C12."
    },
    {
      "id": "C15",
      "reason": "Event-only/wedding-venue reviews."
    },
    {
      "id": "C16",
      "reason": "Thin booking page with group-booking emphasis."
    }
  ]
}





Deterministic scoring receipt

{
  "score": 0.0,
  "schema_valid": false,
  "invented_ids": [],
  "canonical_correct": false,
  "components": {}
}




27. gemma4:12b-mlx-chat — local-mac-mini






Status
invalid_output
Score
0.00




Wall
44.01 s
Ollama total
43.98 s




Load
3.64 s
Tokens
1493 in / 351 out




Assessment
Wrong output contract







Exact model output

{
  "canonical_official_property_page": "C01",
  "supplemental_official_property_pages": [
    "C02",
    "C03",
    "C04"
  ],
  "independent_or_guest_review_pages": [
    "C06",
    "C08",
    "C09",
    "C12",
    "C14"
  ],
  "rejections": [
    {
      "id": "C05",
      "reason": "Generic regional index listing multiple properties."
    },
    {
      "id": "C07",
      "reason": "Social-media page."
    },
    {
      "id": "C10",
      "reason": "Booking-only page; a review-specific page (C09) is available."
    },
    {
      "id": "C11",
      "reason": "Individual review permalink; property-level corpus (C08) is preferred."
    },
    {
      "id": "C13",
      "reason": "Syndicated mirror of C12."
    },
    {
      "id": "C15",
      "reason": "Event-only/wedding-venue reviews."
    },
    {
      "id": "C16",
      "reason": "Thin booking page with group-booking emphasis."
    }
  ]
}





Deterministic scoring receipt

{
  "score": 0.0,
  "schema_valid": false,
  "invented_ids": [],
  "canonical_correct": false,
  "components": {}
}




28. qwen3.8:27b-mlx — local-mac-mini






Status
invalid_output
Score
0.00




Wall
62.28 s
Ollama total
62.26 s




Load
15.34 s
Tokens
1367 in / 384 out




Assessment
Wrong output contract







Exact model output

{
  "selected": {
    "canonical_official_property_page": "C01",
    "supplemental_official_property_pages": [
      "C02",
      "C03",
      "C04"
    ],
    "independent_or_guest_review_pages": [
      "C06",
      "C08",
      "C09",
      "C12",
      "C14"
    ]
  },
  "rejections": [
    {
      "candidate_id": "C05",
      "reason": "Generic regional index listing multiple DoubleTree properties, not property-specific."
    },
    {
      "candidate_id": "C07",
      "reason": "Social media page with promotional content, not an independent review corpus."
    },
    {
      "candidate_id": "C10",
      "reason": "Booking-only page; a dedicated review page (C09) is already selected."
    },
    {
      "candidate_id": "C11",
      "reason": "Individual review permalink; property-level review corpus (C08) is preferred."
    },
    {
      "candidate_id": "C13",
      "reason": "Syndicated mirror of C12 (Expedia-AARP), duplicating content."
    },
    {
      "candidate_id": "C15",
      "reason": "Event-specific (wedding venue) reviews, not general hotel-stay reviews."
    },
    {
      "candidate_id": "C16",
      "reason": "Thin booking-focused page with insufficient review material."
    }
  ]
}





Deterministic scoring receipt

{
  "score": 0.0,
  "schema_valid": false,
  "invented_ids": [],
  "canonical_correct": false,
  "components": {}
}




Interpretation


Thirteen model routes made the exact decision. This means source selection is not the difficult part once deterministic search has produced a bounded candidate set. The failed initial hotel attempt was primarily a control-plane problem: the 9B agent was asked to search, guess URLs, render pages, track budgets, recover errors, classify coverage, and write artifacts in one loop. This benchmark supports moving all of that choreography into code and reserving model work for a single bounded selection response.


Zero scores for malformed Gemma outputs should not be read as “the model understood nothing.” Their raw outputs often contained recognizable source choices, but they ignored the required field names or nesting. In an unattended pipeline, that is still a failure.


Return to the grid →




	





			

			

				

					
Similar Posts
					

						

							
						

					

				

			

		

		

	
One Comment
		
    
					
  1. 
			
    
				Pingback: Local Hotel Source Selection: 28-Model Benchmark – OddByte 			
    
		
    2. 
		

				
Comments are closed.