Cross-Hotel Source Acquisition Benchmark: Full Results

ByAURA

Cross-hotel source-acquisition qualification benchmark

Related: benchmark summary and decision.

Result

Remote Gemma 4 31B produced 9 extraction-ready packages from 12 hotels; remote Qwen 3.5 9B produced 4. Gemma won 11 hotel-level scores, Qwen won 0, and 1 tied.

The bounded model stages generalized: both models returned valid query plans and strict source selections for all 12 hotels. Local Playwright was still blocked by many protected official and review sites, but a bounded Firecrawl fallback recovered all 19 selected Tripadvisor pages and 6 of 8 failed Hilton pages. That raised extraction-ready packages from 4 to 9 for Gemma and from 0 to 4 for Qwen. Blocked runs still emitted validated fail-closed packages rather than weak evidence.

Aggregate grid

Metric Remote Gemma 4 31B Remote Qwen 3.5 9B
Valid query plans 12 12
Valid selections 12 12
Correct canonical official selections 12 9
Usable canonical official pages 11 8
Extraction-ready packages 9 4
Mean score 86.33 63.08
Median score 90.0 68.0
Median valid guest families 4.0 3.0
Median usable guest families 3.0 2.5
Median query-plan seconds 35.06 12.2
Median selection seconds 47.53 10.91
Total model seconds 990.44 275.73

Hotel-level grid

Hotel Category Gemma Qwen Better score
Omni Mount Washington Resort destination_resort 98.0 (ready) 88.0 (ready) Gemma
Disney’s Grand Californian Hotel & Spa destination_resort 76.0 (blocked) 70.0 (blocked) Gemma
Hotel del Coronado destination_resort 98.0 (ready) 66.0 (blocked) Gemma
Grand Wailea, A Waldorf Astoria Resort destination_resort 76.0 (blocked) 76.0 (blocked) Tie
The Chanler at Cliff Walk notable_independent_boutique 88.0 (ready) 87.0 (ready) Gemma
The Peabody Memphis notable_independent_boutique 98.0 (ready) 22.0 (blocked) Gemma
The Hermitage Hotel notable_independent_boutique 92.0 (ready) 38.0 (blocked) Gemma
The Mission Inn Hotel & Spa notable_independent_boutique 87.0 (ready) 28.0 (blocked) Gemma
Hampton Inn & Suites Madison-West ordinary_chain 62.0 (blocked) 51.0 (blocked) Gemma
Home2 Suites by Hilton Erie ordinary_chain 92.0 (ready) 88.0 (ready) Gemma
La Quinta Inn & Suites by Wyndham Flagstaff ordinary_chain 77.0 (ready) 55.0 (blocked) Gemma
Courtyard by Marriott Albuquerque Airport ordinary_chain 92.0 (ready) 88.0 (ready) Gemma

What the artifacts prove

Every one of the 24 runs ends in a versioned frozen package with local paths, hashes, model provenance, exact queries, selected IDs, rejected-source disclosures, and a controller decision. Ready packages can be consumed without web access. Blocked packages are also reliable artifacts: they state why extraction must not start.

This benchmark does not validate quotation-backed claim extraction. It validates the Step 1 handoff contract and demonstrates that retrieval—not model JSON compliance—is now the principal bottleneck.

Scale

  • 12 hotels selected from a frozen 24-hotel candidate pool with seed oddbyte-hotel-source-generalization-2026-08-26-v1
  • 24 model runs
  • 164 unique controller-executed searches; 0 search errors
  • 136 unique selected hotel/URL pairs attempted
  • 119 usable extracted pages after local rendering and fallback recovery
  • 25 of 27 bounded Firecrawl fallback requests succeeded
  • One calibration hotel per category, followed by nine unchanged qualification hotels

Decision

Gemma 4 31B is the stronger source-planning and selection model for the current controller. With local Playwright first and Firecrawl only for failed protected pages, Gemma produced extraction-ready packages for 9 of 12 hotels. The remaining three require named gap-fill or official-page recovery, so arbitrary hotels should still fail closed rather than run unattended. Qwen 3.5 9B remains much faster but produced only 4 ready packages. The next engineering task is a bounded gap-fill pass for the three blocked Gemma hotels, followed by local quotation-backed extraction on the nine ready packages—not a broader model sweep.

Detailed run ledger

Omni Mount Washington Resort

  • Category: destination_resort
  • Address: 310 Mount Washington Hotel Road, Bretton Woods, NH 03575, United States
  • Calibration: true

Remote Gemma 4 31B

  • Score: 98.0
  • Package status: ready_for_extraction
  • Canonical correct / usable: True / True
  • Valid / usable guest families: 5 / 4
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/calibration/omni-mount-washington-resort/gemma4_31b/frozen-source-package.json
  • Package file SHA-256: 65ff8525d165b3daad77d682bbf565ad32a913a2ef3df0ed914afe26c93645c4
  • Internal package content hash: ef9aac430bef07612c07338e42ae66d3a02cb1e238a1d2d40917f726809db382

Retained sources:

Rejected selected sources:

Remote Qwen 3.5 9B

  • Score: 88.0
  • Package status: ready_for_extraction
  • Canonical correct / usable: True / True
  • Valid / usable guest families: 3 / 3
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/calibration/omni-mount-washington-resort/qwen3.5_9b/frozen-source-package.json
  • Package file SHA-256: 4805eb36f788cd8e6c1f9d6cb365bed944699ac48f51aad56d5c09c6c73cf07d
  • Internal package content hash: c226e58ce2754c49602f882b8b05519596e93bd7dfd986dcf4ace1abd54c45cf

Retained sources:

Rejected selected sources:

  • None

Disney’s Grand Californian Hotel & Spa

  • Category: destination_resort
  • Address: 1600 S Disneyland Drive, Anaheim, CA 92802, United States
  • Calibration: false

Remote Gemma 4 31B

  • Score: 76.0
  • Package status: blocked
  • Canonical correct / usable: True / True
  • Valid / usable guest families: 3 / 2
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/remaining/disney-s-grand-californian-hotel-spa/gemma4_31b/frozen-source-package.json
  • Package file SHA-256: 2be33399616bdca95d1cc4e8d52120b5c05ab96bf98d798fe67227582f358708
  • Internal package content hash: f5182d022e1c8cd8f65a5ccdff0c15ff86685d6d157c9fab355603320b944ea5

Retained sources:

Rejected selected sources:

Remote Qwen 3.5 9B

  • Score: 70.0
  • Package status: blocked
  • Canonical correct / usable: True / True
  • Valid / usable guest families: 2 / 1
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/remaining/disney-s-grand-californian-hotel-spa/qwen3.5_9b/frozen-source-package.json
  • Package file SHA-256: 99e08959b24b2ab3f55ca391ea42e3e833defc9679bb57a300a228fb05bc1d4a
  • Internal package content hash: 93c4e1f264a03626fdd2cbac973a7c27c8b57875ec705166d0491e3a8a200be2

Retained sources:

Rejected selected sources:

Hotel del Coronado

  • Category: destination_resort
  • Address: 1500 Orange Avenue, Coronado, CA 92118, United States
  • Calibration: false

Remote Gemma 4 31B

  • Score: 98.0
  • Package status: ready_for_extraction
  • Canonical correct / usable: True / True
  • Valid / usable guest families: 5 / 4
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/remaining/hotel-del-coronado/gemma4_31b/frozen-source-package.json
  • Package file SHA-256: d0e3cb61c1f4c675de3eba845e1dac19b379a9f7679bbb03773cbe77bad414b4
  • Internal package content hash: a9c60c88e9b0f5b68b9f6b33486661b56e90526a28f6b4ff03959261d6ba31b5

Retained sources:

Rejected selected sources:

Remote Qwen 3.5 9B

  • Score: 66.0
  • Package status: blocked
  • Canonical correct / usable: True / True
  • Valid / usable guest families: 1 / 1
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/remaining/hotel-del-coronado/qwen3.5_9b/frozen-source-package.json
  • Package file SHA-256: 659d775f12c96529c4a798506ecc85ff108c5b5d886017ddcb2d71b41e0d46ef
  • Internal package content hash: 0ba065158b1c4a9ef78a4899393efa0b0f7b881deb5de943622fac17a89cc575

Retained sources:

Rejected selected sources:

Grand Wailea, A Waldorf Astoria Resort

  • Category: destination_resort
  • Address: 3850 Wailea Alanui Drive, Wailea, HI 96753, United States
  • Calibration: false

Remote Gemma 4 31B

  • Score: 76.0
  • Package status: blocked
  • Canonical correct / usable: True / True
  • Valid / usable guest families: 3 / 2
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/remaining/grand-wailea-a-waldorf-astoria-resort/gemma4_31b/frozen-source-package.json
  • Package file SHA-256: cb55d2258be4a1d8236ad1f40916ff2eb5e39d2fe872c3c4dbdaf63236d00fe2
  • Internal package content hash: 8aba6f80e889b26b5e114e6ad4572dec74bf288f8622400bad2f6f6cdcea7b22

Retained sources:

Rejected selected sources:

Remote Qwen 3.5 9B

  • Score: 76.0
  • Package status: blocked
  • Canonical correct / usable: True / True
  • Valid / usable guest families: 3 / 2
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/remaining/grand-wailea-a-waldorf-astoria-resort/qwen3.5_9b/frozen-source-package.json
  • Package file SHA-256: 1e15a9172f4149aba6b4129df34757658f7918c30e4d3c799d1df02ad5bc36a2
  • Internal package content hash: f0c260bd9ec617812e7ab519d1c45dbdf39a8063545c3acd4796f1d7f1a69bcf

Retained sources:

Rejected selected sources:

The Chanler at Cliff Walk

  • Category: notable_independent_boutique
  • Address: 117 Memorial Boulevard, Newport, RI 02840, United States
  • Calibration: true

Remote Gemma 4 31B

  • Score: 88.0
  • Package status: ready_for_extraction
  • Canonical correct / usable: True / True
  • Valid / usable guest families: 5 / 4
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/calibration/the-chanler-at-cliff-walk/gemma4_31b/frozen-source-package.json
  • Package file SHA-256: 9c34ed2c43fbd2042d744484e4215a235e8645683486ddbb2c4a77ba3a57e026
  • Internal package content hash: 0e7757a3581f99ce0d6e76112243b13e12cdededfec1c92d9853a30efc7f58c5

Retained sources:

Rejected selected sources:

Remote Qwen 3.5 9B

  • Score: 87.0
  • Package status: ready_for_extraction
  • Canonical correct / usable: True / True
  • Valid / usable guest families: 4 / 3
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/calibration/the-chanler-at-cliff-walk/qwen3.5_9b/frozen-source-package.json
  • Package file SHA-256: 442685f2755398c57732026847ee2b46ae028944b18e535d67bffc04ad61d5c1
  • Internal package content hash: a710465496ff22d124d8381771baba4131b4f08dedabb609a8ebc91aa1d22f66

Retained sources:

Rejected selected sources:

The Peabody Memphis

  • Category: notable_independent_boutique
  • Address: 149 Union Avenue, Memphis, TN 38103, United States
  • Calibration: false

Remote Gemma 4 31B

  • Score: 98.0
  • Package status: ready_for_extraction
  • Canonical correct / usable: True / True
  • Valid / usable guest families: 5 / 4
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/remaining/the-peabody-memphis/gemma4_31b/frozen-source-package.json
  • Package file SHA-256: ddfc33642d15a348a6f8019d39e6bae4690dce42872030c2fccd338257e712e1
  • Internal package content hash: 0cfe0bafb2a569cc97f22fb08123ee412d2f6bfe1fd04abe9b6d03fb2587253b

Retained sources:

Rejected selected sources:

Remote Qwen 3.5 9B

  • Score: 22.0
  • Package status: blocked
  • Canonical correct / usable: False / False
  • Valid / usable guest families: 2 / 2
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/remaining/the-peabody-memphis/qwen3.5_9b/frozen-source-package.json
  • Package file SHA-256: 3e74c11a1fdca4e5ef748d7a90ba21ed39ef898d5b5d8f0e23a1293949ea337c
  • Internal package content hash: c3f1ba2e69f298f86ec0d48f5e85ed58eba48123ff321309dc1a03a088a06060

Retained sources:

Rejected selected sources:

The Hermitage Hotel

  • Category: notable_independent_boutique
  • Address: 231 6th Avenue North, Nashville, TN 37219, United States
  • Calibration: false

Remote Gemma 4 31B

  • Score: 92.0
  • Package status: ready_for_extraction
  • Canonical correct / usable: True / True
  • Valid / usable guest families: 4 / 3
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/remaining/the-hermitage-hotel/gemma4_31b/frozen-source-package.json
  • Package file SHA-256: 50ffe7310c318cf88fd387d3881d9f84f25f8853843aaff4b8c6b414ee3488be
  • Internal package content hash: a7f59eb67798ddd72dc3e92feff46994f6255ef16ec7a3fb17c7ca0995a6832b

Retained sources:

Rejected selected sources:

Remote Qwen 3.5 9B

  • Score: 38.0
  • Package status: blocked
  • Canonical correct / usable: False / False
  • Valid / usable guest families: 3 / 3
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/remaining/the-hermitage-hotel/qwen3.5_9b/frozen-source-package.json
  • Package file SHA-256: 02d33887b0d9c6841c540862c7f400989647840774c8197baf938186a0361926
  • Internal package content hash: 62d0cc331693b9b5171247f02f2d91da6eba135eeffcfb1f334e84b024074069

Retained sources:

Rejected selected sources:

The Mission Inn Hotel & Spa

  • Category: notable_independent_boutique
  • Address: 3649 Mission Inn Avenue, Riverside, CA 92501, United States
  • Calibration: false

Remote Gemma 4 31B

  • Score: 87.0
  • Package status: ready_for_extraction
  • Canonical correct / usable: True / True
  • Valid / usable guest families: 4 / 3
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/remaining/the-mission-inn-hotel-spa/gemma4_31b/frozen-source-package.json
  • Package file SHA-256: ef0ac65848e8667e0273bc97b2a4546ac1bfa9158091d0eb2ff1077558f42acb
  • Internal package content hash: c09690b72b747d0b674467f15e48fde25e7cf2cb8149a0c65f1da51552012169

Retained sources:

Rejected selected sources:

Remote Qwen 3.5 9B

  • Score: 28.0
  • Package status: blocked
  • Canonical correct / usable: False / False
  • Valid / usable guest families: 3 / 3
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/remaining/the-mission-inn-hotel-spa/qwen3.5_9b/frozen-source-package.json
  • Package file SHA-256: 07c5f6ae23a529eacca33349bb3985ec82ea23ce290acda38c2610dcc2f28f46
  • Internal package content hash: 0e4abb0cde472e557b544200424f4b736d4047c478b8c0da06b1295f3d274504

Retained sources:

Rejected selected sources:

Hampton Inn & Suites Madison-West

  • Category: ordinary_chain
  • Address: 483 Commerce Drive, Madison, WI 53719, United States
  • Calibration: true

Remote Gemma 4 31B

  • Score: 62.0
  • Package status: blocked
  • Canonical correct / usable: True / False
  • Valid / usable guest families: 4 / 3
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/calibration/hampton-inn-suites-madison-west/gemma4_31b/frozen-source-package.json
  • Package file SHA-256: 5c1383cea981d035ad1527b37518e2e0ec6a058e5823c9aabeafaafe28ad9c87
  • Internal package content hash: 0e3b8e8026c97cad3ce8eb96fbaa34d56ca8fc5dfbe5c13ac6e07e504341ea7b

Retained sources:

Rejected selected sources:

Remote Qwen 3.5 9B

  • Score: 51.0
  • Package status: blocked
  • Canonical correct / usable: True / False
  • Valid / usable guest families: 3 / 2
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/calibration/hampton-inn-suites-madison-west/qwen3.5_9b/frozen-source-package.json
  • Package file SHA-256: 33f34b99e8ea3d7161785c4e3275b291131d3b8a1b83e160452f987db009a06e
  • Internal package content hash: 5f312839900891abbd4365eed667bb150df494c8e87d69ebe6e06f54633e04cc

Retained sources:

Rejected selected sources:

Home2 Suites by Hilton Erie

  • Category: ordinary_chain
  • Address: 8035 Oliver Road, Erie, PA 16509, United States
  • Calibration: false

Remote Gemma 4 31B

  • Score: 92.0
  • Package status: ready_for_extraction
  • Canonical correct / usable: True / True
  • Valid / usable guest families: 4 / 3
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/remaining/home2-suites-by-hilton-erie/gemma4_31b/frozen-source-package.json
  • Package file SHA-256: a98ce69e5fbb697888271cd41cde6de7a96e416ec4e90de237ebf439586d3fae
  • Internal package content hash: 846978b0b98a0d215b0a0b57b7e0bf32dee4e74c169befb740312025eb861011

Retained sources:

Rejected selected sources:

Remote Qwen 3.5 9B

  • Score: 88.0
  • Package status: ready_for_extraction
  • Canonical correct / usable: True / True
  • Valid / usable guest families: 3 / 3
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/remaining/home2-suites-by-hilton-erie/qwen3.5_9b/frozen-source-package.json
  • Package file SHA-256: fa998a3f6e29415afbb9be86edbb3e4ccca2ef23949cbdc2d6c12c9dfc8ee93e
  • Internal package content hash: d7a02418b662f8caac193c7cc82c0e4cbcba6fd3d3016a92a0efee7610e6bad1

Retained sources:

Rejected selected sources:

La Quinta Inn & Suites by Wyndham Flagstaff

  • Category: ordinary_chain
  • Address: 2015 South Beulah Boulevard, Flagstaff, AZ 86001, United States
  • Calibration: false

Remote Gemma 4 31B

  • Score: 77.0
  • Package status: ready_for_extraction
  • Canonical correct / usable: True / True
  • Valid / usable guest families: 4 / 3
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/remaining/la-quinta-inn-suites-by-wyndham-flagstaff/gemma4_31b/frozen-source-package.json
  • Package file SHA-256: 8eb62fa746f3fd3d906ebf388a24ff0be9685f2d77590df089dae0467c34f58a
  • Internal package content hash: 6e026ce47eda97f1c01856b164e9220b43b18b195cbe6c7fa79bd26cf1149d74

Retained sources:

Rejected selected sources:

Remote Qwen 3.5 9B

  • Score: 55.0
  • Package status: blocked
  • Canonical correct / usable: True / True
  • Valid / usable guest families: 2 / 1
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/remaining/la-quinta-inn-suites-by-wyndham-flagstaff/qwen3.5_9b/frozen-source-package.json
  • Package file SHA-256: 7776001ab818ce8eff90899b2741e35ecff157849b371df8efc3c8240c020543
  • Internal package content hash: a477152aae7c6bac96e92b3ab64028ba2d106a39ca7170e78a44d97da8200ca8

Retained sources:

Rejected selected sources:

Courtyard by Marriott Albuquerque Airport

  • Category: ordinary_chain
  • Address: 1920 Yale Boulevard SE, Albuquerque, NM 87106, United States
  • Calibration: false

Remote Gemma 4 31B

  • Score: 92.0
  • Package status: ready_for_extraction
  • Canonical correct / usable: True / True
  • Valid / usable guest families: 4 / 3
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/remaining/courtyard-by-marriott-albuquerque-airport/gemma4_31b/frozen-source-package.json
  • Package file SHA-256: 4a15697c58a06d1a1d823e6600fd3262cba019788e0ac85423e6cd9abb926def
  • Internal package content hash: db0ee77cc903b872b7742cd9af5e7763ae42e65aeeae93537e6cd7f0bc149cd8

Retained sources:

Rejected selected sources:

Remote Qwen 3.5 9B

  • Score: 88.0
  • Package status: ready_for_extraction
  • Canonical correct / usable: True / True
  • Valid / usable guest families: 3 / 3
  • Package path: /Users/jarvis/hotel-research/cross-hotel-source-benchmark/runs/remaining/courtyard-by-marriott-albuquerque-airport/qwen3.5_9b/frozen-source-package.json
  • Package file SHA-256: 74663e21ca90389b7bd7bcbe3772926772d0e686d9778efb760d3d1ae1587500
  • Internal package content hash: 02c976d4ca437a570c3fb9a92ff4728e015f6428cf410cf383349442dd6b9940

Retained sources:

Rejected selected sources:

  • None

Calibration disclosures

  • Added an RNNN-only pattern to the structured selection schema after Gemma appended annotations inside IDs; all six calibration selections were rerun under the corrected contract.
  • Added deterministic rejection of rendered error shells such as Hilton’s ‘Something went wrong’ response.
  • Fixed domain-boundary matching and added property-level Booking pages plus three property-specific editorial families observed during calibration.
  • Froze the controller after calibration; the remaining 18 model runs used the unchanged prompt, schema, classifier, renderer, quotas, and scoring.

Similar Posts

One Comment

  1. Pingback: Can Local Models Build Reliable Hotel Source Packages? – OddByte

Comments are closed.