Squash Odysseus development history

docs/SEARCH_HARNESS_AUDIT_20260909.md

@@ -0,0 +1,92 @@
+# Search and compact-tool experiment — 2026-09-09
+
+## Decision
+
+Keep the normal routed profile on 7011. The all-tools compact experiment is
+implemented but **disabled**: direct routing success did not translate into a
+working Agent UI. Do not retrain or promote a profile on these measurements.
+
+## Changes
+
+- Short public-web lookups on the target model have an execution budget: two
+  distinct token-normalized searches, one fetch, and up to three browser calls
+  after the two searches. This bounds attempts, not just recovery prose. Existing
+  permissions still apply; this does not make unavailable tools executable.
+- Failed/weak searches reach the model for evaluation and query refinement,
+  instead of the earlier unconditional terminal evidence veto. Some legacy
+  heuristics and official-site shortcuts remain; this is not a completed rewrite.
+- Search providers retain query/engine/date provenance. Unconfigured credentialed
+  fallbacks are skipped. When SearXNG is the sole configured usable provider, Yep
+  on the same instance is an additional fallback.
+- An unavailable warm-only family no longer vetoes an otherwise ordinary reply.
+- The all-tools experiment offers the trained compact inventory subject to
+  permissions. It requires the exact test-owner environment flag and exact model
+  match. The temporary service flag was removed after failed UI testing.
+
+## Evidence and limits
+
+| Measurement | Result | What it establishes |
+|---|---|---|
+| Focused Python regression suite | 401 passed | Covered policy, contract, provider and recovery-budget behavior |
+| Direct family-only compact schemas | 10/10 tool routing | Small public smoke test, not functional or blind accuracy |
+| Direct all-family compact schemas | 10/10 tool routing | Inventory did not break these first calls; roughly 3–4x slower in this run |
+| Full compact Agent UI, revision 3 | All eight turns failed one or more checks | Not suitable for activation; leaks, duplicate/incorrect rendering or missing expected calls |
+| Normal-profile final UI control | Five passed checks, two product failures, one capture error | Not accepted; suite status incomplete |
+| Clean synthetic notes tool-result continuation | Clean answer with both schema sizes | Model can continue correctly on that isolated input, not proof that UI failure is solely harness |
+
+Direct probes used temperature 0; the UI target-model sampling path can cap at
+0.2. Prompts, history and tool-result serialization also differ. Match those
+before attributing UI failures to weights versus harness. Existing UI checks are
+not a grounded factual-answer benchmark. Unit tests are not UI acceptance.
+
+Normal-profile control details: notes initial, both calendar turns, AI search
+initial, and history initial passed the automated checks. Notes follow-up hit a
+Playwright `Network.getResponseBody` capture error and is inconclusive. AI search
+follow-up explicitly requested a summary with no tools, but the contract still
+required `search_browser` and returned a permission failure. The history search
+follow-up failed the visible-leak check. These are separate from source relevance;
+the earlier warm-only fix did not cover classification as an active requirement.
+There is no matched pre-change control establishing a net improvement.
+
+Provider isolation bypassed app relevance filters. Bing general often returned
+broad or unrelated results despite the full query. Google/Mojeek returned no
+results, DDG hit CAPTCHA, and Presearch timed out. Yep returned useful PostgreSQL
+documentation, but was weak or empty for several other questions. Engine health
+and source quality remain unresolved. Fallback cannot help when an earlier weak
+result survives filtering; there is no claim of universal relevance here.
+
+## Reproduce and inspect
+
+- `scripts/audit_search_pipeline.py`: raw provider comparison, no model.
+- `scripts/compare_compact_tool_inventory.py`: read-only model schema comparison;
+  proposed calls are never executed.
+- `scripts/verify_agent_turn_contract.mjs`: real 7011 Agent UI and persisted-history
+  checks. Use the dedicated test account; reports can contain private tool data.
+- `reports/search-provider-isolation.json`, `reports/search-yep-isolation.json`:
+  public provider evidence.
+- `reports/compact-inventory-ablation.json`: direct routing probe.
+- `reports/full-compact-ui-audit-r3.json`: completed rejected UI experiment.
+  Earlier experiment reports include an initialization error and an aborted run;
+  do not combine them into an accuracy score.
+- `reports/routed-control-ui-audit-final.json`: normal-profile control replay;
+  eight attempts, incomplete because of the capture error; failures retained.
+
+Focused suite:
+
+```sh
+/home/pewds/odysseus-cookbook-fresh/.venv/bin/pytest -q tests/test_turn_contract.py tests/test_turn_contract_integration.py tests/test_service_search_provider_guards.py tests/test_web_recovery_budget.py tests/test_tool_policy.py
+```
+
+UI replay (read-only prompts, creates test chats):
+
+```sh
+node scripts/verify_agent_turn_contract.mjs --families notes,calendar,search_ai,search_history --pairs notes:11,calendar:11,search_ai:11,search_history:11 --max-turns 8 --total-ms 360000 --turn-ms 45000 --report reports/routed-control-ui-audit-final.json
+```
+
+## Next discriminating test
+
+Replay the same captured UI request directly, preserving sampling, compact
+schemas, history and tool results. Then change one layer at a time. Separately
+score retrieved-source relevance and supported answers. Replace failing generic
+boundaries only when the replay identifies them; do not add rules for individual
+user phrasings or treat successful tool routing as successful execution.