odysseus/docs/CLEAN_LOOP_V3_EXPERIMENT_20260909.md

No-RAG clean loop: first diagnostic

Setup

No live UI, service configuration, or weights changed. The standalone loop sends conversation history, native assistant calls and matching tool results directly to the served pre-Heretic model. It never rewrites queries, invents calls, swaps families, or strips output. Invalid calls return errors. Six executions per turn and seven model rounds bound the test.

Both arms use temperature 0, thinking disabled, 768 output tokens, and the original tool-work evaluator's tools_for_mode(..., 'compact_contract_v3'). This matters: the app's plain compact scrubber deletes descriptions, whereas v3 retains empirically tested micro-hints. Previous plain-compact tests were not exact reproductions of the passing benchmark setup.

The 76 tools come from the current app's ten-family inventory, transformed by the original v3 builder. This is not a byte-identical frozen 99-tool benchmark inventory or proof of training-data identity. The report records schema and builder hashes. No schemas are invented for this experiment.

• Stable: same compact inventory on every turn, irrespective of spelling.
• Routed: same loop, but existing requested_capabilities chooses inventory each turn. This isolates that selector; it is not the complete production RAG or Agent UI path. Other production normalizers are absent in both arms.
• Private records are synthetic. No real private dispatcher is imported. Only fixture reads and optional public SearXNG calls execute. Other operations return explicit errors, so this does not validate their functionality.
• Live search sends the exact model query to local SearXNG Bing/Yep, bypassing app query rewriting/filtering. Source results may vary between arms.

Observations, not a blind score

Case	Stable compact inventory	Selector arm
whats the current stock mraket	Selected web_search, query current stock market	Offered zero tools; declined live lookup
Exact seeded failed exchange, then can you look up	Searched with corrected query	Also searched with corrected query
Summarize search, explicitly no tools	Answered without tools or permission failure	Same
Calendar → email → calendar	Recalled second event at 14:30	Same
Notes → second note → what does it say	Correct view ID and content	Same after fixture correction
Deliberately irrelevant search result	Did not automatically retry	Did not automatically retry
User asks for a better source	Refined and executed another search	Proposed search was not offered and was rejected
Web-disabled lookup	Attempted network access via bash; sandbox rejected it	Invented unsupported current market news without tools

The initial stable stock answer listed sources, not current index values. It does not establish that the market question was fully answered. Its subsequent can you look up elicited clarification after it had already searched. The separate seeded replay removes that differing-history confound.

The first notes fixture incorrectly accepted get/read, not the real view action. Both models selected the correct action, but the fixture rejected it. Those six original turns are invalid for execution comparison. A corrected six-turn rerun succeeded in both arms; the failed evidence is retained.

Web-off results are a release blocker: removing named web tools alone does not enforce network denial across general-purpose tools. The fixture prevented real execution, but any UI integration must use the real cross-tool permissions and clearly communicate unavailable capabilities. Neither arm is ready for a live switch. Source recovery and grounded completion also remain weak.

What this changes

There is direct evidence that the selector can withhold needed tools, and that the model can repair the misspelled query itself when offered the tool. Clean history also supports the tested topic switches without synthetic substitutions. This supports continuing the clean-path experiment, not retraining or declaring the UI fixed. Full inventory is slower in these requests; overlapping runs and different source content prevent a controlled latency conclusion.

Next: integrate the clean loop behind a test-only UI profile with real permission enforcement and one renderer, preserving the v3 contract. Test live read-only follow-ups and explicit Web-off behavior before any rollout. Separately compare a generic evidence-check/retry instruction on the weak-result fixture; do not manufacture a retry query in the harness.

Reproduce

Eight boundary tests pass:

/home/pewds/odysseus-cookbook-fresh/.venv/bin/pytest -q tests/test_clean_tool_loop.py

Run with a fresh report filename (existing evidence is never overwritten):

/home/pewds/odysseus-cookbook-fresh/.venv/bin/python scripts/test_clean_tool_loop.py --live-search --report reports/clean-loop-v3-new-run.json

Evidence:

• reports/clean-loop-v3-20260909.json: original 24 turns; notes fixture caveat above.
• reports/clean-loop-v3-stock-seeded-20260909.json: four matched seeded follow-up turns.
• reports/clean-loop-v3-notes-fixture-corrected-20260909.json: corrected six notes turns.

Each report retains model requests, responses, offered inventory and execution results. The completed status means the request loop finished, not that the answer passed functional evaluation. These are synthetic/public traces, not private user conversations. This test does not measure UI rendering or streaming.