odysseus/plans/ODYSSEUS_TOOL_HARDENING_PLAN.md

# Odysseus Tool Runtime Hardening Plan

## Objective

Ship `odysseus-qwen3.5-tools-pre-heretic` with one compact, model-specific tool
runtime that supports realistic multi-turn use. Keep the existing RAG runtime
unchanged for every other model. Prove routing, execution, answer quality,
follow-ups, safety, rendering, latency, and native image/VL understanding through
the real 7011 Agent UI.

Current evidence is a baseline, not a ship claim:

- Corrected v2.5 + compact-v5 development is 327/344 raw (95.06%) and
  327/336 scorable (97.32%). Sealed blind is 311/344 raw (90.41%) and
  311/336 scorable (92.56%), with zero reasoning leakage.
- Notes, Skills, and Cookbook/admin clear 95% scorable blind. Calendar 87.5%,
  Shell/files 86.11%, and Tasks 87.5% remain below the 90% family ship floor.
- Compact-v5 hints improved Email, Search/HF quant, and Shell on development;
  a Calendar hint regressed and was rejected rather than shipped.

- Ten-family focused baseline: 19/20 functional and 20/20 routing/execution.
- Typo and cross-family read flows: 26/26 passed.
- Real use exposed untested write correction and search-to-fetch follow-ups.
- Email production access, browser interaction, search quality, and broader
  multi-turn mutations are not yet proven.
- Nine enabled chat-capable regular API models pass the ten-family read-only
  legacy-RAG baseline (90/90 combined). Their stricter typo/follow-up profile is
  178/180 turns: eight models are 20/20 and Luna is 18/20 due only to its
  misspelled Shell request. One pinned image-generation model is explicitly
  unsupported and two visible local models are currently offline.
- Native VL object/spatial recognition and reload follow-up pass. Exact OCR
  fails equally on the fine-tune and untouched 9B base and remains unresolved.
  PNG, JPEG, and WebP transport all pass.
- Reversible create/correct/API-verify/cleanup flows pass 6/6 across every
  stateful family.
- Search Web-toggle combinations pass 8/8 and the focused quality suite passes
  3/3. Production-path email account/inbox/referential reads pass 3/3.
- The latest regular-model regression is 90/90 across the nine enabled
  chat-capable API models, with zero failed model turns; two local endpoints
  remain offline and the image-only model is unsupported.
- The Epictetus OMLX endpoint was recovered after an unsupported
  `qwen3_5_mtp` model load wedged the server. Its supported Qwen 27B 4-bit
  model passes the ten-family real-7011 legacy-RAG smoke 10/10; the unsupported
  MTP artifact is recorded as a runtime limitation rather than a timeout.
- Fresh compact-v5 UI regressions pass stateful 6/6, Email 3/3, Search 3/3,
  private-browser 3/3, and VL workflow 3/3.
- The exact-model, family-scoped compact runtime now passes 20/20 direct and
  same-family turns across all ten families on the real 7011 Agent UI. A
  separate 36/36 robustness run passes misspellings, bounded repeats, browser
  and news continuation, ambiguous follow-ups, family switchbacks, and a
  greeting before a tool request.
- The mobile active-email editor path passes 1/1: `Write reply this email`
  offers and executes only `update_document`, mutates the open draft, and
  preserves its reply headers and quoted thread.
- The active-editor classifier now also covers short mobile wording without a
  pronoun (`Write reply` / `Draft a reply`) while explicit note, code, file, and
  new-object requests retain their own families. Whole-draft requests are bound
  to the sole offered `update_document` writer until one successful write, then
  tools are removed for the confirmation round. The deployed real-route email
  regression passes 3/3—including the exact unspecified `Write reply to this
  email` form—with one write, verified mutation, and preserved reply headers.
  Clean-v3 now also emits the established `doc_update` event and flattened
  document metadata on `tool_output`, so a successful database write updates
  the already-open editor instead of leaving stale UI beside a success message.
- The client now reuses the existing assistant bubble for `agent_step` round 1
  instead of replacing it before the first token. A real-7011 sampled
  greeting-to-Notes conversation passes 2/2 with stable first-round DOM
  identity; round 2+ remains the only continuation-bubble path.
- Clean-runtime metrics now expose provider-counted initial injected tokens,
  all-round input/output, TTFT, tok/s, schema count, agent rounds, and tool-call
  count. A real 7011 browser run passes 2/2 and visibly renders compact footers
  plus the full details popup; the sampled Notes turns streamed progressively.
- The deployed startup bottleneck was an unindexed quadratic transcript-FTS
  reconciliation. Live-database import fell from about 36 seconds to 0.54
  seconds; 7011 now answers in about 3 seconds after a controlled restart.
- A controlled identical-compact comparison already proves the fine-tune's
  accuracy benefit: 94.48% (325/344) versus the untouched base's 77.91%
  (268/344). Raw serving speed is effectively tied, so product speed comes
  from the compact contract and fewer failed/redundant rounds.
- A fully merged 10,000-row category-repair candidate reached 97.32% scorable
  development but only 92.26% scorable sealed blind. Calendar (87.5%), Tasks
  (87.5%), and Shell/files (86.11%) remained below the family floor, so it was
  rejected and not deployed. Compact-v4/full development A/Bs did not improve
  Calendar or Tasks over compact-v5; full-schema Shell also fell from 97.22%
  to 94.44%. This rules out compactness as the primary cause of the remaining
  blind gaps and supports keeping the compact contract.

## Non-negotiable architecture rules

1. Runtime selection follows exact model identity. The trained Odysseus model
   uses the clean compact runtime across endpoint aliases; all other models use
   legacy RAG. Add a regression test for both sides.
2. Resolve permissions, toggles, and available backends once per turn. Produce
   one immutable contract satisfying `required ⊆ offered ⊆ executable`.
3. Never offer a tool that the preview policy will categorically reject. Add a
   contract self-check covering every offered action/effect combination.
4. Follow-ups consume typed prior evidence: native call, result, success state,
   family, and object identifiers. Do not infer continuity from keyword RAG.
5. Contextual write authority may revise only a recently proven object in the
   same family. It may not authorize a new object, another family, a destructive
   action, or an external side effect.
6. The model chooses tools and valid arguments. The harness validates and
   executes; it does not silently substitute another family, rewrite arguments,
   fabricate success, or replace a failed tool with prose claiming completion.
7. One owner renders each turn: streamed prose or canonical structured output.
   Never both, and never expose hidden prompts or raw untrusted wrappers.
8. No exact-prompt production patches. A fix must name the failed layer, add a
   generic failing invariant test, and cover neighboring cases.

## Failure layers

Every failure is assigned to exactly one primary layer before code changes:

1. **Route:** wrong model runtime or endpoint identity.
2. **Contract:** required tool absent, forbidden tool present, or toggle drift.
3. **Model:** wrong/no tool or semantically wrong required arguments despite a
   correct contract.
4. **Policy:** valid proposed operation incorrectly allowed or denied.
5. **Execution:** canonical arguments, backend dispatch, timeout, or result
   envelope is wrong.
6. **Evidence:** result is empty, irrelevant, truncated badly, or insufficient.
7. **Answer:** model misstates or ignores valid tool evidence.
8. **Rendering:** duplicate, dump-at-end, missing structured output, or stopped
   stream.
9. **Performance:** startup, TTFT, tool latency, or oversized context.

Reports store aggregate category, relevant contract/tool metadata, timings, and
sanitized outputs. Do not copy private hidden benchmark prompts or create a log
dump that nobody can audit.

## Test matrix

Use the real authenticated 7011 Agent UI and the normal `preheret` picker alias.
Use `sft_alex_creator` for reversible writes. Never mutate the personal account
from an automated test.

### A. Every one of the ten families

For calendar, notes, email, tasks, documents, memory, skills, Cookbook/admin,
search/browser, and shell/files, test:

- direct request;
- natural misspelling;
- ambiguous same-family follow-up;
- switch to another family and back;
- no-tool greeting before the tool request;
- requested count/field limit;
- backend failure rendered truthfully;
- reload the permalink before a follow-up.

### B. Stateful mutation families

For notes, calendar, tasks, documents, memory, and skills:

- create → verify by API → referential correction → verify;
- create → list/read → correction → verify;
- typo correction such as name/date/title without repeating the family noun;
- correction after one unrelated conversational turn;
- destructive request is denied atomically;
- failed write never produces a success claim;
- cleanup deletes only the UUID-owned test artifact and verifies absence.

### C. Search and browser conversations

- search → summarize existing results without a new call;
- search → inspect one result with `web_fetch`;
- poor results → refine query once;
- insufficient evidence → say so without fabrication;
- Web toggle combinations `00`, `01`, `10`, and `11` across two turns;
- private browser open/snapshot/click only after its permission boundary is
  deliberately enabled and specified; do not smuggle it in via web search.

Grade source relevance, freshness, authority, and whether claims are supported,
not merely whether `web_search` was called.

### D. Email and shell

- Separate fixture accuracy from production connectivity. A fixture pass cannot
  promote production email health.
- Test account listing, inbox listing, reading, and referential follow-up against
  the configured production-like backend before enabling email actions.
- Shell remains toggle-gated. Test off/on transitions, canonical raw command
  dispatch, read-only output, and denial of network/destructive commands.

### E. Rendering and performance

- Assert first visible streamed token, monotonic DOM growth, one final answer,
  persistence/reload equality, stop behavior, and structured list rendering.
- Record request preparation, TTFT, tool duration, post-tool TTFT, total time,
  input/output tokens, and tool-result bytes.
- Diagnose the 30–40 second 7011 restart separately from inference latency.
- Bound large calendar/search results before replaying them into later rounds,
  while preserving IDs and fields needed for follow-ups.

### F. Image/VL recognition

- Attach real PNG, JPEG, and WebP images through the 7011 UI and verify the
  trained model receives native multimodal message content on its clean route.
- Test object recognition, visible text/OCR, spatial relationships, charts, and
  screenshots. Score required facts instead of stylistic wording.
- Test image → ambiguous follow-up, image → tool request, and tool result → image
  comparison without requiring the user to attach the same image again.
- Verify image references survive persistence and permalink reload without raw
  base64, local paths, or hidden wrappers appearing in chat output.
- Separate direct model vision from `inspect_media`, browser screenshots, and
  image generation. The harness must not silently substitute one for another.
- Compare the fine-tune with its base VL model on the same images to detect
  whether tool training regressed visual understanding.

### G. Regular-model legacy RAG and tool coverage

- Inventory every enabled non-Odysseus endpoint/model visible in 7011, including
  its provider, schema mode, native-tool support, context limit, and configured
  permissions. Do not assume every provider supports the same wire format.
- Assert that no non-Odysseus model enters the clean-v3 runtime. These models
  retain the regular RAG/tool loop and are repaired only in that owning path.
- For each model, test every tool family the effective user policy offers:
  direct request, misspelling, ambiguous follow-up, family switch, backend
  failure, and Web/Bash toggle transitions. Record unsupported families as an
  explicit capability limitation, not a silent pass.
- Test full schemas versus compact schemas only where both are valid for that
  model. Store the selected schema mode in every report.
- Verify provider-native tool calls, textual fallback parsing where required,
  canonical argument conversion, execution, evidence replay, and rendering.
- Group fixes by shared legacy-runtime or provider-adapter defect. Do not add
  model-name prompt exceptions when a transport, schema, or RAG ranking issue is
  responsible.
- Maintain a per-model compatibility matrix so adding or changing an endpoint
  cannot silently regress previously working tools.

## Fix protocol

For each failure:

1. Preserve the raw report and reproduce once on a fresh test session.
2. Identify the primary failure layer from the taxonomy above.
3. Add the smallest generic red test at that layer.
4. Fix the owning module or invariant—not the literal prompt.
5. Run the focused unit tests, the original scenario, two adjacent scenarios,
   and the affected family suite.
6. After a batch of category fixes, rerun the ten-family matrix and legacy-RAG
   isolation test. Do not rerun training unless the contract and harness are
   proven correct and failures remain model-owned.

If three failures share a layer, pause case-by-case patching and refactor that
layer before continuing.

## Execution phases

### Phase 1 — Make the runtime auditable

- Add a sanitized per-turn decision record: model runtime, contract, proposed
  calls, policy decisions with reason codes, executions, render owner, timings.
- Add startup/runtime provenance to the UI so a linked chat proves which harness
  handled it.
- Add the offered-versus-policy compatibility self-test.
- Correct stale preview documentation.

### Phase 2 — Build the conversation suite

- Extend the current Playwright verifier with reusable multi-turn scenarios and
  reversible artifact fixtures.
- Implement the matrix above, prioritizing search continuations and all
  stateful corrections because real usage already exposed those gaps.
- Run independent family groups in parallel, but serialize writes that share a
  backend or fixture account.
- Add a small versioned VL fixture set with locally generated, non-private
  images and deterministic answer keys.

### Phase 3 — Repair by architecture category

- Consolidate model-specific runtime selection in one function.
- Represent prior successful objects explicitly for referential follow-ups.
- Align tool capability classification, contract offering, and policy decisions.
- Standardize tool results into bounded envelopes with source/object IDs.
- Keep search refinement and evidence sufficiency generic.

### Phase 4 — Accuracy and speed comparison

- Compare the clean fine-tune with the base model using identical compact tools,
  prompts, toggles, backend state, and semantic scoring.
- Report functional accuracy, argument accuracy, unsupported success claims,
  TTFT, total latency, and tokens. Do not compare one model on full schemas and
  another on compact schemas.
- Only consider more SFT/RL for failures classified as model-owned after the
  harness audit.

### Phase 4B — Regular-model repair and verification

- Snapshot the enabled non-Odysseus model inventory.
- Run the legacy-RAG compatibility matrix in bounded parallel groups, respecting
  endpoint rate limits and shared backend write serialization.
- Fix shared harness/provider defects first, then rerun all affected models.
- Publish separate per-model scores and limitations; do not blend them into the
  Odysseus fine-tune score.

### Phase 5 — Ship gate

Ship only when:

- every family is at least 90% on sealed functional holdout;
- overall functional accuracy is at least 95%;
- realistic follow-up suite is at least 95%, with no repeated failure category;
- image/VL fixture accuracy does not regress materially from the base model and
  all attachment/follow-up/persistence flows pass;
- routing/execution and safety invariants are 100%;
- all reversible writes are API-verified and cleaned up;
- search quality and production email are reported separately and honestly;
- non-Odysseus models demonstrably retain legacy RAG;
- every enabled regular model has a complete tested-tool compatibility record,
  and every tool advertised as supported passes its functional checks;
- no hidden prompt leakage, duplicate rendering, or false success remains;
- pre-heretic passing weights and merged adapter backups remain recoverable.

## Immediate next batch

1. Expand VL fixtures to charts, screenshots, and image-to-tool turns;
   investigate the shared base-model OCR limitation without hiding it behind a
   silent external fallback.
2. Add deliberately permissioned private-browser open/snapshot/click checks;
   keep browser interaction unavailable when its boundary is not enabled.
3. Bring the two configured local regular models online and run their matrix.
4. Compare fine-tune versus untouched base with identical compact contracts,
   backend state, prompts, and timing instrumentation.
5. Run the sealed all-action holdout and prioritize failures by shared
   layer rather than by prompt.