mirror of
https://github.com/pewdiepie-archdaemon/odysseus.git
synced 2026-10-06 23:12:22 +02:00
65 lines
3.5 KiB
Markdown
65 lines
3.5 KiB
Markdown
# Odysseus Fix Workstreams
|
|
|
|
Evidence source: 626 historical `sft_alex_creator` contract sessions, deduplicated
|
|
to 54 flows and replayed through the current 7011 Agent runtime on 2026-09-11.
|
|
|
|
## Harness
|
|
|
|
- **Resolved — canonical item limits:** Notes and Calendar now honor explicit
|
|
limits such as “at most three” while retaining hidden expansion payloads.
|
|
- **Evaluate separately — shell/files:** two WebUI failures occurred because bash
|
|
is not consistently offered on follow-up. Shell/files belongs to the validated
|
|
`odysseus-native` workspace runtime; do not train the model on WebUI refusals.
|
|
- **Resolved — Calendar argument continuity:** referential repeats preserve the
|
|
preceding successful range; an explicitly new period still replaces it.
|
|
- **Resolved — evaluator:** historical one-turn probes are now retained, and the
|
|
judge treats HTML-comment expansion rows as hidden rather than visible overflow.
|
|
|
|
## Model / SFT
|
|
|
|
- **Remaining — browser evidence use:** the IKEA task routes correctly to
|
|
`private_browser`, but the model clicks opaque refs repeatedly and never extracts
|
|
a chair answer. This is the confirmed SFT repair class.
|
|
- **Remaining — identity attribution:** after successful Email → Calendar
|
|
switching, “Who are you?” can add the false phrase “trained by Google.” Keep
|
|
this as SFT data; do not restore a forced harness identity response.
|
|
- **Resolved in harness — Memory synthesis:** row evidence is compacted before the
|
|
observation cap instead of being truncated inside invalid JSON; Memory is 3/3.
|
|
- **Resolved in harness — Search recovery and source rendering:** equivalent empty
|
|
queries stop after two attempts, freshness words survive query shortening, and
|
|
exact source-link requests render the best relevant first-party result. Search is
|
|
15/16, with only the browser reasoning case above remaining.
|
|
- **Resolved in harness — Cookbook synthesis:** configured server rows use a
|
|
bounded evidence-owned renderer; Cookbook is 3/3.
|
|
|
|
Build repair examples from these behavior classes only after exact replay confirms
|
|
the failure with the intended runtime and rendering owner.
|
|
|
|
## Backend / Data
|
|
|
|
- The Python packaging query returned an unrelated OWASP result. The model reported
|
|
the failure honestly, but should attempt a bounded recovery before stopping.
|
|
- Synthetic email account servers are unavailable. The harness now renders that as
|
|
an outage and blocks invented message IDs; restore the fixture separately.
|
|
|
|
## Current measurement
|
|
|
|
- Historical source sessions: **626**
|
|
- Unique replay flows: **54**
|
|
- Initial judge result: **36 pass / 18 flagged**
|
|
- Post-renderer replay for Notes, Calendar, and switching: **14 pass / 2 flagged**.
|
|
- Final Notes + Calendar replay after continuity and judge fixes: **9 pass / 0 flagged**.
|
|
- Latest Search replay: **15 pass / 1 confirmed SFT failure**.
|
|
- Memory replay: **3 pass / 0 flagged**; Cookbook replay: **3 pass / 0 flagged**.
|
|
- Final WebUI-valid historical matrix: **49 pass / 2 confirmed SFT failures = 96.1%**.
|
|
|
|
Artifacts:
|
|
|
|
- Full run: `tmp/odysseus-conversation-qa/run-20260911-092930.json`
|
|
- Post-renderer replay: `tmp/odysseus-conversation-qa/run-20260911-093333.json`
|
|
- Final Notes + Calendar replay: `tmp/odysseus-conversation-qa/run-20260911-093752.json`
|
|
- Latest Search replay: `tmp/odysseus-conversation-qa/run-20260911-100239.json`
|
|
- Memory replay: `tmp/odysseus-conversation-qa/run-20260911-095320.json`
|
|
- Final WebUI-valid matrix: `tmp/odysseus-conversation-qa/run-20260911-101229.json`
|
|
- Deduplicated queue: `tmp/odysseus-conversation-qa/historical-sft-alex-queue.json`
|