jpmschweitzer/odysseus
1
0
Fork 0
mirror of https://github.com/pewdiepie-archdaemon/odysseus.git synced 2026-10-12 09:52:19 +02:00
Code Issues Packages Projects Releases Wiki Activity

Squash Odysseus development history

Browse Source
This commit is contained in:
pewdiepie-archdaemon
2026-09-11 06:04:19 +00:00
parent c9dd68d890
commit 84aa9a91de
871 changed files with 265870 additions and 27854 deletions
Download Patch File Download Diff File Expand all files Collapse all files
website/original-harness-capability-policy.md
+66
View File
@@ -0,0 +1,66 @@
# Original Harness Capability Policy
Odysseus Original is the canonical product and the only agent orchestrator.
Specialized runners are evidence sources, not merge targets.
## Architecture
The supported shape is:
1. One Original agent loop owns prompting, tool selection, policy, evidence,
recovery, compaction, and completion.
2. Capability contracts describe what the current environment can do.
3. Execution bridges or MCP servers perform work in the owning environment.
4. Personal tools remain available in interactive sessions but are excluded
from external terminal contracts unless explicitly provided.
## Capability Decisions
| Capability | Decision | Canonical form |
| --- | --- | --- |
| Workspace execution | Keep | Request-scoped `AgentExecutionBridge`, moving toward a Workspace MCP boundary |
| File mutation | Keep | `write_file`, `edit_file`, and `apply_patch` with evidence recording |
| Process recovery | Keep | Bounded polling, timeout, exit status, and stale-process recovery behind the execution contract |
| Repeated-action recovery | Keep | Detect identical calls, unchanged successful results, and failed batches separately |
| Completion | Keep | Required artifacts and executable verifier evidence; no evaluator-specific shortcuts |
| Context control | Keep | Deterministic compaction with retained user evidence and bounded tool output |
| Search | Keep | Existing private SearXNG path |
| Browser interaction | Keep | Existing private browser for rendered pages, sessions, clicks, and screenshots |
| Personal tools | Keep, isolated | Separate personal capability surface; never substitute editor documents for workspace files |
| Tool discovery | Keep | Existing tool RAG and capability-aware selection |
| Media ingress | Keep | Bounded image, audio, video-frame, and document ingestion with hashes and trace metadata |
| Visual verification | Candidate | Generic source grounding and rendered-artifact checks, gated by available media capabilities |
| Document access | Candidate | Structured PDF and Office extraction through `read_file` or a Workspace MCP implementation |
| Skills | Keep, generic only | Procedures for artifact completion, recovery, verification, development, media evidence, and research |
## Rejected Merges
Do not merge:
- another agent loop, submit controller, or conversation state machine;
- category names, task identifiers, fixed workspace paths, expected answers,
grader behavior, scoring rules, or evaluator prompts;
- per-category turn thresholds or phase transitions;
- terminal multiplexer control when the environment already exposes direct
process execution;
- repeated prompt guards that do not add new executable evidence;
- browser or search replacements for capabilities Original already owns.
## Merge Gate
A mechanism may enter Original only when all of these are true:
1. It is useful outside the evaluation that revealed it.
2. It is selected by an environment capability, not a task name or path.
3. It fits inside the Original loop or behind an execution boundary.
4. Its success or failure produces traceable evidence.
5. It has focused regressions and a representative held-out run.
6. It removes or contains complexity instead of adding an overlapping mode.
## Current Priority
The next capability work is reliable structured document access. Failed
binary-file inspections must remain failures; they must not be interpreted as
unchanged evidence or trigger early artifact synthesis. Once grounding is
reliable, add generic rendered-artifact verification without importing any
specialized task phases.
website/photo-editor-audit.md
+346
View File
@@ -0,0 +1,346 @@
# Photo Editor Product Audit
Date: 2026-08-30
## Executive Verdict
Odysseus already has the structure of a real layered raster editor. It is not a
mockup: layers, nested groups, masks, selections, retained text, blending,
history, document geometry, project recovery, controlled export, and several AI
workflows operate on an editable document model.
It is **roughly 78% of a dependable everyday photo editor**, but only **about
35% of a professional Photoshop/Photopea alternative**. Those are deliberately
separate scores. The first target needs complete, trustworthy common workflows;
the second also needs non-destructive sources and filters, color management,
professional file interchange, vector/path tooling, automation, and scale.
The main product gap is no longer basic layer infrastructure. It is the absence
of a strong photo-correction and retouching workflow, combined with insufficient
proof that every existing operation preserves pixels, masks, text, and project
state across desktop and touch.
## Current Scorecard
| Area | Everyday readiness | Current assessment |
| --- | ---: | --- |
| Canvas navigation and precision | 90% | Pan/zoom, fit and 1:1, rulers, guides, grid, snapping, and numeric geometry are present |
| Layers and compositing | 96% | Raster/text layers, nested groups, clipping, masks, blend modes, multi-select, locks, subtree reorder, and shared transforms are strong |
| Selections and masks | 97% | Marquee, lasso, wand, SAM, Quick Mask, named selections, affine selection transform, and linked/unlinked layer-mask positioning work |
| Text | 65% | Retained text exists; paragraph layout, tracking, font status, stronger hit testing, and mobile proof do not |
| Painting and retouching | 70% | Brush, eraser, clone, source-free and sampled healing, smudge, retained linear/radial gradients, fill, AI inpaint, background removal, sharpen, eyedropper, dodge/burn, and brush presets exist; advanced raster retouching remains |
| Photo correction | 65% | Retained levels, curves, exposure, white balance, hue/saturation, vibrance, shadows/highlights, color balance, selective color, gradient map, brightness/contrast, and histogram controls exist; camera/lens correction remains |
| Non-destructive editing | 60% | Text and adjustment metadata are retained, image imports and pasted selections create source-backed placed layers, raster layers can be converted to sources, and transforms/replacement preserve source pixels; linked instances and richer source editing remain |
| Save, recovery, and export | 88% | Versioned project recovery and PNG/JPEG/WebP export are strong; pixel-equivalence, metadata, and color-profile policies remain |
| Mobile editing | 45% | Responsive UI, touch navigation, and non-overlapping tool/layer sheets are tested; core editing gestures still need broader coverage |
| Performance and color fidelity | 30% | Safety limits exist; workers/tiles, stress evidence, ICC handling, and high-bit-depth editing do not |
## Verified Baseline
The current implementation was checked on 2026-08-30:
- **72 editor-focused unit tests pass.** Coverage includes the v12 document
model, migrations, validation, geometry, selections, masks, mask offsets,
groups, clipping, multi-select, transforms, retained text, history limits,
guides, and export settings.
- **81 Chromium Playwright workflows pass.** They cover the layered core path,
group masks and reorder, clipping, independent locks, multi-selection,
nested groups, shared transforms, linked/unlinked mask movement and reopen,
exact draft reopen, export dimensions, project recovery, Quick Mask, precise
marquee geometry, transformed selections, named selections, and mobile touch
crop, selection, brush, and transform gestures, plus mobile mask persistence,
movement, and PNG export.
- Mobile editor refresh now has a dedicated recovery gate that restores the
active draft, editor tab, saved status, canvas, and layer stack.
- The adjustment popup has a 320px phone-width regression gate: slider rows stay
inside the sheet and the Apply/Cancel actions remain reachable.
- The Gradient Map adjustment has a dedicated 320px gate: its color controls
collapse to one responsive column and remain inside the adjustment sheet.
- Every retained adjustment popup now has a 320px matrix check for viewport
bounds, horizontal overflow, and reachable Apply/Cancel actions.
- The adjustment export matrix compares decoded PNG pixels against the visible
composite for every retained adjustment family, including Curves and
Gradient Map.
- The adjustment compositing workflow also compares export pixels when a
retained adjustment is clipped, masked, opacity-modified, undone/redone, and
reopened from a saved draft.
- The retained-adjustment export workflow passes in both Chromium and Firefox;
the Playwright harness now supports selecting `chromium`, `firefox`, or
`webkit` through `PHOTO_EDITOR_E2E_BROWSER`.
- Retained-effect previews now fall back cleanly when a worker cannot be
created, and stale worker errors no longer trigger an unnecessary full-size
synchronous render.
- The mobile layer-sheet workflow also verifies touch mask editing, undo/redo,
and mask persistence after browser reload.
- The merge-fidelity workflow compares the rendered composite before and after
Merge All with retained effects and adjustment layers, ensuring those edits
are baked into the resulting raster instead of being dropped.
- The grouped-effects workflow compares a retained group effect against the
exported PNG pixel-for-pixel and verifies its parameters survive draft reopen.
- The export preview workflow verifies matte pixels are cleared when switching
back to transparency.
- The export dialog workflow restores focus to the Save control after closing,
including the menu-launched export path.
- The core workflow now compares SHA-256 digests before and after draft reopen,
requires exact decoded pixels for native PNG export, and bounds premultiplied
pixel error for resized PNG output.
- The linked-mask regression gate verifies that an unlinked mask remains at a
fixed document position when its parent layer moves or transforms. Brush,
Wand, and Fill now resolve independently moved masks from the same origin.
This is solid Chromium/mouse evidence with focused touch coverage for crop,
selection, brush, and transform gestures, plus a focused Firefox export check.
It does not establish full Safari/Firefox compatibility, complete touch
reliability, large-document responsiveness, metadata or ICC fidelity, or
pixel-equivalent exports for every format.
## What Already Works
| Capability | Implementation status |
| --- | --- |
| Layered document | Raster and retained-text layers, nested groups, opacity, visibility, 16 Canvas2D blend modes, clipping, duplicate, merge, and hierarchy-aware reorder |
| Layer control | Multi-select, shared-bounds transforms, full/pixel/transparency/position locks, group masks, paintable layer masks, and linked/unlinked mask position |
| Selection model | Rectangle/ellipse marquee, lasso, Magic Wand, SAM, new/add/subtract/intersect, animated boundary, exact geometry, move/scale/rotate/flip, nudge, invert, Quick Mask, reselect, and named selections |
| Paint and AI | Brush, eraser, clone stamp, selection fill, inpaint, background removal, SAM, harmonize, upscale, denoise, face enhancement, and style operations |
| Geometry | Crop, image resize, canvas resize, rotate, flip, move, transforms, rulers, guides, grid, and snapping |
| Recovery | Validated v12+ project format, explicit migrations, bounded autosave/history, partial corrupt-project recovery, and exact server-draft reopen |
| Delivery | Previewed PNG/JPEG/WebP export with quality, dimensions, aspect lock, transparency/matte, filename, and gallery-copy flow |
## Release Blockers
These are the gaps that prevent calling the editor dependable today.
### P0: Trust Existing Operations
1. **Pixel-equivalent export proof is still too narrow.**
The core layered workflow now proves exact native PNG pixels and bounded
resized-PNG error, and format metadata is covered for JPEG/WebP. Grouped
blending, clipping, matte pixels, and color-profile behavior still need the
same evidence.
2. **Touch editing is only partially release-tested.**
Pan and pinch primitives exist, and paint, selection, crop, and transform
now have focused browser coverage, but broader mobile overlap and
inaccessible-control assertions are still needed around edge cases.
3. **Operation contracts are incomplete.**
Every destructive operation needs an explicit test matrix for raster layers,
retained text, linked/unlinked masks, group masks, selections, locks, clipped
layers, and multi-selection. The independently positioned mask bug found in
Brush/Wand/Fill demonstrates why shared coordinate helpers are required.
4. **Large-document behavior is bounded, not proven.**
Full-canvas Canvas2D compositing and RGBA history snapshots have hard limits,
but there is no stress suite, cancellation contract, worker/offscreen path,
memory telemetry, or degraded-preview strategy.
5. **Color and metadata behavior is undefined.**
Import relies on browser decoding and export writes a new bitmap. Users are
not told whether ICC, EXIF/IPTC, orientation, DPI, or location metadata is
honored, normalized, preserved, or stripped.
## Missing Everyday Features
These have higher value than adding more isolated AI tools.
### P1: Photo Correction
- Composite and per-channel histogram with clipping warnings
- Adjustment presets, reset, and non-destructive before/after compare are implemented
- Actual adjustment layers that affect content below, can be clipped/grouped,
and have their own masks; the current per-raster-layer stack is not equivalent
### P1: Retouching and Paint Ergonomics
- Richer multi-stop and radial gradient controls for raster content
- Healing Brush distinct from source-free Spot Healing and Clone Stamp
- Blur and Smudge brushes
- Content-aware fill workspace built on the existing inpaint capability
### P1: Geometry, Text, and Layer Workflow
- Image Size supports staged pixel/percentage resizing, aspect locking, and interpolation choice
- Canvas Size supports staged pixel/percentage bounds with anchor control
- Align and distribute selected layers (implemented in the multi-selection bar)
- Optional canvas auto-select for visible layers (group-level hit testing remains)
- Paragraph text boxes, tracking, vertical alignment, font loading/fallback
status, and more reliable text hit testing
- Mask density, non-destructive feather, invert, disable, link/unlink position
behavior, mask-only inspection, and applying true layer masks are implemented;
applying a group mask still requires an explicit flattening workflow
## Missing Professional Features
These define the gap to Photopea/Photoshop rather than blocking a credible v1.
### P2: Non-Destructive Core
- Embedded source layers / smart-object equivalent (toolbar/gallery/drop imports, pasted selections, and manual raster-to-source conversion now exist; richer source editing remains)
- Editable transform matrices that preserve original pixels through repeated
scale and rotate operations are now covered for placed and converted raster layers
- Linked instances and replace-source workflow
- Editable filter stacks with visibility, opacity, reorder, masks, and cached
previews
- Blur, sharpen, denoise, high pass, lens correction, and perspective correction
as retained filters
- Non-destructive transform masks, including perspective/warp later
This is the most important architectural gap. Photopea's Smart Objects retain a
separate source so repeated transforms can be recalculated without cumulative
loss, and its Smart Filters remain editable. Krita similarly models transform
and filter masks as non-destructive layer children.
### P2: File and Color Fidelity
- Explicit sRGB conversion and ICC profile awareness
- 16-bit processing before considering 32-bit/HDR
- EXIF/IPTC preservation or intentional stripping controls
- Reliable HEIC/TIFF handling and a RAW handoff/development path
- PSD import/export feasibility and a published compatibility matrix
- DPI/PPI and print-size metadata
### P2: Vector and Layout Work
- Shape layers for rectangle, ellipse, line, and custom paths
- Pen tool, editable Bezier paths, vector masks, and path-based selections
- Layer styles such as stroke, shadow, glow, and overlays
- Channels panel and channel operations
- Artboards only if multi-output design work is a product goal
### P3: Production Workflow
- Actions/macros, batch processing, and batch export
- Templates, reusable presets, and layer comps
- Soft proofing, gamut warning, and print output
- Plugin/filter extension surface
- Version history beyond the local bounded undo stack
## UX Audit
1. **The toolbar prioritizes AI before correction fundamentals.** Healing,
Eyedropper, Gradient, and Curves should be as discoverable as SAM and Inpaint.
2. **The distinction between masks is still cognitively expensive.** Selection,
Quick Mask, AI masks, layer masks, and group masks need consistent names,
thumbnails, active states, and properties rather than relying on sub-row
position alone.
3. **Properties are fragmented.** Tool controls, layer adjustments, transform
values, and mask settings should use one contextual Properties area. This
reduces modal popups and makes the selected target obvious.
4. **The editor needs clearer destructive-action signaling.** Blur, rasterize,
merge, and applied transforms should say when source pixels will be replaced,
with a one-step duplicate/convert-to-source option where appropriate.
5. **Mobile needs a deliberate mode.** Shrinking desktop controls is not enough;
canvas-first editing needs bottom-sheet properties, stable touch targets,
stylus behavior, and predictable two-finger navigation while a tool is active.
The tool and Layers sheets now claim the viewport exclusively, and mobile
layer/mask rows have stable non-scrolling layouts. Paint, crop, transform,
mask save/reopen, mask movement, and layer reorder gestures are covered;
mobile export now has viewport and download coverage, while broader
multi-tool touch workflows remain.
## Credible V1 Definition
A dependable everyday editor is reached when all of these workflows pass as a
single checked-in desktop and touch gate:
1. Import a common web image, correct exposure/color, retouch a blemish, crop,
resize, add text, and export at a chosen size and quality.
2. Build a layered composition with nested groups, clipping, a linked mask and
an independently positioned mask, then close and reopen without state loss.
3. Apply, cancel, undo, and redo each geometry/filter operation without changing
unrelated pixels or retained metadata.
4. Compare the flattened visible composite, reopened composite, and exported
bitmap within a documented pixel tolerance.
5. Complete the same core workflow with mouse and touch, with no inaccessible
controls, accidental page gestures, or silent partial edits.
6. Reject oversized, corrupt, or unsupported documents with a useful warning
while retaining every recoverable layer.
## Recommended Build Order
### Milestone 1: Reliability Gate
- Expand composite-versus-export pixel tests across groups, clipping,
adjustments, transparency/mattes, JPEG, and WebP.
- Add the operation/target matrix for masks, text, groups, clipping, locks, and
multi-select.
- Add Chromium touch/mobile workflows and basic Firefox/WebKit smoke coverage.
- Centralize document/layer/mask coordinate conversion.
- Define import/export color and metadata policy.
**Exit:** a mixed 20-edit document survives undo/redo, close/reopen, and export
with equivalent visible pixels and editable state.
### Milestone 2: Everyday Photo Workflow
- Extend raster retained gradients with richer stop editing
alongside the existing Eyedropper, Gradient, Healing, and Smudge tools.
- Add Curves, Exposure, White Balance, Vibrance, and Shadows/Highlights.
- Finish mask properties, Image/Canvas Size dialogs, text layout, alignment, and
brush presets.
**Exit:** crop, correction, blemish removal, annotation, transparent assets, and
social-image composition can be completed locally without another editor.
### Milestone 3: Non-Destructive Editing
- Introduce source layers and editable transform matrices.
- Promote adjustments into real adjustment layers.
- Add retained filter stacks and filter masks.
**Exit:** normal experimentation no longer requires manually duplicating layers
to protect the original pixels.
### Milestone 4: Interchange and Scale
- Add worker/offscreen rendering, cancellation, telemetry, and stress tests.
- Implement color-profile and metadata policy.
- Run a PSD/HEIC/TIFF/RAW feasibility spike and publish compatibility limits.
## Architecture Direction
The current `layer.canvas + optional metadata` model is reaching its limit.
Before adding adjustment layers, vectors, or source layers, move to a typed
document contract:
```text
Layer = RasterLayer | TextLayer | GroupLayer | AdjustmentLayer | SourceLayer
common: id, name, visible, opacity, blendMode, locks, masks, transform
raster: mutable pixel surface
text: content and typography
group: ordered child ids
adjustment: operation, parameters, clipping, mask
source: immutable embedded pixels, editable transform, filter stack
```
Rendering, serialization, history, geometry, duplicate, merge, and thumbnails
should dispatch through that contract. Coordinate conversion should likewise be
centralized around document, layer, mask, and viewport spaces instead of being
reimplemented inside tools.
History should evolve toward commands plus periodic checkpoints. Bounded full
RGBA snapshots prevent runaway memory today, but remain expensive for large
documents and awkward for retained non-raster layer types.
## Benchmark Basis
This audit uses mature editors as behavior references, not as a requirement to
clone every feature:
- [Photopea feature map](https://www.photopea.com/learn/)
- [Photopea masks and mask properties](https://www.photopea.com/learn/masks)
- [Photopea adjustment layers and Smart Filters](https://www.photopea.com/learn/adjustments-filters)
- [Photopea Smart Objects](https://www.photopea.com/learn/smart-objects)
- [Krita transform masks](https://docs.krita.org/en/reference_manual/layers_and_masks/transformation_masks.html)
- [Krita non-destructive filters](https://docs.krita.org/en/reference_manual/filters.html)
- [Photoshop color-adjustment workflow](https://helpx.adobe.com/photoshop/using/color-adjustments.html)
- [Photoshop Content-Aware Fill](https://helpx.adobe.com/photoshop/desktop/apply-painting-techniques/fill-objects-selections-layers/content-aware-fills.html)
## Immediate Next Slice
Extend the new fidelity/mobile gate across **grouped blending, adjustments,
and real touch paint/transform gestures** next. Selection copy, cut, paste, and
single-step undo now have checked coverage.
After that, implement **Gradient + Healing Brush** as one vertical
everyday-photo slice rather than adding disconnected controls.
website/sft-expansion-launch-ledger-20260830.md
+220
View File
@@ -0,0 +1,220 @@
# SFT Expansion Launch Ledger
## Immutable Inputs
- Approved seed manifest: `tmp/sft_expansion_20260830/frozen_v1/seed_manifest.json`
- Approved seed turns: `tmp/sft_expansion_20260830/frozen_v1/approved_trace.jsonl`
- Environment inventory: `tmp/sft_expansion_20260830/environment_inventories.json`
- Seed baseline: 561 sessions, 1,372 approved turns
- Expansion selection: `tmp/sft_expansion_20260830/selected_200_manifest.json`
- Selection size: 50 owner-bound families, projected to 200 environment cases
## Acceptance Policy
1. Generate only from inventory facts and marker-scoped reversible mutations.
2. Reject unsupported tools, temporal contradictions, vague calendar mutation times, and compound mutation-plus-verification prompts.
3. Run each turn through the real 7011 agent surface with Kimi K3.
4. Require the expected tool and manager action, successful tool output, a nonempty answer, and no internal narration or false-unavailability claim.
5. Restore owner state after every case and delete mechanically failed sessions.
6. Review every mechanically passing session with DeepSeek.
7. Retain only `keep` verdicts scoring at least 80. Delete all other sessions and trace rows; repair by regenerating and replaying, never by inventing missing tool evidence.
8. Preserve the source seed family's train/validation/test split.
## Pilot Record
- Harness regression fixed: contextual calendar entries no longer route to `manage_tasks`.
- Harness regression fixed: contextual calendar reads require fresh `manage_calendar` evidence.
- Runner regression fixed: a manager tool name alone no longer passes; create/list/update/delete actions are checked.
- Generator regression fixed: calendar mutations require an exact time or `ask_user`.
- Unsafe global skill-directory rollback replaced with marker-scoped cleanup.
- Omar calendar lifecycle: deterministic pass, DeepSeek `keep`, 96/100, retained.
- Ambiguous Omar predecessor: DeepSeek `repair`, 60/100, deleted from app and trace.
- Maya ambiguous-date lifecycle: deterministic reject, deleted before review.
## Current Corpus
- Build: `tmp/sft_expansion_20260830/corpus_v2`
- Train: 994 turns
- Validation: 157 turns
- Test: 135 turns
- Sessions: 562
- Retained expansion sessions: 1
- Seed-family leakage: 0
## Clean Source Baseline
- Immutable source remains unchanged: `tmp/sft_expansion_20260830/frozen_v1/approved_trace.jsonl`
- Deterministic hygiene input: 561 sessions, 1,372 turns
- Deterministic hygiene result: 421 sessions, 859 turns retained; 140 sessions rejected
- Full DeepSeek semantic review: 336 keep, 77 repair, 8 delete
- Kimi repair result: 73 validated repaired sessions; 4 additional sessions excluded
- Clean derivative: `tmp/sft_expansion_20260830/clean_v2/approved_trace.jsonl`
- Clean derivative size: 409 sessions, 834 turns
- Post-repair deterministic recheck: 409/409 sessions pass
- Style contract: `docs/sft-style-contract.md`
- Assembly report: `tmp/sft_expansion_20260830/clean_v2/assembly_report.json`
- Semantic verdicts: `tmp/sft_expansion_20260830/clean_v1/deepseek_verdicts.jsonl`
The clean source does not yet support the intended balanced expansion by itself.
Notes has one clean source session, tasks has two, and session-management tools
are absent. Add and review natural live workflows for those domains before the
200-case cross-environment run.
## Commands
```bash
.venv/bin/python scripts/generate_sft_environment_expansion.py \
--seed-manifest tmp/sft_expansion_20260830/selected_200_manifest.json \
--inventories tmp/sft_expansion_20260830/environment_inventories.json \
--out tmp/sft_expansion_20260830/generated_200_cases.json \
--workers 4 --timeout 180 --retries 1
.venv/bin/python scripts/run_sft_environment_expansion.py \
--cases tmp/sft_expansion_20260830/generated_200_cases.json \
--out tmp/sft_expansion_20260830/generated_200_run.json
.venv/bin/python scripts/review_sft_environment_expansion.py \
--run tmp/sft_expansion_20260830/generated_200_run.json \
--out tmp/sft_expansion_20260830/generated_200_review.json
.venv/bin/python scripts/build_sft_expansion_splits.py \
--manifest tmp/sft_expansion_20260830/frozen_v1/seed_manifest.json \
--approved-trace tmp/sft_expansion_20260830/frozen_v1/approved_trace.jsonl \
--review tmp/sft_expansion_20260830/pilot_atomic_exact_v1.review.json \
--review tmp/sft_expansion_20260830/generated_200_review.json \
--out-dir tmp/sft_expansion_20260830/corpus_expanded
```
# All-Tools No-Thinking Validation Update
## Matched-control finding
The previously reported email LoRA speed advantage does not reproduce when the
raw base model and LoRA are served in the same vLLM process with the same GPU,
tool payload, prompt, cache settings, and `enable_thinking=false`.
- Original 24-case email suite, 14 compact tools:
- raw Qwen3.5-9B base: 23/24, 0.914s average
- all-tools iteration-2 LoRA: 23/24, 0.874s average
- Earlier base result of 9.655s was therefore confounded by serving conditions.
- The validated latency win is **thinking off + compact schemas**. LoRA has not
yet shown a material independent speed gain.
## All-tools iterations
- Iteration 1: 365 train sessions, 40 validation sessions, 92 steps, LR 5e-7,
one epoch. Eval loss 0.8364.
- Iteration 2: same corpus, 184 steps, LR 2e-6, two epochs. Eval loss 0.7433.
- Matched 40-case held-out routing result for base, v67, and iteration 2 was
effectively identical: 85% exact tool selection, 85% required-field
presence, 70% constrained required-argument accuracy, and no reasoning
leakage.
- Iteration 2 adapter weights differ from v67 (relative L2 delta 1.02%), so the
tie is not caused by an unchanged adapter file.
## Decision
Do not promote iteration 1 or 2 as a speed improvement. Before another train:
1. Build a larger, balanced, genuinely unseen eval across sparse tool families.
2. Expand sparse training families with multi-turn, context-dependent traces.
3. Evaluate whether LoRA preserves accuracy under more aggressive schema
pruning than base; that is the plausible route to an indirect latency win.
4. Keep no-thinking routing and compact schema selection as harness features,
independent of model promotion.
## Critical Qwen3.5 LoRA Serving Failure
### Symptom
vLLM 0.22 accepted `--enable-lora`, listed each adapter under `/v1/models`, and
logged `Loaded new LoRA adapter`, but Qwen3.5 text-tool adapters were not applied
to inference. Base, v67, iteration 3, and iteration 4 produced byte-identical tool
calls across 177 cases. A direct log-probability probe also returned numerically
identical values for base and every dynamic adapter.
Do **not** treat model registration, successful HTTP responses, different adapter
files, or latency differences as evidence that a LoRA is active.
### Minimal activation check
Before any benchmark, send the same deterministic request to base and adapter
models with `temperature=0`, `logprobs=true`, and thinking disabled. Compare the
token log-probabilities, not only generated text. Identical probabilities across
several prompts mean the adapter path is inactive.
Observed inactive result:
```text
base: cob -0.0484729, alt -0.0000684
v67: cob -0.0484729, alt -0.0000684
iteration4: cob -0.0484729, alt -0.0000684
```
Disabling prefix caching and enabling eager execution did not fix this.
### Merge trap
The tool adapters were trained with `AutoModelForCausalLM`, which gives Qwen3.5
text keys under:
```text
base_model.model.model.layers.*
```
The old merge script auto-selected `AutoModelForImageTextToText`, whose language
tower expects:
```text
base_model.model.model.language_model.layers.*
```
PEFT warned about missing adapter keys, then still wrote a model. That output was
effectively unmodified. A successful `save_pretrained` is therefore not proof of
a successful merge. Treat any missing-adapter-key warning as a hard failure.
Merging with the causal loader applied the adapter, but produced a
`Qwen3_5TextConfig` model that vLLM's production Qwen3.5 multimodal loader refused.
### Working merge path
`scripts/merge_hf_lora.py` now supports the production-safe bridge:
1. Load the full Qwen3.5 model with the image/multimodal loader.
2. Remap causal adapter keys from `model.layers` to
`model.language_model.layers`.
3. Exclude the visual tower from PEFT target matching.
4. Merge and save the full model plus tokenizer and processor metadata.
```bash
python scripts/merge_hf_lora.py \
--model-loader image \
--remap-qwen35-causal-adapter \
--base /mnt/HADES/models/Qwen3.5-9B \
--adapter /path/to/final_adapter \
--output /path/to/merged-full-bf16
```
Verify there are no missing adapter keys, serve the merged model as a standalone
model, and repeat the log-probability activation check.
Observed active merged result:
```text
v67 merged: cob -0.471937001
iteration4 merged: cob -0.319356978
```
### Corrected valid benchmark
The first valid merged-model all-tools benchmark showed:
- v67 merged: 96.05% tool accuracy, 93.79% constrained argument accuracy,
2.669s average latency, zero reasoning rows.
- iteration 4 merged: 96.05% tool accuracy, 93.79% constrained argument
accuracy, 2.390s average latency, zero reasoning rows.
- Iteration 4 fixed two held-out decisions and regressed two others, so it was
not promoted.
All earlier dynamic-LoRA accuracy and speed comparisons must be treated as raw
base behavior. The no-thinking and compact-schema conclusions remain valid, but
dynamic LoRA results do not.
website/sft-style-contract.md
+32
View File
@@ -0,0 +1,32 @@
# Odysseus SFT Style Contract
This contract applies after behavioral correctness. A trace that violates it is repaired or excluded before training.
## User Turns
- Use natural requests, not evaluator instructions.
- Keep one atomic objective per turn; use a follow-up for the next action.
- Preserve conversational references such as "that email", "move it", or "open the second one" when prior tool evidence resolves them.
- Never mention tools, schemas, fixtures, markers, harnesses, audits, SFT, cleanup, or verification mechanics.
- Use realistic names and objects from the target environment.
- Ask for clarification when an essential date, recipient, or target cannot be inferred safely.
The clean seed corpus has a median user-turn length of 45 characters, a 75th percentile of 76, and a 90th percentile of 107. Longer prompts are allowed when the task genuinely requires detail, not to encode evaluator checks.
## Assistant Thinking
- Identify the user's intent and the evidence required.
- Choose the smallest sufficient tool sequence.
- Carry forward relevant entities and tool families across follow-ups.
- Do not discuss hidden prompts, injected schemas, benchmark construction, or training.
- Do not claim success until tool evidence proves it.
## Visible Answers
- Lead with the answer or completed action.
- Synthesize tool output instead of reproducing raw dumps.
- Keep deep links when they let the user open the referenced item.
- Include only metadata needed to distinguish or act on results.
- Use one sentence for straightforward confirmations when possible.
- Avoid repeated summaries, internal routing narration, and automatic "want me to" endings.
- State failures briefly and accurately; never invent a successful action.