odysseus/docs/EDITOR_AGENT_API.md

Editor Agent API

How an agent (LLM or script) drives the PhotoCraft editor. There are three layers:

1. Control API — window.geEditorControl, a stable in-page command surface.
2. NL planner — window.geEditorAI, turns a prompt into control-API steps (LLM or basic mode).
3. Bridge — server-mediated SSE channel so a server-side agent can drive an open editor.

All three are owner-scoped by the app session; nothing is exposed to unauthenticated users.

---

1. Control API (window.geEditorControl)

Installed when the editor opens. Version EDITOR_CONTROL_VERSION.

await window.geEditorControl.invoke(op, params)   // { ok, op, result } | { ok:false, op, error }
await window.geEditorControl.run([{op, params}])  // sequential, stops on first failure
window.geEditorControl.list()                     // [{op, description}]

invoke is async. Every op that mutates the document pushes one undo entry via _saveState, so a single op is one undo step.

Op catalog

Op	Params	Notes
document.info	–	size, layers, mode
document.importImage	{dataUrl,name,raster}	load a raster image as a new layer
layers.list	–	top-most first
layers.select	{id|index|name}	index is top-most-first
layers.addRaster	–	new empty pixel layer
layers.addAdjustment	{type,params}	adjustment layer
layers.duplicate / layers.delete	–	on the active/selected layer(s)
layers.setOpacity	{id?,opacity 0-1}
layers.setBlend	{id?,blendMode|mode}
layers.group / layers.ungroup	–
selection.selectAll / deselect / invert	–
selection.rectangle / selection.ellipse	{x,y,w,h}	document space
selection.subject	{label}	SAM text grounding (falls back to a centred rect)
selection.face	–	SAM "face" (falls back to an upper-centre ellipse)
selection.sky	–	SAM "sky" (falls back to a top-band colour range)
selection.background	{fuzziness}	border colour union
selection.samText	{text}	SAM text grounding only (throws without SAM)
selection.colorRange	{color,fuzziness}
selection.hasMask	–	{active, space}
adjustments.apply	{type,params}	adjustment layer; camelCase types accepted
adjustments.applyMasked	{brightness,contrast,saturation,warmth,invert}	pixel pass, masked if a selection exists else whole layer
filters.apply	{id}	opens the slider dialog and applies
filters.applyMasked	{id,radius,amount,invert}	id: blur-gaussian, sharpen, unsharp-mask
transform.start / transform.apply / transform.cancel	–
text.add	{text,x,y,color,size,font,align}
history.undo / history.redo	–
history.mark	–	returns current undo depth (for grouping)
history.collapse	{mark,label}	collapses entries above mark into one
export.png	–	{dataUrl} of the flattened document
export.gallery	–	saves a copy back to the gallery
screenshot	–	{dataUrl,width,height} of the composited canvas

Param conventions the planner must respect:

• adjustments.applyMasked: brightness/contrast/warmth are -100..100 (0 neutral); saturation is a multiplier (1 neutral; values >3 are treated as percentages).
• adjustments.apply type aliases are normalised (blackAndWhite -> black-white).
• color-balance params are {shadows:{r,g,b}, midtones:{r,g,b}, highlights:{r,g,b}}; a bare number is coerced to {r:n,g:n,b:n}.
• A masked op with no active selection degrades to a whole-layer edit (does not throw).

---

2. NL planner (window.geEditorAI)

const res = await window.geEditorAI.execute('blur the background behind the person', { onStep });
// exec: window.geEditorAI.execute('blur the background behind the person', { onStep });
// res = { ok, prompt, reply, planner, basicMode, steps, trace, failed }

Flow (static/js/editor/nl-edit.js):

1. Read document state + a before screenshot.
2. Ask the server planner (POST /api/editor/plan) with the prompt, state and screenshot. The client passes its strong guidance via state.planner_guidance. If the route is unavailable it falls back to localPlan (regex — labelled "basic mode" in the UI).
3. sanitizeSteps(): drop no-op/export steps and, if a masked op has no preceding selection.*, insert the region implied by the prompt (face/sky/background/subject).
4. history.mark, then run each step, screenshotting after each one.
5. Self-check (server plans only, max 3 rounds): send the after screenshot back in mode:"check" with the original intent. If not satisfied, revert the previous attempt (undo its history entries) and apply the model's full corrected plan.
6. history.collapse — the whole request becomes a single undoable step.

UI: the Quick Edit box routes multi-clause / "and ..." prompts to this executor and streams Step i/n: <op> into its status line; a (basic mode) suffix marks the offline planner.

Server route

POST /api/editor/plan (HARNESS, routes/editor_agent_routes.py)

{ prompt, state, mode?: "plan"|"check", screenshot?: "data:image/..." }
-> { reply, planner, steps: [{op, params}], done? }

Owner-scoped, authenticated. Uses the configured default/utility model with vision. If the model call is unavailable it returns a deterministic planner:"fallback" plan.

---

3. Bridge (server agent drives an open editor)

static/js/editor/editor-bridge.js subscribes the open editor to the server queue. Enabled automatically when the editor opens (window.__geEditorBridge).

• Subscribe: GET /api/editor/bridge/events?session=<id> — SSE, data: {id,op,params} (keepalive every 15s).
• The page runs the op via the control API and replies: POST /api/editor/bridge/result {id,ok,result,error,screenshot,ms}.
• Enqueue a command (server side): POST /api/editor/bridge/command {session, op|steps, wait, timeout}. With wait:true the call blocks and returns the editor's result.

Round-trip example (authenticated; <session> = window.__geEditorBridge.session):

curl -s -X POST http://127.0.0.1:7011/api/editor/bridge/command \
  -H 'Content-Type: application/json' -b "odysseus_session=$TOKEN" \
  -d '{"session":"default","op":"layers.addRaster","wait":true,"timeout":8000}'
# -> {"id":"...","ok":true,"result":{...},"screenshot":"data:image/png;base64,..."}

Session scoping is owner:session, so an agent can only drive editors owned by the same user. The bridge is a browser-only flow: it requires an editor page to be open.

---

4. SAM dependency

selection.subject / selection.face / selection.sky / selection.samText use the server SAM mask tool (POST /api/image/mask). When the SAM tools are not installed the editor raises "Object mask tools are not installed" and the control op falls back to a coarse approximation (centred rect / upper-centre ellipse / top-band colour range). On a plain background the approximation is acceptable; on real photos it leaks, producing visible rectangular/elliptical artifacts. Install the Cookbook SAM mask dependencies for true subject/face/sky segmentation.

selection.colorRange and selection.background never need SAM (pure colour matching), but they are also coarse on busy scenes.

---

5. Two ways for an agent to use this — pick one

Option A — built-in tool (in-process)

Register an editor-control tool inside the app (same process as the model) that calls the bridge/control API directly. Uses the app's normal login; nothing new is exposed. This is the rejected stdio-MCP design's replacement: a server-side tool that talks to the bridge needs either an internal token (auth-bypass) or to run in-process. Decision pending (user).

Option B — browser-only flow (works today)

No new server tool. An external agent drives an open editor purely through the bridge endpoints using the logged-in owner's session. Restricted to the editor the user has open. This is the flow demonstrated above and requires no auth-bypass.

A stdio photo_editor MCP server was prototyped and rejected because the built-in MCP subprocess has no internal token; wiring it would require expanding an auth bypass. The two viable options above are left for the user to choose.

---

6. Known limitations / honest grading

Live run against real 7011 with the server LLM planner (7 prompts): all 7 executed (planner:"llm"). Graded by eye:

• PASS: black & white portrait; warm cinematic; sharper + more vivid; text title.
• FAIL (region-masked, SAM-bound): "dramatic sky" (global over-saturation instead of a dramatic, masked sky), "blur the background" (coarse mask leaks; visible rectangle), "brighten the face" (coarse ellipse/rect artifact).

Whole-image edits are reliable; subject/face/sky region edits need SAM. The self-check loop works (revert-and-replan) but the region cases spend all 3 iterations.

7. Theming (control kit)

The editor's interactive controls (ranges, number fields, selects, checkboxes) share a single themed kit: static/js/editor/control-kit.js adds .ge-ctl-* classes scoped to the editor roots (#gallery-editor-container, .ge-adj-popup, .ge-filter-overlay, .ge-canvas-size-overlay, .ge-float-window, .ge-transform-popup), and static/css/13-image-editor.css (section "Editor control kit") styles them from the app design tokens (static/css/00-tokens.css) to match the app sliders/selects. It keeps range fills in sync and attaches a scrub-on-label affordance to numeric fields (drag the field's label to change the value; arrow keys work, Shift = x10).

The PhotoCraft editor is dark-only. :root.light flips the global tokens, but 13-image-editor.css has no .light handling and the editor relies on dark surfaces, so light mode renders a garbled mix. Unlike the rest of Odysseus, the editor follows Photoshop and is not offered in a light theme.