mirror of
https://github.com/pewdiepie-archdaemon/odysseus.git
synced 2026-09-12 03:02:21 +02:00
RaresKeYandStressTestor
7026cf40b5
* docs(specs): restore bootstrap after dev rewrite
* docs(specs): remove runtime inventory snapshot
* docs(specs): reconcile current dev truth
* docs(specs): document scheduled task actions as an owner-attribution source
Owner Attribution covered cookie, bearer-token and internal-loopback
requests. Scheduled task actions are a fourth source and behave
differently: _execute_action passes owner=task.owner off the stored
ScheduledTask row, so no request and no resolved principal are in
flight, and route-level require_user() never runs.
Webhook triggers are the sharp case. They are unauthenticated by
design with the token as the only credential and execute under the
stored task.owner.
Paths cite routes/task/task_routes.py, the canonical location after
the task subpackage move (#6081); routes/task_routes.py on current dev
is the backward-compat shim.
* docs(specs): add chained tasks to the trigger list, refresh dev stamp
Review feedback from RaresKeY on the previous commit.
"Every trigger path" was too broad: success-chained tasks are another
path into _execute_action. Added them with their own citation, and
noted that chaining additionally requires the target task to share
task.owner and rejects cycles, which is stricter than the trigger-side
checks. Softened the lead-in to "these trigger paths".
Line 56 still pointed at routes/task_routes.py for webhook credential
validation. That path is the backward-compat shim on current dev after
the task subpackage move (#6081); repointed to the canonical
routes/task/task_routes.py.
Stamp moved to dev@2a6b09b. Inspection backing that bump was scoped:
every file path cited in this spec was mechanically checked to resolve
on 2a6b09b, and every file:line in the Owner Attribution additions was
read against it. Behavioral claims elsewhere in the file were not
re-audited.
* docs(specs): correct SECURE_COOKIES description to match current behavior
Third of the stale details RaresKeY enumerated. The cookie section
described SECURE_COOKIES as purely opt-in, which stopped being true.
_secure_cookie() (routes/auth_routes.py:89) treats an explicit true or
false as authoritative and derives the Secure attribute from the
request otherwise, including when the variable is unset and when
docker-compose injects it present-but-empty. Either the connection
scheme or the first X-Forwarded-Proto hop being https is enough.
* docs(specs): refresh current dev truth
---------
Co-authored-by: StressTestor <212606152+StressTestor@users.noreply.github.com>
175 lines
14 KiB
Markdown
175 lines
14 KiB
Markdown
# Shell And MCP
|
|
|
|
Last updated: dev@2e2bb52 | 2026-08-16
|
|
|
|
## Scope
|
|
|
|
This spec covers shell and MCP behavior in:
|
|
|
|
- shell routes in `routes/shell_routes.py`;
|
|
- the standalone shell helper in `services/shell/service.py`;
|
|
- agent shell/background execution in `src/tool_execution.py`, `src/agent_tools/subprocess_tools.py`, `src/bg_jobs.py`, and `src/bg_monitor.py`;
|
|
- app wiring and startup/shutdown in `app.py`;
|
|
- MCP configuration routes in canonical `routes/mcp/mcp_routes.py`, with `routes/mcp_routes.py` as a compatibility shim;
|
|
- MCP runtime state in `src/mcp_manager.py`;
|
|
- generic MCP OAuth helpers in `src/mcp_oauth.py`;
|
|
- built-in server registration in `src/builtin_mcp.py`;
|
|
- persisted `McpServer` config in `core/database.py`;
|
|
- MCP tool exposure in `src/agent_loop.py`, `src/tool_index.py`, `src/tool_schemas.py`, `src/tool_parsing.py`, `src/tool_implementations.py`, and `src/tool_security.py`;
|
|
- admin MCP/tool helpers in `src/agent_tools/admin_tools.py`;
|
|
- built-in servers in `mcp_servers/*.py`;
|
|
- Settings/Admin UI in `static/js/settings.js` and `static/js/admin.js`;
|
|
- CLI helper `scripts/odysseus-mcp`;
|
|
- Docker/native dependency context in `Dockerfile` and `docker-compose.yml`.
|
|
|
|
Cookbook model-serving shell flows are covered in `cookbook-hwfit.md`; this spec owns the shared shell and MCP surfaces they reuse.
|
|
|
|
## Shell Routes
|
|
|
|
`routes.shell_routes` owns `/api/shell/exec` and `/api/shell/stream`. These routes are powerful by design and are admin-only. They execute admin-provided command strings through the host shell.
|
|
|
|
Runtime behavior:
|
|
|
|
- `/api/shell/exec` runs a bounded command and returns stdout, stderr, and exit code;
|
|
- `/api/shell/stream` streams SSE output through plain pipes, POSIX PTY, POSIX tmux log tailing, or a Windows detached-log fallback depending on request flags and platform;
|
|
- empty commands return an error result without spawning a shell;
|
|
- timeouts kill the subprocess where possible;
|
|
- disconnects can stop streaming subprocesses;
|
|
- POSIX PTY support is optional and reports an unsupported event when unavailable.
|
|
|
|
`routes.shell_routes` also owns shell-adjacent Cookbook dependency endpoints:
|
|
|
|
- `/api/cookbook/packages`;
|
|
- `/api/cookbook/packages/install`;
|
|
- `/api/cookbook/rebuild-engine`.
|
|
|
|
Those endpoints probe local or SSH-remote packages, prepend user install bins for pip CLIs, validate SSH host/port through shared route validators, validate remote venv values, and restrict package installs to allowlisted dependencies.
|
|
|
|
`services.shell.service.ShellService` is a small standalone subprocess abstraction with output caps. It does not own live route behavior, PTY/tmux paths, Windows shell selection, admin checks, or Cookbook package probes.
|
|
|
|
## Agent Shell And Background Jobs
|
|
|
|
`src.tool_execution` owns agent-side `bash` execution and the `#!bg` marker. A `bash` block whose first line is `#!bg` starts a detached background job instead of holding the chat stream open. On Windows, request-scoped workspace shell execution prefers Git Bash when available so POSIX-style agent commands and path confinement use the intended shell instead of `cmd.exe` parsing.
|
|
|
|
`src.bg_jobs` owns disk-backed job state under `data/bg_jobs.json` and `data/bg_jobs/*`. It stores wrapper scripts, logs, exit-code files, timestamps, status, and capped result text.
|
|
|
|
`src.bg_monitor` owns polling and auto-continuation. When a job finishes, it injects the job result into the session, drains the agent stream, persists only the assistant continuation plus `bg_result` metadata, and marks the job followed up.
|
|
|
|
Runtime behavior:
|
|
|
|
- background jobs are restart-tolerant while their state files remain;
|
|
- jobs have a maximum runtime and stale cleanup window;
|
|
- output is capped with head/tail retention;
|
|
- active sessions can defer follow-up until the next monitor pass.
|
|
|
|
## Configured MCP Servers
|
|
|
|
`routes.mcp.mcp_routes` owns admin HTTP configuration for MCP servers:
|
|
|
|
- list/add/reconnect/enable/disable/delete servers;
|
|
- list tools and per-server tools;
|
|
- update per-server disabled tool lists;
|
|
- Google OAuth authorize/callback/manual exchange pages and generic Streamable HTTP OAuth redirect handling.
|
|
|
|
`core.database.McpServer` persists transport, command, args, env, URL, enabled state, OAuth config, disabled tool names, and encrypted generic OAuth token/client state. `McpServer.env` is plaintext JSON in the database.
|
|
|
|
`src.mcp_manager.McpManager` owns live connection state, stdio/SSE/Streamable HTTP transports, sessions, tool schemas, qualified names, and tool calls. HTTP route operations update both database state and live manager state where applicable. Streamable HTTP connects in a background task, can report `connecting` or `needs_auth`, and surfaces an authorization URL when the OAuth client flow redirects. Enabled configured servers connect concurrently at startup; each server has its own 20-second connection timeout and records `timeout` state without delaying siblings. The startup task has no second outer timeout.
|
|
|
|
Stdio and SSE connection setup registers the session, exit stack, tool list,
|
|
and status as one completed unit. If initialization or tool discovery fails
|
|
before registration, the partial `AsyncExitStack` is closed so transports do
|
|
not leak into later reconnect attempts.
|
|
|
|
`src.agent_tools.admin_tools.do_manage_mcp()` is the agent/admin tool path for MCP config and is re-exported lazily through `src.tool_implementations` for compatibility. It is narrower than the HTTP routes: add is stdio-only, command values are checked against an allowlist/denylist before persistence, and enable/disable primarily flips DB config. `scripts/odysseus-mcp` is config-only; it reads and mutates database rows, redacts env values by default, and does not report live manager connection state.
|
|
|
|
## Built-In MCP Servers
|
|
|
|
`src.builtin_mcp` owns startup registration of built-in MCP servers unless `ODYSSEUS_DISABLE_MCP` is enabled.
|
|
|
|
Python stdio built-ins:
|
|
|
|
- image generation;
|
|
- memory;
|
|
- RAG;
|
|
- email.
|
|
|
|
The optional browser built-in uses `npx -y @playwright/mcp@latest --headless --caps vision`. It is cache-gated by checking npm's `_npx` cache for the requested package and falling back to `npx --no-install`; uncached/missing browser MCP is logged with install guidance and skipped rather than blocking startup or downloading packages at boot. Python built-ins are omitted from OpenAI function schemas because native/code-block paths already describe those capabilities; the browser built-in is exposed through MCP function schemas when connected.
|
|
|
|
Built-in Python servers prepend the app root to inherited `PYTHONPATH` rather
|
|
than replacing the environment, so container/dev site-packages remain visible
|
|
on initial connect and automatic reconnect. They can be reconnected once on
|
|
tool-call failure. User-configured MCP servers return the call failure instead
|
|
of automatic reconnect.
|
|
|
|
The built-in email MCP server is owner-aware when an owner is supplied by the
|
|
caller or configured through `ODYSSEUS_MCP_EMAIL_OWNER` /
|
|
`ODYSSEUS_EMAIL_OWNER`; if owner-scoped email accounts exist and no owner is
|
|
available, email MCP fails closed instead of exposing global accounts. Other
|
|
built-in servers remain process-global/admin trust-boundary tools unless their
|
|
own subsystem spec says otherwise.
|
|
|
|
## Agent MCP Exposure
|
|
|
|
`McpManager` owns raw qualified tool calls named `mcp__{server_id}__{tool_name}`. It does not own admin, owner, public-user, or disabled-tool policy; callers must enforce policy before dispatch.
|
|
|
|
Current exposure path:
|
|
|
|
- `routes.mcp.mcp_routes` stores disabled tool names;
|
|
- `src.agent_loop` loads disabled maps for prompts/schemas;
|
|
- `McpManager.get_all_openai_schemas()` and prompt descriptions filter disabled tools;
|
|
- `src.tool_index` indexes MCP prompt descriptions by manager generation;
|
|
- `src.tool_security` blocks all `mcp__*` tools for non-admin/public users;
|
|
- `src.tool_execution` dispatches received `mcp__*` calls to `McpManager.call_tool()`.
|
|
|
|
Per-server disabled MCP tools currently hide tools from prompts/schemas while listings still return tools with disabled metadata. They are not a complete execution-time gate if a disabled qualified name reaches tool execution. Plan mode additionally asks `McpManager.plan_mode_blocked_mcp()` to hide write/unknown MCP tools and add qualified names to the runtime disabled set for that turn.
|
|
|
|
After model-visible external/workspace context, arbitrary MCP actions classify fail-high and require an exact one-use approval unless a specific low-impact capability classification says otherwise. MCP results are marked external-untrusted for continuation security even when a call returns a failed status with remote payload.
|
|
|
|
## Degraded And Platform Behavior
|
|
|
|
- `app.py` starts the background monitor and MCP startup tasks asynchronously; MCP startup is non-critical to app readiness.
|
|
- Configured MCP servers start concurrently with a per-server 20-second bound;
|
|
timeout state is stored per server and partial connection resources are
|
|
closed before returning.
|
|
- Missing Python `mcp` dependency degrades attempted MCP connections to error status.
|
|
- Missing or uncached browser NPX package is optional and log-only during built-in startup; startup should not perform an implicit package download.
|
|
- Windows does not support POSIX PTY/tmux paths; streaming falls back to pipes or detached logfile behavior.
|
|
- Docker images include selected shell dependencies and the Docker CLI, but host Docker socket access from inside the app container remains unavailable unless the operator explicitly enables `docker/host-docker.yml`/`ODYSSEUS_ENABLE_HOST_DOCKER=true` and mounts a real socket.
|
|
- OAuth supports Google `installed` or `web` key shapes, a remote paste-back exchange page, and generic Streamable HTTP OAuth token storage through encrypted `McpServer.oauth_tokens`. Valid JSON values that are not objects are treated as empty token state and replaced by an object on the next write. Google and generic MCP OAuth share `src.mcp_oauth.REDIRECT_URI`, built from `OAUTH_REDIRECT_BASE_URL`, then `APP_PUBLIC_URL`, then `http://localhost:${APP_PORT:-7000}`, plus `/api/mcp/oauth/callback`. Reverse proxies, public domains, and Docker host-port mappings should set an explicit public base because container bind state cannot infer the browser origin.
|
|
- `services.shell.service` remains a transitional/simple facade separate from route-level compatibility behavior.
|
|
|
|
## Security And Provenance
|
|
|
|
- Admin shell is intentional host command execution; do not expose shell routes or shell tools to regular users.
|
|
- `_require_admin()` gates shell routes and MCP config routes. The internal-tool loopback can be admin-equivalent only after auth middleware validates the internal token and loopback client.
|
|
- `_reject_cross_site()` currently applies to `/api/cookbook/packages`; `/api/shell/exec`, `/api/shell/stream`, package install, rebuild, and MCP write/OAuth routes do not call it directly.
|
|
- Shell helper paths use argv-based SSH, reject option-like hosts, validate SSH ports through shared helpers, restrict remote venv characters, and allowlist package installs.
|
|
- Non-admin/public tool policy blocks `bash`, `python`, file tools, `manage_mcp`, and all `mcp__*` tools.
|
|
- MCP stdio server registration is arbitrary host process execution and is admin-only.
|
|
- MCP OAuth key/token file paths supplied through routes are confined under `data/mcp_oauth`; generic Streamable HTTP OAuth token state is encrypted in the database.
|
|
- Built-in MCP servers are local/admin trust-boundary tools and are not
|
|
automatically equivalent to owner-scoped HTTP route behavior. Email MCP is
|
|
the current exception with explicit owner filtering; other built-ins need
|
|
their own owner policy before being treated as scoped surfaces.
|
|
- MCP output is external-untrusted tool output and arms the high-impact continuation gate when model-visible. Current MCP text output is still not centrally capped before model re-entry.
|
|
|
|
## Testing Notes
|
|
|
|
Current targeted coverage includes Windows PTY import degradation, PTY unsupported stream events, the cross-site helper, `ShellService` stream deadline behavior, background store/monitor basics, concurrent MCP startup, per-server timeout isolation and cleanup, MCP manager cache/reconnect args, built-in `PYTHONPATH` preservation, non-object generic OAuth-token storage recovery, MCP CLI JSON/env serialization, MCP common truncation helper, action intent shell verbs, and public blocked-tool fail-closed behavior.
|
|
|
|
The shell/MCP audit ran the targeted venv subset with 78 passing tests and one warning.
|
|
|
|
## Current Gaps
|
|
|
|
- Decide whether `/api/shell/exec`, `/api/shell/stream`, package install, rebuild, and MCP config/OAuth writes should call `_reject_cross_site()` directly.
|
|
- Add route-level shell exec/stream tests for admin gate, cross-site behavior, empty command, plain exec, timeout, PTY, tmux, and Windows detached fallback.
|
|
- Add background job tests for launch isolation, output truncation, done/failed/timeout/died states, pending follow-ups, and result text.
|
|
- Add route-level MCP CRUD/OAuth/disabled-tool tests with a fake manager and temp database.
|
|
- Add hard per-server disabled MCP execution checks or document disabled tools as prompt/schema filtering only.
|
|
- Make MCP tool indexing sensitive to disabled-map changes, not only manager generation.
|
|
- Fix stale outer prompt/cache behavior when MCP disabled tools change.
|
|
- Add one central truncation layer for MCP result text and images before model re-entry; untrusted-result marking and exact-action continuation approval are now implemented.
|
|
- Decide whether `McpServer.env` and OAuth key files need masking, encryption, and chmod beyond admin-only access.
|
|
- Decide whether built-in MCP servers should become owner-aware or remain documented as admin/global compatibility surfaces.
|
|
- Decide whether optional browser MCP cache misses should surface in `/api/mcp` status instead of startup logs only.
|