odysseus/docs/skills-audit-architecture.md

Skills evaluation and audit architecture

The Skills backend separates HTTP transport, model evaluation, agent execution, audit orchestration and publication policy. services/skills/ coordinates those responsibilities; services/memory/skills.py remains the canonical persistence and lifecycle owner. The extraction preserves the existing HTTP and Python contracts. PR #6711 tracks this boundary change under issue #6710.

Responsibilities and dependency boundaries

Module	Responsibility
routes/skills_routes.py	FastAPI request models, endpoint declarations, authentication, ownership, HTTP errors, exact approval validation and Python compatibility bindings
services/skills/job_coordinator.py	Transient job creation, continuation after approval validation, candidate selection, public polling projections and cancellation requests
services/skills/agent_runner.py	Manual execution, audit arms, agent loop integration, transcripts, authority propagation, approval pauses and same-task stream closure
services/skills/audit_workflow.py	Per-skill audit sequencing, self-edit retries, teacher escalation, sequential batches and scheduled execution
services/skills/evaluation.py	Model grading, verdict parsing, necessity and retrieval checks, efficiency fields and markdown improvement generation
services/skills/publication.py	Confidence and publishing policy, duplicate rules, batch finalization and markdown updates through SkillsManager
services/skills/model_selection.py	Owner-scoped worker/teacher resolution, utility fallback, endpoint selection and credential lookup
services/skills/prompts.py	Shared task construction and explicitly untrusted skill context
services/skills/dependencies.py	Frozen AuditDependencies containing callable bindings and transient registry references
services/memory/skills.py	Canonical skill identity, owner isolation, durable storage, indexing, retrieval and lifecycle state

The package initializer identifies the domain without adding a service facade. Evaluation does not schedule jobs or persist skills. Publication computes policy and applies updates through the supplied SkillsManager. Services do not import routes.skills_routes. The existing preferences adapter in publication.py still imports routes.prefs_routes; moving that adapter is a separate change.

flowchart TD
    HTTP[Authenticated HTTP requests] --> Routes[skills_routes]
    Builtins[builtin_actions callers] --> Routes
    Routes --> Jobs[job_coordinator]
    Routes --> Models[model_selection]
    Routes --> Evaluation[evaluation]
    Routes --> Runner[agent_runner]
    Routes --> Workflow[audit_workflow]
    Jobs -->|injected calls| Runner
    Jobs -->|injected calls| Workflow
    Workflow -->|injected calls| Runner
    Workflow -->|injected calls| Evaluation
    Workflow -->|injected calls| Publication[publication]
    Runner -->|injected calls| Evaluation
    Runner --> Runtime[Agent runtime and authority enforcement]
    Runner --> Prompts[prompts and untrusted context]
    Jobs --> Prompts
    Workflow --> Prompts
    Routes --> Manager[Canonical SkillsManager]
    Runner --> Manager
    Workflow --> Manager
    Publication --> Manager
    Publication --> Preferences[Existing preferences adapter]
    Models --> Resolver[Endpoint resolver and owner-scoped credentials]

Arrows include injected calls, rather than only Python imports. The workflow imports SkillAuditUnavailable from the runner. Route wrappers construct AuditDependencies on entry from the current route bindings, so existing callers and monkeypatches reach the same collaborators without rebinding shared service globals.

The manual test registry belongs to agent_runner.py; the audit registry belongs to audit_workflow.py. Routes expose aliases to those dictionaries. Registries hold process-local job state and are not durable skill stores. All durable metadata, confidence, markdown and publication changes use SkillsManager. Published-only retrieval and local exclusion of remote-only skills stay in the manager/index boundary. Service loggers use their module names; the route module retains its own logger. Root handlers supply the existing log output.

Execution and publication lifecycle

1. The HTTP handler authenticates the caller, verifies ownership and validates the request. Manual model choices retain registered-endpoint validation and owner-scoped credential lookup.
2. Job coordination creates the owner-keyed transient record and launches the execution coroutine. Replacing a manual test retires its pending approval. Polling exposes only the existing public fields, excluding credentials, private transcripts, continuation state and asyncio tasks.
3. The runner uses the originating request authority or the consumed sealed approval authority. It captures the transcript and closes the async stream with aclosing inside the task that opened it, including early approval pauses, cancellation and exceptions.
4. Manual approval pauses return to the HTTP validation boundary. Continuation reconstructs context from the original untrusted skill and accumulated untrusted transcript. Manual completion records audit metadata and adjusts confidence (pass .95, needs_work .6, fail .4) without publishing.
5. An automated audit checks deterministic duplicates, advisory necessity and, when indicated, retrieval precision. It runs skill and no-skill arms on the same task, grades the result, then may self-edit and retry. An optional distinct teacher rewrites the skill; the worker performs the retry.
6. Publication applies owner preferences, confidence thresholds, necessity, utility and duplicate rules through SkillsManager. A normal sequential batch finalizes its processed results. Cancellation or infrastructure unavailability skips batch finalization and clears task/current state.

A clean audit pass sets confidence to .95, a self-edit pass to .85 and a teacher rewrite pass to .8. Continued failure becomes draft at .35. Unknown and inconclusive verdicts preserve publication state. Automatic publishing requires an enabled owner policy, a passing verdict at or above the configured threshold, and no necessity, worse-baseline or duplicate blocker. The default threshold is .85; owner values are clamped to [0, 1]. Markdown edits pin the existing skill name so usage and audit sidecars remain attached to the same identity.

Per-skill duplicate blocking and batch duplicate finalization have distinct selection rules and must not be consolidated without a behavior proposal. Batch finalization excludes builtins and binned records, changes only processed skills, protects approval-required results, and bins duplicate losers. Duplicate similarity uses the existing .38 threshold and name-suffix grouping.

Batches run sequentially per owner, retain current/done/results and timestamps, and cap progress logs at 1,000 entries. Ordinary exceptions record inconclusive skill audits. Infrastructure failures stop the batch without inventing a skill verdict. Scheduled execution skips an active owner job, selects existing lifecycle candidates by oldest audit, caps the batch (default eight), and marks model phases as background work. These services retain the existing scheduling and cancellation semantics.

Security and approval invariants

HTTP authentication and owner verification remain at the transport boundary. Manager reads and writes receive the owner. Ownerless builtins retain their existing visibility; an authenticated user cannot access an ownerless ordinary skill. Existing ownerless deployment behavior is preserved.

The /test-approval handler validates the pending job, approval ID, decision, owner and empty session binding, then destructively consumes the sealed approval with allow_continuation=False before calling the coordinator. Stale, mismatched or reused approvals fail with 409. The consumed grant authorizes only its exact action. Services do not manufacture approval authority or relax process identity, workspace confinement or request-authority enforcement in the agent runtime.

Skill markdown and resumed transcripts remain explicitly untrusted context, never trusted system instructions. Audit arms suppress saved skill injection. Unattended runs deny encountered approval records instead of leaving reusable grants. An initial approval-required audit result preserves the skill's publication and confidence and directs the user to manual approval. Retry handling has an inherited gap: an approval pause after a rewrite can still lead to demotion. Same-task stream closure preserves the originating ContextVar cleanup, but manual cancellation can retain private continuation state. These are separate hardening items, not guarantees supplied by the extraction.

Model selection and evaluation

Audit resolution uses an explicit endpoint/model pair when supplied, looking up credentials among enabled owner-visible endpoints. A model-only choice uses the existing owner-scoped model resolver. Otherwise it resolves the utility model, including the resolver's default behavior, then takes the first configured utility fallback if the primary result is incomplete. A missing worker raises ValueError. Served-model normalization keeps an exact match, then a basename match, then the first served ID; discovery exceptions keep the resolved choice.

An enabled teacher resolves separately with the same owner. Its credentials are not borrowed from the worker. Teacher resolution failure logs the existing warning and leaves the teacher absent. Manual tests retain their HTTP endpoint validation and registered fallback credentials.

Owner reasoning effort is resolved for each evaluation or rewrite call. Workload flags, credentials, retries, prompts and permissive parsing remain unchanged:

Phase	Token budget	Timeout in seconds
Run review, up to two attempts	32768	180
Necessity	8192	120
Retrieval precision	4096	90
Markdown improvement	16384	180

Review parsing retains think-block stripping, trailing-comma recovery, last usable verdict selection, prose fallback, bounded fields and unknown outcomes. These model outputs are advisory and do not grant execution authority.

Public HTTP contract

All paths below have prefix /api/skills. Successful status defaults to 200. Request models and endpoint declaration order remain fixed. A frozen OpenAPI and declaration-order fixture records the baseline contract independently of the extracted implementation.

Method and suffix	Successful response	Authorization and errors
GET empty	skills,count	Owner-filtered load
GET /index	index,count	Owner-scoped eligible index
GET /slash-catalog	skills,count	Owner-scoped index; sorted slash entries
GET /builtin	builtin,count and optional error	Existing shared capability text
GET /builtin/{name}	name,text,default,is_overridden	404 missing tool; 500 import failure
PUT /builtin/{name}	ok,name,is_overridden	Admin; 404 unknown, 400 missing text
DELETE /builtin/{name}	ok,name,is_overridden	Admin
POST /import-from-url	ok,skill,files	Admin and current owner; 400 import, 502 HTTP, 500 other failure
POST /add	ok,deduped,skill	Current owner passed to manager; unchanged old/new request fields
POST /{skill_id}/invoke	ok,type,name,command,message	Eligible owner index only; 404 unavailable/source missing
GET /{skill_id}	Full skill dictionary	Owner load; 404 missing
GET /{skill_id}/markdown	name,markdown	Owner verification; 404 missing/source missing
POST /{skill_id}/test	ok,status,skill,model	Owner verification, HTTP request authority, registered fallback credentials; 400 missing/invalid model
POST /{skill_id}/test-approval	ok,status,decision	User approval request, owner verification, pending ID/owner/empty session match; 400 invalid body/decision, 409 stale/mismatch/consumed
GET /{skill_id}/test-status	status or status,task,model,log,verdict,approval	Owner-keyed lookup; server continuation/credentials omitted
POST /audit-all	ok,status,total plus running done,model or empty results,log	Owner-keyed singleton; 400 model resolution
GET /audit-all/status	status or job progress/result/log/timestamps	Owner-keyed lookup; task and credentials omitted
POST /audit-all/cancel	ok,status	Owner-keyed lookup and cancellation
POST /{skill_id}/markdown	ok,name	Owner verification; 400 missing/parse, 404 missing, 500 update
PUT /{skill_id}	ok	Owner verification and scoped manager update; 404 missing
DELETE /{skill_id}	ok	Owner verification and scoped manager deletion; 404 missing
POST /search	skills,query,count	Owner-filtered input; 400 empty query

Strict owner verification allows an ownerless builtin but denies an ownerless ordinary skill for an authenticated user. The existing ownerless deployment behavior remains. Global auth middleware is unchanged.

Python compatibility and frozen contracts

app.py imports setup_skills_routes. src/builtin_actions.py uses the existing route helpers and audit registry. Route aliases and wrappers preserve the original 28 inspectable interfaces, SkillAuditUnavailable, registry identity and patchable collaborators. Lazy model/agent imports and route-level effort injection keep existing offline test seams effective.

The task and message callbacks are bound through AuditDependencies at service entry. Patches to route-level test tasks and test/baseline messages reach manual execution, approval continuation, both audit arms, worker retries and scheduled audits. The _skill_baseline_task compatibility alias delegates to the current route test-task builder. Execution continues to reuse one task for both arms, as in the original implementation; it does not independently derive a baseline task through that alias.

tests/fixtures/skills_api_baseline.json records declaration order and structural signatures captured from pre-extraction revision 86b886c611338282af6d4a717246ea4e256f58df. Signature records retain parameter names, order, kinds, defaults, default factories and annotations, including Pydantic constraint metadata. They compare typing origins and arguments instead of Python-version-dependent annotation representations. The expanded records also pin field validation aliases, alias priority, discriminators, validate_default, rebuilt field annotations and model configuration. Field order remains visible through the constructor parameter sequence. Only the known builtin factories list, tuple, dict, set and frozenset are supported. Custom factories, including closures and factories taking validated data, fail closed without being executed. Unsupported metadata/configuration values also raise rather than using unstable representations. This is a structural contract for supported interfaces, not proof of behavioral equivalence for arbitrary validators or Python code.

The expanded fixture was derived again from the original revision above in a detached scratch worktree, with the updated comparator copied in. Python 3.11 and 3.12 produced byte-identical snapshots. The previous parameter/return records, all 28 interface names, all 22 endpoint declarations and the original OpenAPI digest were checked unchanged before accepting the new field records.

tests/fixtures/skills_openapi_baseline.json is the same baseline's complete Skills-router OpenAPI document, generated with a standalone FastAPI application. Sorted keys, ASCII escaping, two-space indentation and a final newline make the fixture deterministic. Tests require exact canonical equality and print a unified diff on mismatch. The redundant digest field and assertions have been removed; they provided no independent compatibility guarantee. The fixture pins model schemas and constraints. CI reads committed fixtures and needs no Git history for these tests. Baseline updates require an intentional compatibility decision, rather than regenerating fixtures from a failing implementation.

The narrow .gitleaks.toml exception remains necessary for the historical public digest in earlier fixture and documentation revisions. It applies to the exact digest, the generic-api-key rule and the two named paths. Removing a current copy cannot remove findings from published history. Keep the exception unless the pinned full-history scan proves it unnecessary; do not rewrite history to remove it. The checksum-verified 8.30.1 scan without this exception still reports both historical copies in commit a547c4d8; retaining the exception passes the full PR/dev history scan. Reconsider it after squash merge only after verifying the resulting reachable history with the pinned full-history scan.

The ReDoS source guard scans services/skills/evaluation.py, the canonical evaluation parser module, as well as the retained route and memory-extractor locations. Verdict regex runtime tests import the canonical module directly. The source guard must reject reintroducing the lazy think-block expression; the ordinary-input and adversarial-flood runtime tests remain in place. An isolated source mutation reproduced the old guard's false green for the evaluation module. The corrected guard rejected the vulnerable expression in each of the three locations and passed again after restoring the temporary copies. The mutated regex was never executed, and production parser behavior was not changed.

Reproducible validation

From the repository root, use a virtual environment with requirements-dev.txt installed and the repository's frontend dependencies available. The examples use .venv/bin/python; repeat the contract tests with a Python 3.12 environment as well as Python 3.11. Record interpreter/dependency versions with results.

.venv/bin/python -m pytest -q \
  tests/test_skills_audit_characterization.py tests/test_skills_audit_services.py
.venv/bin/python -m compileall -q app.py core routes src services scripts tests
.venv/bin/python -c 'import routes.skills_routes; import services.skills.agent_runner; import services.skills.audit_workflow'
.venv/bin/python scripts/build_frontend.py --check
node scripts/check_frontend.mjs
node --test tests/skillsApproval.test.mjs tests/frontendArchitecture.test.mjs
git diff --check

The focused Skills, authority and approval selection is reproducible with:

.venv/bin/python - <<'PYTHON'
from pathlib import Path
import subprocess
import sys

files = set()
for path in Path("tests").rglob("test_*.py"):
    text = path.read_text()
    if ("skill" in path.name or "approval" in path.name
        or "utility_reasoning" in path.name or "skills_routes" in text
        or "review_endpoint_credentials" in path.name):
        files.add(str(path))
files.update([
    "tests/test_request_authority.py",
    "tests/test_endpoint_registered_authority.py",
    "tests/test_redos_verdict_continuation.py",
    "tests/test_redos_residual_python.py",
])
subprocess.run([sys.executable, "-m", "pytest", "-q", "-rs",
                "--durations=15", *sorted(files)], check=True)
PYTHON

The original selection contained 729 cases. Additional characterization cases increase that count without removing the original coverage. For the full suite, remove a development APP_PORT and put the environment's tools on PATH:

env -u APP_PORT PATH="$PWD/.venv/bin:$PATH" \
  PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/ms-playwright" \
  .venv/bin/python -m pytest -q -rs --durations=25

Run the checksum-verified Gitleaks binary pinned in .github/workflows/secret-scan.yml on full reachable history, including the PR head and fetched official dev. The current workflow pins 8.30.1. Its CI command is gitleaks git --no-banner --redact --verbose . with a full-history checkout. A local checkout can restrict refs to the relevant histories with --log-opts='HEAD upstream/dev'; do not restrict the commit range to just the latest patch. Scan uncommitted fixtures too before committing.

Historical evidence and validation limits

The initial extraction validation at a547c4d8 reported 42 characterization and service cases, 729 focused cases, 13 relevant Node cases and 17 isolated uvicorn HTTP checks passing. The historical repository-wide run reported 13,959 passed, 60 skipped and three failures. These counts describe the extraction validation, not the current revision. The HTTP checks used isolated state and authentication and included second-owner denial, but exercised neither a configured live model nor an external tool action.

The two containment failures reproduced on unchanged baseline 86b886c6 with extra Failed to create stream fd: No such file or directory stderr output:

• test_namespace_handshake_closes_model_stdin
• test_execution_environment_cannot_replace_probed_bwrap[False]

test_legacy_cleanup_against_a_private_real_tmux_server failed under the original full-suite load and passed isolated reruns. Its cause was not fully diagnosed. These historical failures must be distinguished from new regressions; successful CI and maintainer review remain integration gates. Earlier AST comparisons and the maintainer review provide historical extraction evidence, not an independent review of later fixes.

Known limitations and follow-up boundaries

Issue #6635 remains open: manual tests without an appropriate workspace can still fail to execute their proposed action. Stream cleanup and exact approvals are preserved, but do not supply a missing workspace.

Jobs remain process-local and do not survive restarts. Durable jobs, moving the preferences adapter below transport, consolidating storage under the domain package, or changing duplicate/publication policies need separate designs. The canonical manager, skill format/lifecycle modules, provider implementations, agent authority enforcement and frontend contracts remain their existing owners.

The second-review correction report records the restored prompt seams, independently expanded constructor baseline, actual validation results and a separate prioritized plan for the six inherited P2 failure paths. Model-controlled lifecycle fields and preference-load failures require maintainer action before automatic rewrite/publication deployment.