odysseus/specs/testing-devops.md

# Testing And Devops

Last updated: dev@c74f475 | 2026-10-15

## Scope

This spec covers development and validation surfaces in:

- `tests/`, `tests/conftest.py`, `tests/*.mjs`, and `tests/bombadil-spec.ts`;
- `tests/run_focus.py`, `tests/run_order_report.py`, `tests/_taxonomy.py`, `tests/TESTING_STANDARD.md`, and `tests/LAYOUT_INVENTORY.md`;
- `pyproject.toml`;
- `requirements.txt` and `requirements-optional.txt`;
- `package.json` and `package-lock.json`;
- `Dockerfile`, `docker-compose.yml`, `docker/gpu.nvidia.yml`, `docker/gpu.amd.yml`, `docker/host-docker.yml`, top-level standalone GPU compose files, and `docker/entrypoint.sh`;
- `scripts/`, `scripts/odysseus`, `scripts/_lib/cli.py`, `scripts/_completion/*`, `scripts/pr_blocker_audit.py`, and `scripts/odysseus-*`;
- GPU helper scripts `scripts/check-docker-gpu.sh` and `scripts/check-docker-amd-gpu.sh`;
- `.github/` templates, workflows, and description-check scripts;
- contributor workflow docs in `CONTRIBUTING.md` and `website/pr-blocker-audit.md`;
- platform launchers `launch-windows.ps1`, `launcher.py`, `Odysseus.spec`, `build-windows-portable.ps1`, `start-macos.sh`, `build-macos-app.sh`, and `update_windows.bat`;
- setup/service files such as `setup.py`, `install-service.sh`, and `odysseus-ui.service`.

## Test Runtime

Pytest is configured in `pyproject.toml` with:

- `testpaths = ["tests"]`;
- `asyncio_mode = "auto"`;
- marker and fast-lane/duration-reporting settings used by focused test runs.

The expected local command uses the project venv:

```bash
./venv/bin/pytest <test path>
```

Activated-venv `python -m pytest <test path>` is equivalent. System/global `pytest` is not authoritative for this repo because installed versus stubbed dependencies can change collection behavior.

`tests/conftest.py` inserts the repo root on `sys.path` and conditionally stubs missing heavy/runtime dependencies such as SQLAlchemy, FastAPI, Starlette, Pydantic, httpx, bcrypt, and pyotp. Tests that need real dependencies use explicit imports/skips. Tests that stub `sys.modules`, environment variables, globals, or parent packages must restore them with `monkeypatch` or an equivalent cleanup pattern.

The suite currently contains a growing set of `test_*.py` files. Treat that count as a moving source metric, not a target; focused regression tests are still preferred for narrow changes.

Focused regression tests are preferred for narrow behavior changes. Broaden tests when touching shared contracts such as auth, owner filtering, OAuth/token custody, tool output, context building, provider calls, persistence, frontend rendering, or route/API shapes.

`tests/run_focus.py` and `tests/_taxonomy.py` provide a local focused-run helper and category map. `.github/scripts/focused_test_guidance.py` maps changed files to suggested focused tests for PR review, while the configured full pytest CI job is authoritative. `tests/TESTING_STANDARD.md` documents expectations for targeted validation, and `tests/LAYOUT_INVENTORY.md` records the test-suite layout. CLI tests live under `tests/cli/`.

## JS And UI Tests

`frontend/modules.json` maps canonical `frontend/src/` modules to committed outputs under `static/`. `scripts/build_frontend.py --check` verifies output freshness/publication attributes; `scripts/check_frontend.mjs` checks architecture/import boundaries. Root `package.json` owns Bombadil, Playwright and Acorn plus photo-editor browser-test scripts; the lockfile owns npm integrity/version state. This is an incremental source migration rather than a bundler-wide conversion.

Current frontend/JS validation includes:

- pytest wrappers that run Node snippets and usually skip when `node` is missing;
- direct `.mjs` regressions under `tests/`;
- `tests/bombadil-spec.ts`, which requires npm-installed Bombadil dev dependencies and a running/browser-capable UI workflow when used.

Use `node --check static/js/<changed-file>.js` for syntax checks on changed JS files when applicable. This is not a full module-graph, browser-global, or DOM integration check.

## Dependencies

`requirements.txt` owns core runtime and test dependencies, including `psycopg2-binary`, pytest, pytest-asyncio, MCP, Chroma HTTP client, fastembed, qrcode, and core parsing/search/calendar dependencies.

`requirements-optional.txt` owns optional feature dependencies:

- `faster-whisper` for local STT;
- `kokoro==0.9.4` and `soundfile` for local TTS on Python 3.11-3.12 only; Kokoro is deliberately skipped on Python 3.13+ because its package metadata excludes those runtimes, and a CUDA-capable torch/GPU is still required at runtime;
- `ddgs` for DDG library support, while provider code can fall back to HTML scraping;
- `rapidocr==3.9.2` and `onnxruntime>=1.20,<2` for local OCR, and `psd-tools` for PSD previews/layer inspection;
- `PyMuPDF` for PDF forms/rendering with AGPL implications for a network-served app;
- `markitdown[docx,pptx,xlsx,xls]` for Office/EPUB extraction, pinned to a release older than 30 days.

Optional dependencies should produce clear degraded behavior when absent unless intentionally promoted to core. MarkItDown and PyMuPDF already have focused degraded-path coverage; local STT missing-`faster-whisper` behavior is a remaining coverage gap. Core runtime requirements include `httpx2` where compatibility tests depend on it. The official Docker image additionally installs `libmagic1` plus `python-magic==0.4.27` for content-based upload MIME sniffing; that pairing is image-owned because `python-magic` needs the system shared library at import time.

Chroma has two compatibility modes:

- Docker uses a separate `chromadb` service and core `chromadb-client`/`fastembed`;
- native macOS setup removes conflicting `chromadb-client` and installs full `chromadb`.

Vector features should fail fast or degrade to unhealthy/keyword fallback when the service is unavailable.

## Docker Runtime

Docker Compose is the primary deployment path:

```bash
docker compose up -d --build
docker compose ps
docker compose logs --tail=120 odysseus
```

`docker-compose.yml` starts Odysseus, ChromaDB, SearXNG, and ntfy. It binds services to loopback by default through `APP_BIND`, `CHROMADB_BIND`, and `NTFY_BIND`, persists configurable `APP_DATA_DIR`/`APP_LOGS_DIR`, SSH identity, HuggingFace cache, and user-local Python installs, and gives the Odysseus container host-loopback reachability through `host.docker.internal`.

Compose variants forward `ODYSSEUS_TTS_CACHE_MAX_BYTES`, defaulting in the service to 500 MiB, and run the mounted `scripts/migrate_searxng_settings.py` helper so retained SearXNG YAML gains default inheritance without replacement. The helper preserves file metadata and formatting where possible and writes atomically; migration failure is non-fatal to the wrapper command. MCP OAuth callback setup follows `OAUTH_REDIRECT_BASE_URL`, `APP_PUBLIC_URL`, or the launcher/bind `APP_PORT`, so externally remapped deployments should set a public base explicitly.

`Dockerfile` builds a Python 3.14 slim image with Node/npm, tmux, OpenSSH client, git/cmake, the pinned Docker CLI `29.6.2`, `gosu`, `libmagic1`, and the image-only `python-magic` wrapper.

`docker/entrypoint.sh` owns writable path ownership repair, PUID/PGID user/group creation and privilege drop, optional host-Docker socket group handling, vLLM/CUDA environment defaults, idempotent `setup.py`, and final uvicorn execution.

Docker does not mount the host Docker socket by default. Mounting it would grant powerful host access and is outside the default trust boundary. `docker/host-docker.yml` is the explicit opt-in overlay and sets `ODYSSEUS_ENABLE_HOST_DOCKER=true`; tests guard that the default and GPU compose files do not enable host Docker accidentally.

Compose selects `ODYSSEUS_IMAGE` (default GHCR `latest`) alongside a local build and publishes container port 7000 on default host port 7011. SearXNG is tag-and-digest pinned. `docker/host-workspace.yml` explicitly mounts a selected host workspace; `docker/host-network.yml` opts into Linux host networking, resets published ports, and starts uvicorn on the configured host bind/port. These overlays expand access beyond default deployment. The image includes Chromium, Noto CJK fonts, network inspection utilities, pinned `agent-browser@0.35.0`, and redistribution notices. Entrypoint ownership repair touches HuggingFace cache roots without traversing existing model files.

## GPU And Platform

Base `docker-compose.yml` plus `docker/gpu.nvidia.yml` or `docker/gpu.amd.yml` are the GPU source of truth. Top-level `docker-compose.gpu-nvidia.yml` and `docker-compose.gpu-amd.yml` are standalone mirrors for stack-management UIs that accept one compose file. `tests/test_gpu_compose_standalone.py` guards drift between those forms.

GPU overlays pass host devices/runtime flags only. They do not install CUDA/ROCm userspace or serving engines; those are installed later through Cookbook/dependency flows.

NVIDIA helper behavior:

- `scripts/check-docker-gpu.sh` diagnoses passthrough;
- it is read-only by default;
- toolkit install and `.env` edits require explicit user flags and successful passthrough checks.

AMD helper behavior:

- `scripts/check-docker-amd-gpu.sh` is read-only;
- it prints expected `COMPOSE_FILE`/`RENDER_GID` values and verifies `/dev/kfd`/`/dev/dri` visibility.

Native platform launchers:

- `launch-windows.ps1` requires Python 3.11+, creates `venv`, installs `requirements.txt`, runs `setup.py`, discovers per-user Git Bash installs where possible, warns when Git Bash is missing, and starts uvicorn on port 7000 by default.
- `launcher.py`, `Odysseus.spec`, and `build-windows-portable.ps1` own the PyInstaller-style portable Windows launcher path, including app-root/data-dir differences covered by `src.runtime_paths`.
- `start-macos.sh` reads `.env`, defaults to port 7860 to avoid AirPlay conflicts, prefers Homebrew arm64 Python, installs/tolerates Homebrew Cookbook deps, handles Chroma package conflicts, starts ChromaDB for native runs, runs `setup.py`, and starts uvicorn.
- `build-macos-app.sh` builds a launcher app around the existing repo venv and logs to `logs/odysseus-app.log`.
- `update_windows.bat` owns the tested Windows Docker update flow.

## Scripts And CLI

`scripts/odysseus` is the umbrella dispatcher for executable `scripts/odysseus-*` commands. It discovers subcommands and executes them through the project venv Python when available.

`scripts/_lib/cli.py` owns shared CLI behavior:

- repo-root importability;
- quiet logging;
- JSON output and `--pretty`;
- `--version`;
- common parser scaffolding;
- exit handling.

`LOG_LEVEL` is the shared process logging toggle. CLI helpers default it to
`WARNING` to keep JSON command output clean; the web app defaults it to `INFO`
and applies it to root, console, rotating-file, and direct-uvicorn logging.
Shell completions in `scripts/_completion/` introspect CLI `--help` output through the venv and cache subcommands.

`scripts/odysseus-*` provide local CLI surfaces for backup, calendar, contacts, Cookbook, docs, gallery, logs, mail, MCP, memory, notes, personal docs, presets, research, sessions, signatures, skills, tasks, theme, and webhooks.

When route/API behavior changes, check whether a matching CLI script depends on the old shape. There is no central CLI scrubber: each credential/log/mail/task/backup/MCP/webhook script owns its own sensitive-output behavior.

## GitHub Metadata

`.github/` owns issue/PR templates, a copyable PR review template, description-check workflows, security/governance workflows, Docker publishing, and CI. Current CI runs on pushes to `main` and `dev` plus pull requests, compiles Python, checks committed frontend outputs, validates frontend architecture/syntax, emits report-only focused-test guidance, and partitions pytest across four file-based shards through `tests/_shards.py`. Pytest setup requires functional bubblewrap PID/mount containment, FFmpeg and Chromium. Its prose-only bypass excludes `website/` and `assets/branding/`; those published-file contracts remain tested.

`CONTRIBUTING.md` owns the branch model: PRs target `dev`; `main` is the curated user-running branch fast-forwarded from stable `dev` commits. Contributors who accidentally target `main` should retarget the PR base without rebasing.

PR description checks:

- run on `pull_request_target`;
- check out only base-branch `.github/scripts`;
- skip bot PRs;
- require Summary, Linked Issue, Type of Change, duplicate-search checklist, and substantive How to Test content as the hard description gate;
- classify changed paths as docs-only, tooling, backend/runtime, or UI-sensitive from GitHub's file API while executing only base-branch checker code;
- treat app-run and screenshot/clip checkboxes as author attestations, require an actual media link/attachment for UI-sensitive changes, and report runtime/visual evidence gaps separately from malformed descriptions;
- serialize mergeability labeling behind description validation and avoid granting `ready for review` to drafts or changes with outstanding runtime/visual evidence;
- update a bot comment and reconcile `ready for review`, `needs work`, `needs runtime validation`, and `needs visual evidence` labels where those labels exist.

Issue description checks:

- validate bug or feature sections based on labels;
- require bug reports to include the exact 12-character revision/date shape produced by `git show -s --abbrev=12 --format='%h (%cs)' HEAD`;
- flag unfilled dropdown placeholders such as `-- Please Select --`;
- route public vulnerability reports toward GitHub Security Advisories;
- update a bot comment and swap status labels;
- remove the workflow-owned review label when an issue closes so closed issues do not retain stale readiness state.

Security metadata includes container Trivy SARIF upload, Dockerfile lint, dependency review, secret scan, workflow security linting, GitHub default-setup CodeQL, Dependabot metadata, and hardened PR/issue description checks that avoid unsafe head-branch execution. `website/security-ci.md` documents CodeQL as a dynamic GitHub default-setup workflow; the repo should not add a checked-in CodeQL workflow while that default setup is active.

`scripts/pr_blocker_audit.py` is a read-only maintainer/contributor triage helper documented in `website/pr-blocker-audit.md`. It can fetch or ingest open PR metadata, estimate hot files and possible duplicate groups, and emit Markdown, JSON, or terminal reports. Its duplicate/blocker output is advisory, not an authority that a PR is blocked.

Before posting PRs or issues, compare drafts against current templates on latest `main` or current `dev` as appropriate for the target. Keep unpublished drafts and raw related-search exports out of tracked implementation specs unless intentionally promoted.

## Artifacts And Secrets

- Do not read `.env*` files unless a user explicitly asks for a controlled setup/debug step; never print their values.
- Backup files, logs, CLI JSON, and raw issue/PR search exports can contain sensitive local data.
- Do not commit raw GitHub JSON unless there is an explicit maintainer reason. Prefer compact Markdown reports when publishing analysis.
- Specs are implementation truth. Planning, research, branch notes, and draft reports belong in tracked project docs when promoted.

## Development Checks

Common local checks:

```bash
./venv/bin/pytest tests/path.py::test_name
./venv/bin/python -m py_compile app.py routes/*.py src/*.py
node --check static/js/changed-file.js
docker compose config
docker compose up -d --build
docker compose logs --tail=120 odysseus
```

Run the app for user-facing or integration changes. Unit tests and syntax checks do not replace end-to-end verification for UI, Docker, provider, auth, or routing behavior.

## Shared Test Helpers

`tests/helpers/` owns reusable test scaffolding. `cli_loader.load_script()` loads CLI files without running their `main()` entrypoint. `db_stubs` owns small DB stand-ins for tests that should not import a real app database. `import_state` owns conservative `sys.modules` and parent-module-attribute restoration for tests that install fake modules or import route files under alternate stubs. `tests/README.md` documents helper conventions and review expectations.

`tests/test_runtime_behavior_regressions.py` drives the agent loop with a fake streaming model to check offered tools, clarification/plan policy, explicit native request scope and mocked browser cleanup. Negative-web cases require a model call, an answer delta and terminal completion while withholding web tools. These deterministic runtime assertions do not establish live-model or browser acceptance.

## Harness And Publication Boundaries

`tests/conftest.py` defaults the static origin to an ephemeral loopback port and publishes/restores `ODYSSEUS_TEST_STATIC_ORIGIN`; an explicit validated port can still collide. Threaded serving avoids idle-connection head-of-line blocking. Per-process random private data/mail/embedding/cache/temp paths and optional xdist basetemp avoid sharing runtime state; explicit basetemp wins. Disposable database helpers use isolated factories/cleanup rather than ORM reloads. A teardown guard removes and fails newly leaked bare `src`/`core` modules, using exact ModuleType/falsey-file detection; preexisting modules and mocks are outside that guard. The Unix socket path length fixture is character-bounded, not a universal byte-length proof.

`tests/_shards.py` partitions collected files deterministically by weights and lexical tie-breaks across four CI shards; deselection occurs after collection/import. Optional parallel smoke and serial-only guards do not make every test xdist-safe. Pytest CI uses Node 20, locked npm dependencies, cached Chromium downloads keyed by OS/lockfile, system browser libraries, FFmpeg and a functional unprivileged bubblewrap probe. It uses setup-uv cache without automatic prune and pytest skip reasons/duration reporting. These workflow provisions are not evidence that every browser, media or containment test passed on another host.

`tests/helpers/document_source.py` provides ordered implementation membership and lexical construct extraction; email JS helper enumeration is immediate rather than recursive. Stylesheet helpers follow shell order and cache identities; keyframe/hygiene/import/procfs guards are bounded textual or AST checks, not full semantic control-flow/browser proofs. The computed-style fixture, browser artifacts and reported historical suite counts remain specific to their recorded environments.

`scripts/odysseus-dev` owns separate development data/database/browser-cache paths, path-hashed port blocks, service/reserved-port refusal, readiness login and fingerprinted stop state. Hash collisions, released fallback-port races and environment-derived defaults prevent a universal isolation guarantee. `scripts/odysseus-smoke` exercises deterministic model/email HTTP scenarios and reports declared gaps; it is not full browser/product acceptance and cleanup/reused-instance state remains scoped. `scripts/ref_parity_audit.py` compares heuristic added-line probes and exact filtered tree-path sets, not behavioral equivalence; mutable refs can be reread during a run. `scripts/generate_env_reference.py` checks generated reference bytes and handwritten notes from bounded AST/helper/regex discovery, not exhaustive runtime environment use; malformed Python can yield no detected rows and source citations are file-based.

`.github/workflows/deploy-pages.yml` builds `website/` into `_site`; preview encoding targets website assets and branding lives under `assets/branding/`. Docker excludes both website and branding source roots. The macOS app script uses moved branding with best-effort `sips` icon generation; no native build or license clearance follows from source inspection. Docker publishing emits version/short-SHA trace tags and inspects those plus latest, without registry-enforced immutability. Trivy jobs use the Docker Buildx driver and free local build cache before vulnerability database work; this is CI lifecycle behavior, not an instruction to prune controller state.

Description checks retain the public numeric Linked Issue requirement; an exact maintainer-preview repository allows a beginning `N/A` value while still requiring the section after comment stripping. This checks syntax, not issue existence or branch rulesets. Dependency-review blocking applies to canonical repository pull requests; fork pip-audit advice is a separate policy. None of these workflows grants publication authority.

Maintained frontend publication requires repository-owned LF text attributes on both mapped paths, checked through isolated Git attribute evaluation. It validates the complete plan before writes, copies bytes without rewriting imports or bundling and never deletes unowned static assets; publication is not a transactional live deployment. Acorn checks manifest-selected dependency direction, canonical URL identity, used legacy exceptions, exact precache arrays and cycles, not all legacy browser behavior.

## Chroma Mount And Focused Safety Regressions

All three Compose variants mount `chromadb-data:/data`, matching the current image configuration rather than pre-1.0 `/chroma/chroma`. Existing writable-layer data needs rescue before container recreation; `docs/chromadb-volume-migration.md` describes writer shutdown/cold backup, preserved backup image, SQLite/HNSW checks, empty-volume seeding, isolated staging and count comparison. `website/backup-restore.md` links that procedure. Static YAML/document tests cover declared paths and safety markers; they do not execute or certify migration against a live Chroma image.

Focused regressions now cover invalid history truncation preserving messages, failed bulk Trash moves/selective expunge, disabled-auth app configuration gates, durable cleanup message counts/delete-boundary races and generated-image ownership/cache/attribution. Test presence and historical counts do not establish a new successful run; runtime validation evidence is separate.

## Notes Extraction Validation Surface

Notes tests include domain/characterization/application/infrastructure/reminder/workflow contracts, recurrence differential cases across timezones, raw-state cloneability, draft timing and selected-context acceptance. Browser fixtures exercise canonical/facade identity, desktop/mobile/tablet interactions, teardown and first-time offline graph loading; source guards now read feature modules through `tests/helpers/notes_source.py`. `scripts/verify_frontend_runtime.py` checks all manifest outputs in a Python-only distribution, while `scripts/verify_notes_browser_runtime.py` probes an authenticated temporary native app. These require their declared dependencies/browser environment; committed receipts and screenshots are historical evidence, not a fresh acceptance run. Runtime serving still requires no frontend build or Node installation.

## Skills Extraction And Upload Contract Tests

`tests/test_skills_audit_characterization.py` freezes 22 route declarations, 28 Python interfaces and canonical OpenAPI (17 paths, 22 operations, five schemas) from committed fixtures; constrained fields, supported default factories, model configuration and aliases are checked structurally. Unsupported factories fail without execution. `tests/test_skills_audit_services.py` exercises real route/service wiring with offline model/agent/manager substitutes, including prompt seams, owner/request authority, one-use approvals and cancellation/stream closure. Fixtures do not prove arbitrary validators, live execution or authorization by themselves. ReDoS guards now scan/import `services/skills/evaluation.py`. The Gitleaks exception is constrained to one exact historical digest, the generic-api-key rule and two named paths; this maintenance does not claim a new full-history scan.

`tests/test_upload_no_login.py` covers disabled-auth access after setup, missing/unconfigured manager failures, token ownership/scopes, legacy ownerless uploads and OCR preservation on rejection. Its loopback-host parametrization does not itself change TestClient transport identity, so source behavior and narrower assertions remain the evidence for absence of a loopback bypass.

## Current Gaps

- Fresh install smoke coverage across Linux native, Docker, macOS native/app, Windows native, WSL/Git Bash, missing Node/npm, missing Chroma service, and GPU overlays remains a roadmap item.
- Frontend source/output and architecture gates exist; they do not establish full browser or platform correctness.
- Real deployment, GPU and cross-platform install verification still require separate runs beyond CI gates.
- Optional dependency behavior is broad; remaining gaps include local STT missing-`faster-whisper`, Kokoro's Python/GPU degraded matrix, and provider/OAuth combinations not covered by focused tests.
- GitHub description-check scripts and `scripts/pr_blocker_audit.py` need continued local fixtures for section parsing, placeholder stripping, label swaps, workflow-safe behavior, and duplicate/hot-file heuristics.
- Spec bootstrap rules lack meta tests for reading `_readme.md`, spec shape, `.env*` handling, draft/report placement, and shared helper conventions.
- NVIDIA helper install/`.env` mutation paths and real Docker/GPU startup are not covered by local tests.
- Bash/Zsh completion behavior is not covered.
- `tests/KNOWN_FAILURES.md` records observed environment-dependent full-suite failures and prerequisites; its measured basis is historical, so verify applicability rather than treating it as a passing waiver.
- There is no central CLI redaction/sensitive-output regression matrix across backup, logs, mail, MCP, tasks, and webhook scripts.
- Dependency/image pinning policy is mixed: Python requirements are mostly unpinned, SearXNG is pinned, Chroma image currently uses `latest`, npm uses a lockfile, and browser MCP uses cache-gated `@playwright/mcp@latest`.