odysseus/specs/persistence.md

Persistence

Last updated: dev@c74f475 | 2026-10-15

Scope

This spec covers durable state in:

• core/database.py;
• src/database.py;
• src/runtime_paths.py;
• src/constants.py;
• core/models.py;
• core/session_manager.py;
• core/atomic_io.py;
• src/attachment_refs.py, src/upload_handler.py, and routes/upload_routes.py for durable upload references and retention;
• JSON stores managed by core/auth.py, src/settings.py, src/api_key_manager.py, src/preset_manager.py, src/integrations.py, src/upload_handler.py, src/personal_docs.py, src/research_handler.py, src/bg_jobs.py, routes/prefs_routes.py, canonical routes/contacts/contacts_routes.py and routes/vault/vault_routes.py plus their shims, routes/cookbook_routes.py, and memory/skills managers;
• canonical routes/email/email_helpers.py scheduled-email storage;
• routes/backup_routes.py and scripts/odysseus-backup;
• runtime data under data/.

Database Shape

core/database.py owns SQLAlchemy models and startup migrations. src/database.py is a compatibility re-export for legacy imports. Route and service code commonly owns its own SessionLocal() lifecycle instead of using one central unit-of-work wrapper.

The default database is SQLite at DATA_DIR/app.db. src.runtime_paths and src.constants own the data-dir default: source runs use the repository data/ directory, frozen builds default to ~/.odysseus/data, and ODYSSEUS_DATA_DIR overrides both. The core requirements include psycopg2-binary for PostgreSQL driver availability. SQLAlchemy can point at a non-SQLite DATABASE_URL, but current startup migrations/backfills are SQLite-first and often use sqlite3, PRAGMA, or SQLite catalog queries. External DBs are not fully migration-compatible unless those helpers are made backend-neutral.

After Base.metadata.create_all(), init_db() resolves file-backed SQLite paths from SQLAlchemy's parsed engine URL and attempts to restrict the main database plus existing -journal, -wal, and -shm sidecars to 0600 on POSIX. Driver-qualified, query-tagged, and local file: URI forms are covered; non-SQLite, in-memory SQLite, and Windows paths are skipped. A failed POSIX chmod is logged because the database and sidecars can contain password/token hashes and encrypted provider material.

Timestamp defaults use utcnow_naive() so existing naive DateTime columns stay UTC without the deprecated datetime.utcnow() default.

Current model families include:

• chat sessions, messages, and chat_messages_fts transcript-search state/triggers;
• documents and document versions;
• gallery albums/images, editor drafts, signatures, generated-media metadata;
• email accounts, model endpoints, MCP servers, comparisons;
• provider auth sessions for OAuth/device-flow-backed provider credentials;
• API tokens, admin-global webhooks, user tools/tool data, integrations;
• crew members, scheduled tasks, task runs, notification logs, background tool jobs, notes;
• memory rows, calendar calendars, calendar events, and email-calendar invitation revision/tombstone state.

Chat persistence stores model-readable text plus compact attachment-reference lines in chat_messages.content, while structured references remain in message metadata. Provider data URLs used by the live turn are not duplicated into the durable transcript. The FTS migration recreates insert/update triggers to omit inline media and scrubs legacy indexed rows that still contain data URLs.

Current calendar/task persistence includes CalDAV remote identity columns (CalendarCal.remote_href, CalendarCal.remote_etag, CalendarEvent.remote_href, CalendarEvent.remote_etag), CalendarEvent.caldav_sync_pending for retryable writeback state, and ScheduledTask.character_id for built-in task persona selection.

Note.gallery_id links drawing notes to stable gallery images. CalendarEvent source-email identifiers preserve invitation provenance; EmailCalendarInvitation keeps owner/sender/source/recurrence revision and cancellation state for replay decisions.

EmailAccount includes encrypted password fields plus Google OAuth fields (oauth_provider, encrypted access/refresh tokens, token expiry) and optional display_name. Startup migrations add those OAuth/display columns idempotently for older databases.

Email default-account state is serialized per owner. Startup normalizes legacy duplicate defaults and installs a per-owner unique default constraint/index; create, delete/promotion, set-default, demo teardown, and user rename perform their default transition in one locked transaction. Multi-owner rename locks are acquired in canonical order, so a stale concurrent default mutation fails closed instead of recreating multiple defaults.

core/models.py owns pure dataclasses used by SessionManager. It does not own database persistence.

routes/email/email_helpers.py owns a second SQLite database at data/scheduled_emails.db for scheduled email, summary, reply, tag, sender-signature, urgency-alert, calendar-extraction, and cache state. Its migrations and owner backfills are local to that module, not core/database.py, and those auxiliary tables are owner-scoped.

Migration Policy

Odysseus does not use Alembic. core.database.init_db() runs at module import, before FastAPI lifespan startup. Base.metadata.create_all() creates missing tables; hand-written _migrate_* functions add or reshape legacy columns.

Runtime behavior:

• migrations must be idempotent;
• SQLite foreign keys are enabled for every engine connection;
• new SQLAlchemy columns need matching startup migration code;
• legacy ownerless/shared rows may exist and must be handled by owner-aware route helpers.

Startup backfills include document-owner backfill from linked sessions, blanket legacy owner assignment for SQL and selected JSON stores, user_prefs.json per-user nesting, email account seeding from legacy settings, and encryption rewrites for legacy plaintext endpoint, signature, and email secrets. Failed encryption rewrites are logged and retried on later startup.

Owner-claiming is partly automatic and partly manual. core.database._migrate_assign_legacy_owner() assigns many ownerless SQL rows and selected JSON records to the primary admin when auth data exists, while scripts/claim_ownerless.py is an explicit local utility for claiming older ownerless memories, skills, sessions, documents, gallery rows, and comparisons.

Ownership And Access

Owner columns are security-relevant. Current owner-bearing domains include sessions, documents, gallery images/albums, editor drafts, model endpoints, signatures, API tokens, user tools/tool data, comparisons, crew members, scheduled tasks/task runs, memories, notes, calendars/events, email accounts, and integrations. Webhooks are admin-global today and do not have an owner column.

Route code owns filtering for its domain. src.auth_helpers.owner_filter() is the common helper where available; gallery, documents, calendar, email, skills, and other surfaces also use local filters. Null-owner compatibility is domain-specific: shared endpoints may include null owners, while strict gates and disk stores may reject them. Do not rely on frontend filtering for access control.

src.owner_identity defines the storage-only Default/Local owner __odysseus_local__. effective_storage_owner() maps an absent caller to it only when auth is explicitly disabled, preserves named owners, and rejects request sentinels; storage_owner_for_request() also resolves bearer tokens to their real owner. This is a new canonical contract, not a completed migration. SQL NULL and missing JSON owners still usually mean legacy/shared/unscoped compatibility; older route dependencies can return "", chat/agent paths can pass None, and calendar routes retain fallback-owner behavior. Email account helpers treat ownerless rows as single-user/global only for empty-owner mode; for non-empty owners, old ownerless rows are visible only when mailbox/from-address matches. Multi-user callers must continue to pass or derive a non-empty effective owner deliberately.

Secrets And Local Stores

ModelEndpoint includes cached/hidden/pinned model lists, endpoint kind, refresh mode/interval/timeout, model type, supports-tools, owner, optional provider_auth_id, provider metadata, and encrypted API key columns. New endpoint columns need matching startup migration helpers.

ProviderAuthSession rows hold OAuth/device-flow credential state for providers such as ChatGPT Subscription. Endpoints can reference those rows through provider_auth_id; deletion/cleanup must preserve auth rows still referenced by another endpoint and remove orphaned provider-auth rows only after the last endpoint reference is gone.

McpServer includes stdio/SSE/HTTP transport config, plaintext env JSON, OAuth config, disabled tool names, and encrypted generic OAuth token/client state in oauth_tokens. Generic MCP token storage treats valid non-object JSON as empty state on reads and replaces it with an object on the next write instead of crashing callers.

CalendarCal.account_id links synced local calendars back to one saved CalDAV account so multi-account sync/writeback can round-trip remote calendar identity. Remote href/etag columns on calendars and events preserve CalDAV server identity across pull/push cycles, while caldav_sync_pending marks local create/update/delete work that still needs remote writeback.

EncryptedText owns transparent encrypted-at-rest DB columns via src.secret_storage for model endpoint keys and signatures. Email passwords and Google OAuth access/refresh tokens are String columns encrypted/decrypted manually. Integrations, CalDAV/CardDAV prefs, and other JSON stores can use src.secret_storage directly. API tokens are bcrypt-hashed, API-key manager state uses data/.key plus data/api_keys.json with restrictive chmod where supported, and vault state in data/vault.json is chmod-restricted JSON. Legacy plaintext rows are tolerated until migration or rewrite.

Current JSON/local stores include:

• data/auth.json for users, password hashes, TOTP, privileges, and auth settings;
• data/sessions.json for persisted browser session tokens;
• data/settings.json, user preferences, feature flags, integration settings, and data/embedding_endpoint.json;
• presets, API key manager state, memory/skills state, upload metadata, personal docs indexes, research JSON, background jobs, contacts/vault JSON, and task/cookbook auxiliary state.

Cookbook state lives under the shared DATA_DIR path through the COOKBOOK_STATE_FILE constant. Search cache/analytics, FastEmbed cache fallback, uploads, generated media, logs, and auxiliary SQLite stores also resolve from shared data-dir constants and must work with source, Docker, and frozen data-dir defaults.

core.atomic_io owns atomic file-write behavior for auth/settings/integration-style stores. Its JSON and text writers use a random UUID suffix per write, so concurrent writers in the same process cannot collide on a constant PID-derived temporary path, and a finally cleanup unlinks any orphaned temp after serialization, fsync, or replace failure while ignoring cleanup errors. Upload metadata uses its own locked atomic writer with .bak recovery and can rewrite owner fields plus owner-qualified index keys during user rename. Its cache signature covers the live and backup files by device, inode, size, nanosecond mtime, and ctime; reads recheck the whole signature so same-timestamp corruption or replacement cannot pair stale parsed data with a fresh identity. Destructive reads require a valid live index and never use backup recovery as deletion authority. Attachment-bearing chat/session, document, note, and calendar writers take owner-checked upload reservations before durable writes; reservations share the upload-index lock with cleanup and access-time refresh. Cleanup receives a complete reference snapshot and removes only expired uploads proven unreferenced with coherent index state. Missing/incomplete scans fail closed, and index rows are restored when byte deletion fails.

Memory mutations have their own fail-closed durability contract: MemoryManager.load_all_for_update() raises MemoryStoreUnreadable for a corrupt or unreadable memory.json, and read-modify-write callers use that strict path so they cannot replace an unreadable store with an empty one. Read-only load_all() remains lenient and can degrade to no memories; legacy memory.txt migration remains supported.

Persisted memories, skills, documents, email, RAG chunks, notes, and other user-editable data are untrusted when reintroduced to model context. Route and processor code must pass them through the untrusted-context contract described in context-building.md and auth-security.md.

Backup And Restore

routes/backup_routes.py owns narrow admin HTTP JSON export/import for memories, presets, skills, settings, features, and prefs. Skill import writes through the disk-backed skills manager API. This is not a full system restore path.

scripts/odysseus-backup owns local data/ snapshot/restore, with some large/runtime subtrees such as deep research and mail attachments behind flags. It uses SQLite backup APIs, includes secret-bearing key files and stores, validates restore archives against path escapes and link entries, and skips list entries that disappear or become unstatable during directory iteration. Backup artifacts should be treated as sensitive.

Transitional Notes

The repo still mixes database-backed and JSON-backed persistence. Some domains have both legacy manager state and newer SQLAlchemy rows. src.database remains a live compatibility import path. services/memory/memory.py and services/memory/memory_vector.py now re-export canonical src memory classes; preserve compatibility unless the change explicitly migrates a store and includes backfill/tests.

Docker bind-mounts data/, logs/, cache/local state, and optional Chroma state. The entrypoint repairs ownership for PUID/PGID before dropping privileges. POSIX secret files attempt restrictive chmod; Windows permission hardening is best-effort/no-op through platform compatibility helpers.

ChromaDB/vector stores are optional durable storage outside data/app.db; missing Chroma degrades RAG, memory-vector, and tool-index features without blocking core SQLite/JSON persistence. Vector collections can be lane-suffixed for custom HTTP embeddings versus FastEmbed fallback. See documents-rag-uploads.md.

Run Evidence And New Durable State

Session adds cwd, endpoint id, memory-extraction/injection and skill-injection toggles, thinking mode, temperature/max-token overrides, and cumulative cost. ModelEndpoint.model_tool_modes stores per-model tool-schema preferences. ScheduledTask.request_authority_json stores the server-only admitted request snapshot. Startup migration helpers add these columns to legacy schemas. BackgroundToolJob holds origin owner/session/tool/query, status, payload/summary, and delivered message id; NotificationLog persists owner-attributed task completion/error notification text.

src.agent_runtime.effect_log stores append-only claims, outcomes, and observations in DATA_DIR/effects/<root-run-id>.jsonl, with launch-generation indexes alongside them. Claims are fsynced before backend invocation; persistence failure refuses execution. Reload appends interrupted outcomes for unsettled non-running claims, while eligible owned RUNNING process claims survive for background lifecycle settlement; it never manufactures success or fresh state. Appends merge durable history under an advisory lock before allocating positions. Cross-process locking and directory-fsync guarantees are POSIX-specific; other platforms have narrower guarantees. Claim files use restrictive creation, no-follow/regular-file/single-link checks, strict row validation and a tolerated torn final tail. Non-claim persistence can degrade to volatile evidence; durable claim failure still refuses invocation. Advisory-lock append merges durable rows, allocates positions and repairs only a torn tail; reader locking does not silently repair. Launch indexes use fsync/replace ordering, with narrower non-POSIX guarantees. These checks do not imply authenticated tamper-proof logs, constant-cost growth or identical index hardlink checks. This store is evidence, not operation authority.

data/containment_grants.json and data/bg_jobs.json preserve containment/process identity across restart. Browser producer state resolves through BROWSER_RESOURCES_DIR in shared constants. These server control-plane stores are denied to native model filesystem operations; ownership and runtime integrity must remain separate from editable user artifacts.

Cleanup Counts And Chroma Volumes

src/cleanup_service.py derives preview and deletion eligibility from durable message rows, not the session counter. Recent-session protection uses a limited ID query; final DELETE rechecks owner/archive/age/importance/message threshold. DELETE RETURNING identifies cleanup-owned deletions where supported; legacy SQLite uses rowcount plus survivor queries, with estimated space attribution bounded to the deleted count. Session-cache eviction occurs after commit.

The standard, AMD and NVIDIA Compose files mount chromadb-data at /data; native Chroma uses data/chroma/. Pre-fix Docker stores may have live data in the old container writable layer despite an empty volume at /chroma/chroma; recreation can destroy that layer. docs/chromadb-volume-migration.md documents cold rescue, image preservation, empty-volume seeding and staging/count verification before cutover. This is an operator procedure, not startup migration or proof of data recovery.

Current Gaps

• Migration behavior is centralized but long and manual.
• Ownerless legacy rows make access-control reasoning harder.
• Some JSON store shapes are only documented by manager code and tests.
• Startup migrations lack a legacy-schema/idempotence test harness for owner backfills, encrypted-secret rewrites, and repeated runs.
• JSON-store atomicity is inconsistent across stores, though shared atomic writers, upload metadata recovery, prefs, and strict memory mutations now have focused coverage.
• New local stores require explicit ownership, backup, and control-plane access decisions; a filesystem root alone does not grant access to server state.