mirror of
https://github.com/pewdiepie-archdaemon/odysseus.git
synced 2026-10-15 19:32:21 +02:00
Nguyen Thanh DatandPR #6166 Reviewer
c3819e3cbf
* fix(docker): mount the ChromaDB volume where the image actually persists
chromadb/chroma runs `chroma run /config.yaml`, and that config is one line:
persist_path: "/data". The image declares no VOLUME, so everything the store
writes lives in the container's writable layer. The chromadb-data volume was
mounted at /chroma/chroma — where pre-1.0 images kept the store — so it caught
nothing and the entire vector store was destroyed on every container recreate.
Nothing errors when that happens. The collection survives with a count of 0, so
retrieval simply returns nothing from then on, which reads as "no match".
docs/backup-restore.md tells users to archive the chromadb-data volume, which
means the documented backup was empty too.
Fixed in all three compose files. The volume was never written to, so there is
nothing to migrate.
* fix(tests,docs): harden volume-path test helper and document upgrade migration
tests/test_chromadb_volume_path.py
- _chromadb_volumes() now normalises long-form dict volume entries
(source/target/type map) to 'source:target' strings, preventing
AttributeError crashes if a future Compose change uses that syntax.
- test_chromadb_does_not_use_the_pre_1_0_path now uses split(':')[1]
comparison instead of endswith(), so mount-option suffixes (:z, :ro)
cannot mask a wrong legacy target path.
- Three new unit tests validate the helper against both YAML syntaxes
and the mount-option edge case explicitly.
docs/chromadb-volume-migration.md (new)
- Step-by-step guide for users upgrading from pre-#6166 Docker installs
whose ChromaDB data accumulated in the container's writable layer.
- Covers: detecting the situation, backing up via 'docker cp', seeding
the named volume before recreating the container, verification, and
rollback. Validated with disposable Docker volumes.
- Distinguishes fresh installs (no action needed) from existing
installs with live writable-layer data (action needed before recreate).
docs/backup-restore.md
- Extended ChromaDB caveat block to cross-reference the new migration
guide for users upgrading from pre-#6166.
Closes the P1 documentation gap and P2/P3 test-robustness findings
from the PR #6166 adversarial review.
* fix(docs,tests): eliminate ChromaDB migration data-loss risks and enforce safe cutover
- Require service quiescence before cold database copy to prevent torn SQLite pages
- Mandate container preservation via immutable Docker image commit before recreation
- Enforce positive empty check on destination volume prior to seeding
- Add staging container validation on an isolated loopback port before Compose cutover
- Provide verified rollback procedures prohibiting return to legacy /chroma/chroma mount
- Add regression test coverage for all migration safety invariants
---------
Co-authored-by: PR #6166 Reviewer <reviewer@odysseus-review.local>
143 lines
5.8 KiB
Markdown
143 lines
5.8 KiB
Markdown
---
|
|
layout: default
|
|
---
|
|
|
|
# Backup & Restore
|
|
|
|
Odysseus keeps all of your state in the `data/` directory — the SQLite database
|
|
(`app.db`), the Fernet encryption key (`data/.app_key`), the vault, memory, RAG
|
|
indexes, personal documents, and uploads. The `scripts/odysseus-backup` tool
|
|
snapshots that directory into a single gzip tarball and restores it later.
|
|
|
|
Snapshots are safe to take while the app is running: SQLite databases are copied
|
|
through SQLite's own `.backup` API rather than a raw file copy, so an in-flight
|
|
write can't corrupt the snapshot.
|
|
|
|
> **A snapshot contains your secrets.** The tarball includes the Fernet
|
|
> encryption key (`data/.app_key`), the vault, sessions, and any stored
|
|
> provider/API tokens — so treat it like a password. Store backups somewhere
|
|
> private, never commit them to Git, and prefer an encrypted destination when
|
|
> copying them offsite.
|
|
|
|
## Quick start
|
|
|
|
Run the tool from the repository root:
|
|
|
|
```bash
|
|
# Create a snapshot → backups/odysseus-backup-<YYYYMMDD-HHMMSS>.tar.gz
|
|
./scripts/odysseus-backup snapshot
|
|
|
|
# List existing snapshots (most recent first)
|
|
./scripts/odysseus-backup list
|
|
|
|
# Check a tarball's integrity without extracting it
|
|
./scripts/odysseus-backup verify backups/odysseus-backup-20260101-120000.tar.gz
|
|
|
|
# Restore (destructive — see the warning below)
|
|
./scripts/odysseus-backup restore backups/odysseus-backup-20260101-120000.tar.gz --yes
|
|
```
|
|
|
|
The script depends only on the Python standard library, so any `python3` on your
|
|
`PATH` will run it — you don't need the app's virtualenv active.
|
|
|
|
Every command prints a JSON result. Add `--pretty` for indented output.
|
|
|
|
## Commands
|
|
|
|
### `snapshot`
|
|
|
|
Writes a `tar.gz` of `data/` to `backups/<timestamp>.tar.gz`.
|
|
|
|
| Flag | Effect |
|
|
| --- | --- |
|
|
| `--out PATH` | Write to a specific path instead of the default `backups/` location. Must be **outside** `data/`. |
|
|
| `--include-research` | Include `data/deep_research/` (skipped by default — research runs are large). |
|
|
| `--include-attachments` | Include `data/mail-attachments/` (skipped by default — cached IMAP extractions, re-derivable). |
|
|
|
|
By default the snapshot includes everything under `data/` **except**
|
|
`deep_research/` and `mail-attachments/`. Personal uploads and documents are
|
|
included.
|
|
|
|
```bash
|
|
# Snapshot straight to a mounted NAS path
|
|
./scripts/odysseus-backup snapshot --out /mnt/nas/odysseus-$(date +%F).tar.gz
|
|
|
|
# Full snapshot including research runs and mail attachments
|
|
./scripts/odysseus-backup snapshot --include-research --include-attachments
|
|
```
|
|
|
|
### `list`
|
|
|
|
Lists the tarballs in `backups/`, most recent first, with size and modification
|
|
time.
|
|
|
|
### `verify PATH`
|
|
|
|
Opens the tarball read-only and walks every member to confirm it is intact and
|
|
safe to restore. Nothing is extracted. Use this before relying on an old backup
|
|
or after copying one across machines.
|
|
|
|
### `restore PATH --yes`
|
|
|
|
Overwrites `data/` from a tarball.
|
|
|
|
> **Restore is destructive.** It replaces the current `data/` directory. `--yes`
|
|
> is required so a mistyped command can't wipe your live state.
|
|
|
|
Restore is not a blind delete: before extracting, the tool **renames your current
|
|
`data/` to `data.before-restore-<timestamp>`** in the repository root. If a
|
|
restore turns out to be wrong, your previous state is still there — delete the
|
|
restored `data/` and rename the stashed directory back. The restore path is also
|
|
validated entry-by-entry: archives containing absolute paths, `..` segments,
|
|
symlinks, or anything outside `data/` are rejected.
|
|
|
|
## Scheduling offsite backups
|
|
|
|
The tarball output composes cleanly with cron and any copy tool. For example, a
|
|
nightly snapshot copied offsite:
|
|
|
|
```cron
|
|
0 3 * * * cd /path/to/odysseus && ./scripts/odysseus-backup snapshot --out "/mnt/nas/odysseus-$(date +\%F).tar.gz"
|
|
```
|
|
|
|
Swap the `--out` target for `scp`, `rclone`, `s3cmd`, or similar to push the
|
|
snapshot to remote storage.
|
|
|
|
## Docker vs native installs
|
|
|
|
The tool reads `data/` and writes `backups/` relative to the repository root, so
|
|
where you run it matters:
|
|
|
|
- **Native installs** — run it from the repo root as shown above. `data/` and
|
|
`backups/` are both in the repo directory.
|
|
- **Docker** — `docker-compose.yml` bind-mounts the host's `./data` to
|
|
`/app/data`, so the live data is also present on the host. **Run the tool on
|
|
the host** from the repo root; the snapshot reads the bind-mounted `./data` and
|
|
writes to `./backups` on the host. Running it *inside* the container is not
|
|
recommended, because `backups/` is not a mounted volume and the tarball would
|
|
be lost when the container is recreated.
|
|
|
|
> **ChromaDB caveat (Docker only).** In the Docker setup, ChromaDB stores its
|
|
> vectors in a separate Compose-managed volume (declared as `chromadb-data`),
|
|
> **not** under `./data`. `odysseus-backup` therefore does not capture the Docker
|
|
> ChromaDB store. Back it up separately if you need it. Compose prefixes the
|
|
> volume with the project name, so find the real name first
|
|
> (`docker volume ls | grep chromadb`), then archive it — for example:
|
|
>
|
|
> ```bash
|
|
> docker run --rm -v <project>_chromadb-data:/data -v "$PWD":/backup \
|
|
> alpine tar czf /backup/chromadb.tar.gz -C /data .
|
|
> ```
|
|
>
|
|
> On native installs ChromaDB lives at `data/chroma/` and is included in the
|
|
> snapshot normally.
|
|
>
|
|
> **Upgrading from a pre-#6166 install?** The old Compose files mounted
|
|
> `chromadb-data` at the wrong path (`/chroma/chroma`), so the volume was
|
|
> always empty while live data accumulated in the container's writable layer
|
|
> under `/data`. If you have a running pre-#6166 ChromaDB container, follow
|
|
> [docs/chromadb-volume-migration.md](https://github.com/odysseus-dev/odysseus/blob/dev/docs/chromadb-volume-migration.md)
|
|
> to rescue that data before recreating or deleting existing containers. The
|
|
> migration guide provides step-by-step instructions for recovery, backup
|
|
> verification, volume seeding, rollback, and re-indexing.
|