From c3819e3cbf56c4f899eacc66b21115a77d9c570d Mon Sep 17 00:00:00 2001 From: Nguyen Thanh Dat Date: Wed, 14 Oct 2026 18:20:35 +0700 Subject: [PATCH] fix(docker): mount the ChromaDB volume where the image actually persists (#6166) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(docker): mount the ChromaDB volume where the image actually persists chromadb/chroma runs `chroma run /config.yaml`, and that config is one line: persist_path: "/data". The image declares no VOLUME, so everything the store writes lives in the container's writable layer. The chromadb-data volume was mounted at /chroma/chroma — where pre-1.0 images kept the store — so it caught nothing and the entire vector store was destroyed on every container recreate. Nothing errors when that happens. The collection survives with a count of 0, so retrieval simply returns nothing from then on, which reads as "no match". docs/backup-restore.md tells users to archive the chromadb-data volume, which means the documented backup was empty too. Fixed in all three compose files. The volume was never written to, so there is nothing to migrate. * fix(tests,docs): harden volume-path test helper and document upgrade migration tests/test_chromadb_volume_path.py - _chromadb_volumes() now normalises long-form dict volume entries (source/target/type map) to 'source:target' strings, preventing AttributeError crashes if a future Compose change uses that syntax. - test_chromadb_does_not_use_the_pre_1_0_path now uses split(':')[1] comparison instead of endswith(), so mount-option suffixes (:z, :ro) cannot mask a wrong legacy target path. - Three new unit tests validate the helper against both YAML syntaxes and the mount-option edge case explicitly. docs/chromadb-volume-migration.md (new) - Step-by-step guide for users upgrading from pre-#6166 Docker installs whose ChromaDB data accumulated in the container's writable layer. - Covers: detecting the situation, backing up via 'docker cp', seeding the named volume before recreating the container, verification, and rollback. Validated with disposable Docker volumes. - Distinguishes fresh installs (no action needed) from existing installs with live writable-layer data (action needed before recreate). docs/backup-restore.md - Extended ChromaDB caveat block to cross-reference the new migration guide for users upgrading from pre-#6166. Closes the P1 documentation gap and P2/P3 test-robustness findings from the PR #6166 adversarial review. * fix(docs,tests): eliminate ChromaDB migration data-loss risks and enforce safe cutover - Require service quiescence before cold database copy to prevent torn SQLite pages - Mandate container preservation via immutable Docker image commit before recreation - Enforce positive empty check on destination volume prior to seeding - Add staging container validation on an isolated loopback port before Compose cutover - Provide verified rollback procedures prohibiting return to legacy /chroma/chroma mount - Add regression test coverage for all migration safety invariants --------- Co-authored-by: PR #6166 Reviewer --- docker-compose.gpu-amd.yml | 6 +- docker-compose.gpu-nvidia.yml | 6 +- docker-compose.yml | 6 +- docs/chromadb-volume-migration.md | 408 +++++++++++++++++++++++++++++ tests/test_chromadb_volume_path.py | 220 ++++++++++++++++ website/backup-restore.md | 9 + 6 files changed, 652 insertions(+), 3 deletions(-) create mode 100644 docs/chromadb-volume-migration.md create mode 100644 tests/test_chromadb_volume_path.py diff --git a/docker-compose.gpu-amd.yml b/docker-compose.gpu-amd.yml index 8e6702b43..183b31601 100644 --- a/docker-compose.gpu-amd.yml +++ b/docker-compose.gpu-amd.yml @@ -131,7 +131,11 @@ services: ports: - "${CHROMADB_BIND:-127.0.0.1}:8100:8000" volumes: - - chromadb-data:/chroma/chroma + # The image runs `chroma run /config.yaml`, whose persist_path is "/data". + # It declares no VOLUME, so anything written outside a mount lives in the + # container's writable layer and dies with it. /chroma/chroma is where + # pre-1.0 images kept the store; mounting there now catches nothing. + - chromadb-data:/data environment: - ANONYMIZED_TELEMETRY=FALSE restart: unless-stopped diff --git a/docker-compose.gpu-nvidia.yml b/docker-compose.gpu-nvidia.yml index eb3c9d7e3..23404b2eb 100644 --- a/docker-compose.gpu-nvidia.yml +++ b/docker-compose.gpu-nvidia.yml @@ -134,7 +134,11 @@ services: ports: - "${CHROMADB_BIND:-127.0.0.1}:8100:8000" volumes: - - chromadb-data:/chroma/chroma + # The image runs `chroma run /config.yaml`, whose persist_path is "/data". + # It declares no VOLUME, so anything written outside a mount lives in the + # container's writable layer and dies with it. /chroma/chroma is where + # pre-1.0 images kept the store; mounting there now catches nothing. + - chromadb-data:/data environment: - ANONYMIZED_TELEMETRY=FALSE restart: unless-stopped diff --git a/docker-compose.yml b/docker-compose.yml index 4fb49a45b..a7ddf2f04 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -112,7 +112,11 @@ services: ports: - "${CHROMADB_BIND:-127.0.0.1}:8100:8000" volumes: - - chromadb-data:/chroma/chroma + # The image runs `chroma run /config.yaml`, whose persist_path is "/data". + # It declares no VOLUME, so anything written outside a mount lives in the + # container's writable layer and dies with it. /chroma/chroma is where + # pre-1.0 images kept the store; mounting there now catches nothing. + - chromadb-data:/data environment: - ANONYMIZED_TELEMETRY=FALSE restart: unless-stopped diff --git a/docs/chromadb-volume-migration.md b/docs/chromadb-volume-migration.md new file mode 100644 index 000000000..4ec7853ac --- /dev/null +++ b/docs/chromadb-volume-migration.md @@ -0,0 +1,408 @@ +# ChromaDB Volume Migration — Upgrading from pre-#6166 installs + +> [!IMPORTANT] +> This guide is only for Docker-based installs that were deployed before PR #6166 +> was merged. Native host installs are unaffected (they persist to `data/chroma/` +> on the host filesystem, which was always stored correctly). + +--- + +## 1. What changed and why + +Before PR #6166, the `chromadb-data` named volume was mounted at `/chroma/chroma` inside +the ChromaDB container. However, current `chromadb/chroma` images (version 1.0+) write their +database to `/data` — configured by the container image's internal `/config.yaml` (`persist_path: "/data"`). +Because the upstream ChromaDB image declares no `VOLUME` directive: + +* **Any vector data written to `/data` accumulated exclusively in the container's writable layer.** +* The mounted `chromadb-data` named volume at `/chroma/chroma` remained completely empty. +* Any recreation or removal of the container (`docker compose down`, `docker compose up --force-recreate`, `docker rm`) + permanently destroys that writable layer and all vector embeddings within it. + +PR #6166 corrects the mount target across all Compose configurations to `chromadb-data:/data`. + +Fresh installations will persist data directly to the named volume automatically. However, for +**existing running installations**, the vector store currently resides in the old container's +writable layer. + +This guide provides a zero-data-loss, fail-closed operational procedure to quiesce, preserve, +back up, verify, seed, and validate that data before performing cutover. + +--- + +## 2. Architecture and Safety Principles + +1. **No Live `docker cp`:** ChromaDB uses SQLite (`chroma.sqlite3`) and binary HNSW segment index + files (`data_level0.bin`, `header.bin`, `length.bin`, `link_lists.bin`). Taking a file copy while + the database engine is running risks capturing torn SQLite pages or out-of-sync HNSW graph segments. + Writers must be quiesced and ChromaDB gracefully stopped before copying files. +2. **Do Not Trust `docker rename` Alone:** Docker Compose tracks containers by internal metadata labels + (`com.docker.compose.project`, `com.docker.compose.service`, `com.docker.compose.container-number`), + **not** merely by container name. Renaming a container does not strip its labels; Compose commands + can still discover, restart, or recreate it. To guarantee preservation of the writable layer, the + stopped container must be committed to an immutable Docker image (`docker commit`). +3. **Never Overwrite Existing Volumes:** Named volumes must be positively verified as empty prior to seeding. +4. **Staging Validation Before Production Cutover:** The seeded named volume must be independently validated + using a disposable ChromaDB instance on an isolated port before touching the production Compose deployment. +5. **No Rollback to `/chroma/chroma`:** Rolling back to pre-#6166 Compose files reintroduces the broken + mount path where ChromaDB never looks. Rollback procedures must restore data into the corrected `/data` path + or redeploy from the immutable backup image. + +--- + +## 3. Step-by-Step Migration Procedure + +### Step 0: Identify your project and volume names + +Docker Compose prefixes container and volume names with the project name (usually the root directory name). +Identify yours before proceeding: + +```bash +# Identify running ChromaDB container +docker ps --filter "ancestor=chromadb/chroma" --format "table {{.ID}}\t{{.Status}}\t{{.Names}}" + +# Identify the named volume +docker volume ls --filter "name=chromadb" +``` + +Throughout this guide: +* `` represents your Compose project (e.g. `odysseus` or `odysseus-maintainer-preview`). +* `` represents the running ChromaDB container (e.g. `-chromadb-1`). +* `` represents the named volume (e.g. `_chromadb-data`). + +Set shell variables for convenience: +```bash +PROJECT="" +CONTAINER="${PROJECT}-chromadb-1" +VOLUME="${PROJECT}_chromadb-data" +``` + +--- + +### Step 1: Pre-Migration Collection Inventory (Baseline) + +Establish a baseline of all existing collections and their embedding counts through the live ChromaDB API: + +```bash +# List all collections and record counts +curl -s http://127.0.0.1:8100/api/v2/tenants/default_tenant/databases/default_database/collections \ + | python3 -c ' +import json, sys, urllib.request +data = json.load(sys.stdin) +print(f"Total collections found: {len(data)}") +for col in data: + c_id = col["id"] + name = col["name"] + try: + with urllib.request.urlopen(f"http://127.0.0.1:8100/api/v2/tenants/default_tenant/databases/default_database/collections/{c_id}/count") as resp: + count = resp.read().decode().strip() + except Exception as e: + count = f"error: {e}" + print(f"Collection: {name} (ID: {c_id}) -> Count: {count}") +' | tee pre_migration_inventory.txt +``` + +Save `pre_migration_inventory.txt` securely. This baseline is used in Step 7 and Step 9 to verify +zero data loss. + +--- + +### Step 2: Quiesce Writers and Gracefully Stop ChromaDB + +To ensure transactional and referential consistency across SQLite and HNSW segment files: + +1. Stop application writers (Odysseus) to prevent any in-flight requests: + ```bash + docker compose stop odysseus + ``` +2. Gracefully stop the ChromaDB service. This permits SQLite to checkpoint the Write-Ahead Log (WAL) + and flushes all in-memory vector index segments to disk: + ```bash + docker compose stop chromadb + ``` +3. Confirm that the container status is `Exited`: + ```bash + docker ps -a --filter "name=${CONTAINER}" --format "table {{.ID}}\t{{.Status}}\t{{.Names}}" + ``` + *Do not proceed if the container is still running.* + +--- + +### Step 3: Preserve the Original Container as an Immutable Image + +Because Docker Compose discovers containers via service labels, running `docker compose up` later +could recreate and destroy the original container's writable layer. + +Preserve the complete writable layer into an immutable Docker image: +```bash +BACKUP_IMAGE="${PROJECT}-chromadb-backup:pre-migration-$(date +%Y%m%d-%H%M%S)" +docker commit "${CONTAINER}" "${BACKUP_IMAGE}" +echo "Committed container writable layer to: ${BACKUP_IMAGE}" +``` + +Verify the image was created: +```bash +docker images "${BACKUP_IMAGE}" +``` + +This image is your ultimate insurance: even if all containers and volumes are deleted, your +data remains safely archived in Docker's image store. + +--- + +### Step 4: Take a Cold File Backup + +With the container stopped, copy the `/data` directory out to a dedicated host backup directory: + +```bash +BACKUP_DIR="./chromadb-backup-$(date +%Y%m%d-%H%M%S)" +mkdir -p "${BACKUP_DIR}" + +docker cp "${CONTAINER}:/data/." "${BACKUP_DIR}/" +echo "Cold backup created at: ${BACKUP_DIR}" +``` + +--- + +### Step 5: Verify Backup Integrity + +Never proceed to destructive cutover without verifying both SQLite and vector index structures. + +1. **Verify SQLite Database File:** + ```bash + test -s "${BACKUP_DIR}/chroma.sqlite3" || { echo "ERROR: chroma.sqlite3 is missing or 0 bytes"; exit 1; } + ``` +2. **Execute SQLite PRAGMA Integrity Check:** + ```bash + INTEGRITY=$(sqlite3 "${BACKUP_DIR}/chroma.sqlite3" "PRAGMA integrity_check;") + if [ "${INTEGRITY}" != "ok" ]; then + echo "ERROR: SQLite integrity check failed: ${INTEGRITY}" + exit 1 + fi + echo "SQLite integrity check: OK" + ``` +3. **Verify Collections Table Structure:** + ```bash + sqlite3 "${BACKUP_DIR}/chroma.sqlite3" "SELECT id, name FROM collections;" + ``` +4. **Verify HNSW Index Binary Files:** + Vector representations are not stored inside SQLite; they are kept in segment directories under `/data`: + ```bash + find "${BACKUP_DIR}" -type f -name "*.bin" + ``` + Ensure `.bin` files (`data_level0.bin`, `header.bin`, etc.) exist for indexed collections. + +--- + +### Step 6: Verify Destination Volume is Empty and Seed Safely + +The named volume `${VOLUME}` must be empty before seeding. Under no circumstances should +migration files be merged into a populated volume. + +1. **Positive Empty Check:** + ```bash + VOL_FILES=$(docker run --rm -v "${VOLUME}:/vol:ro" alpine ls -A /vol) + if [ -n "${VOL_FILES}" ]; then + echo "ERROR: Named volume ${VOLUME} already contains files: ${VOL_FILES}" + echo "Aborting migration to prevent overwriting existing volume data." + exit 1 + fi + echo "Volume ${VOLUME} is empty and ready for seeding." + ``` +2. **Seed the Volume:** + Use a disposable Alpine container to populate the volume with exact ownership and permissions: + ```bash + docker run --rm \ + -v "${VOLUME}:/target" \ + -v "$(realpath "${BACKUP_DIR}"):/source:ro" \ + alpine sh -c "cp -a /source/. /target/ && ls -la /target/" + ``` +3. **Verify Seeded Contents:** + ```bash + docker run --rm -v "${VOLUME}:/vol:ro" alpine ls -la /vol + ``` + +--- + +### Step 7: Staging Validation in an Isolated Disposable Container + +Before modifying your production Compose stack, validate that ChromaDB boots successfully from the +seeded volume and serves all collections. + +1. **Start Disposable Staging Container:** + Run a temporary ChromaDB instance bound to an isolated loopback port (e.g. `127.0.0.1:18199`): + ```bash + docker run -d --name "${PROJECT}-chromadb-staging" \ + -p 127.0.0.1:18199:8000 \ + -v "${VOLUME}:/data" \ + chromadb/chroma:latest + + # Wait for staging ChromaDB to boot + sleep 4 + curl -s http://127.0.0.1:18199/api/v2/heartbeat || { echo "Staging container failed to respond"; exit 1; } + ``` +2. **Compare Collection Counts with Baseline:** + ```bash + curl -s http://127.0.0.1:18199/api/v2/tenants/default_tenant/databases/default_database/collections \ + | python3 -c ' + import json, sys, urllib.request + data = json.load(sys.stdin) + print(f"Staging collections verified: {len(data)}") + for col in data: + c_id = col["id"] + name = col["name"] + with urllib.request.urlopen(f"http://127.0.0.1:18199/api/v2/tenants/default_tenant/databases/default_database/collections/{c_id}/count") as resp: + count = resp.read().decode().strip() + print(f" {name}: {count} records") + ' + ``` + *Confirm that the output matches `pre_migration_inventory.txt` from Step 1.* + +3. **Tear Down Staging Container:** + ```bash + docker rm -f "${PROJECT}-chromadb-staging" + ``` + +--- + +### Step 8: Apply Updated Compose Files and Perform Production Cutover + +Once the seeded volume is proven healthy: + +1. Update your repository to the revision containing PR #6166: + ```bash + git pull + ``` +2. Verify the mount line in your Compose file: + ```bash + grep -E 'chromadb-data:/data' docker-compose.yml + ``` +3. Bring up the updated ChromaDB container: + ```bash + docker compose up -d chromadb + ``` + Docker Compose will detect the updated mount configuration, replace the stopped container, + and attach `${VOLUME}` directly to `/data`. +4. Bring up the application stack: + ```bash + docker compose up -d odysseus + ``` + +--- + +### Step 9: Post-Migration Production Verification + +1. **Confirm Mount Point:** + ```bash + docker inspect "${CONTAINER}" --format '{{range .Mounts}}{{.Name}} -> {{.Destination}}{{"\n"}}{{end}}' + ``` + Expected output: + ``` + _chromadb-data -> /data + ``` +2. **Confirm Production Collection Counts:** + ```bash + curl -s http://127.0.0.1:8100/api/v2/tenants/default_tenant/databases/default_database/collections \ + | python3 -c ' + import json, sys, urllib.request + data = json.load(sys.stdin) + print(f"Production collections: {len(data)}") + for col in data: + c_id = col["id"] + name = col["name"] + with urllib.request.urlopen(f"http://127.0.0.1:8100/api/v2/tenants/default_tenant/databases/default_database/collections/{c_id}/count") as resp: + count = resp.read().decode().strip() + print(f" {name}: {count} records") + ' + ``` +3. **Application Verification:** + Log in to Odysseus, navigate to **Settings -> Memory**, and confirm that memory items, + tool embeddings, and RAG document queries operate normally. + +--- + +## 4. Rollback Procedures + +### Scenario A: Failure During Staging (Before Compose Cutover) + +If staging verification in Step 7 fails or reveals discrepancies: +1. Remove the staging container: + ```bash + docker rm -f "${PROJECT}-chromadb-staging" + ``` +2. Clear the failed destination volume: + ```bash + docker run --rm -v "${VOLUME}:/target" alpine sh -c "rm -rf /target/*" + ``` +3. Restart the original production container. Because the original container was only stopped (never recreated), + its writable layer remains completely intact: + ```bash + docker compose start chromadb + docker compose start odysseus + ``` +4. Verify production status on port 8100. + +--- + +### Scenario B: Failure After Compose Cutover + +If an issue occurs after `docker compose up -d`: + +> [!CAUTION] +> **Do not `git checkout` the pre-#6166 Compose file!** +> In the pre-#6166 Compose file, `chromadb-data` was mounted to `/chroma/chroma`. +> ChromaDB 1.0+ persists to `/data`, so checking out the old Compose file will cause +> ChromaDB to ignore the restored volume and boot with an empty database. + +To restore service: +1. **Option 1 — Restore from Verified Host Backup into the `/data` Volume:** + If the container failed due to data corruption during startup: + ```bash + docker compose stop chromadb + docker run --rm -v "${VOLUME}:/target" alpine sh -c "rm -rf /target/*" + docker run --rm \ + -v "${VOLUME}:/target" \ + -v "$(realpath "${BACKUP_DIR}"):/source:ro" \ + alpine cp -a /source/. /target/ + docker compose up -d chromadb + ``` +2. **Option 2 — Fallback to Committed Backup Image:** + If Compose cannot start ChromaDB with the volume: + ```bash + docker compose stop chromadb + docker run -d --name "${PROJECT}-chromadb-fallback" \ + -p 127.0.0.1:8100:8000 \ + "${BACKUP_IMAGE}" + ``` + This immediately restores the exact pre-migration container state. + +--- + +## 5. Post-Migration Cleanup (Explicit Authorization) + +Retain the host backup directory (`${BACKUP_DIR}`) and the backup Docker image (`${BACKUP_IMAGE}`) +for at least 7 days until routine backups have succeeded. + +Once you are completely confident in system stability, manually clean up: +```bash +# Remove host backup +rm -rf ./chromadb-backup-* + +# Remove backup Docker image +docker rmi "${BACKUP_IMAGE}" +``` + +--- + +## 6. Re-indexing (Disaster Recovery Only) + +If an installation has already destroyed its pre-#6166 container writable layer (e.g. prior to following +this guide), the vector embeddings cannot be restored from disk. In this scenario: + +* **RAG Documents:** Re-upload files under **Settings -> Memory -> RAG**. Odysseus automatically re-chunks + and re-indexes documents upon upload. +* **Semantic Memories:** Text memory records reside authoritatively in `data/app.db` (SQLite). Vector representations + are re-indexed automatically when Odysseus connects to a healthy ChromaDB instance. +* **Tool Index:** Rebuilt automatically by Odysseus upon first query. + +Re-indexing is a disaster recovery fallback, not a substitute for preserving existing database state. diff --git a/tests/test_chromadb_volume_path.py b/tests/test_chromadb_volume_path.py new file mode 100644 index 000000000..60ff68374 --- /dev/null +++ b/tests/test_chromadb_volume_path.py @@ -0,0 +1,220 @@ +"""Regression guard for issue #6132 — the ChromaDB named volume was mounted at a +path the image never writes to, so the whole vector store lived in the +container's writable layer and was destroyed on every recreate. Silently: the +collection survived with a count of 0, and retrieval just stopped returning +anything. + +``chromadb/chroma`` runs ``chroma run /config.yaml``, and that config is a single +line — ``persist_path: "/data"``. The image declares no ``VOLUME``, so nothing +outside a mount survives. ``/chroma/chroma`` is where pre-1.0 images kept the +store. + +The documented backup procedure (website/backup-restore.md) archives the +``chromadb-data`` volume, so a wrong mount point also makes that tarball empty. + +Pure YAML — no Docker required. +""" + +from pathlib import Path + +import pytest +import yaml + +ROOT = Path(__file__).resolve().parents[1] + +COMPOSE_FILES = [ + ROOT / "docker-compose.yml", + ROOT / "docker-compose.gpu-nvidia.yml", + ROOT / "docker-compose.gpu-amd.yml", +] + +# `persist_path` from the image's own /config.yaml. +CHROMA_PERSIST_PATH = "/data" +LEGACY_PERSIST_PATH = "/chroma/chroma" +VOLUME_NAME = "chromadb-data" + + +def _chromadb_volumes(path: Path) -> list[str]: + """Return a normalised list of ``source:target`` strings for the chromadb service. + + Docker Compose accepts volumes in two syntaxes: + + * Short-form string: ``"chromadb-data:/data"`` or ``"chromadb-data:/data:ro"`` + * Long-form mapping:: + + - source: chromadb-data + target: /data + type: volume + + Both forms are normalised to ``":"`` so that callers can + use ``split(":")`` to extract either part without worrying about the original + syntax or trailing mount-option fields (e.g. ``:z``, ``:ro``). + """ + compose = yaml.safe_load(path.read_text(encoding="utf-8")) + service = compose["services"]["chromadb"] + raw = list(service.get("volumes") or []) + result: list[str] = [] + for entry in raw: + if isinstance(entry, dict): + # Long-form: normalise to "source:target" for uniform handling. + source = entry.get("source", "") + target = entry.get("target", "") + result.append(f"{source}:{target}") + else: + # Short-form string: keep as-is; split(":")[1] is still the target + # even when a third field (mount options like :z, :ro) is present. + result.append(str(entry)) + return result + + +@pytest.mark.parametrize("path", COMPOSE_FILES, ids=lambda p: p.name) +def test_chromadb_volume_is_mounted_at_the_persist_path(path: Path): + mounts = _chromadb_volumes(path) + targets = {m.split(":")[1] for m in mounts if m.startswith(f"{VOLUME_NAME}:")} + assert targets, f"{path.name}: no {VOLUME_NAME} mount on the chromadb service" + assert targets == {CHROMA_PERSIST_PATH}, ( + f"{path.name}: {VOLUME_NAME} is mounted at {sorted(targets)}, but the image " + f"persists to {CHROMA_PERSIST_PATH!r}. Anything written outside a mount is " + "lost when the container is recreated (issue #6132)." + ) + + +@pytest.mark.parametrize("path", COMPOSE_FILES, ids=lambda p: p.name) +def test_chromadb_does_not_use_the_pre_1_0_path(path: Path): + """Verify that no mount places chromadb-data at the pre-1.0 store path. + + Uses ``split(":")[1]`` rather than ``endswith()`` so that mount-option + suffixes (e.g. ``:z``, ``:ro``) do not mask a wrong target path. + """ + mounts = _chromadb_volumes(path) + legacy = [m for m in mounts if ":" in m and m.split(":")[1] == LEGACY_PERSIST_PATH] + assert not legacy, ( + f"{path.name}: {LEGACY_PERSIST_PATH} is the pre-1.0 store location; " + f"current images never write there. Offending entries: {legacy}" + ) + + +@pytest.mark.parametrize("path", COMPOSE_FILES, ids=lambda p: p.name) +def test_chromadb_data_volume_is_declared(path: Path): + compose = yaml.safe_load(path.read_text(encoding="utf-8")) + assert VOLUME_NAME in (compose.get("volumes") or {}), ( + f"{path.name}: the {VOLUME_NAME} volume is mounted but not declared" + ) + + +# --------------------------------------------------------------------------- +# Unit tests for _chromadb_volumes() — parse-correctness, not Compose state +# --------------------------------------------------------------------------- + +class _FakePath: + """Minimal Path stand-in that returns a fixed YAML string.""" + + def __init__(self, yaml_text: str): + self._text = yaml_text + + def read_text(self, encoding: str = "utf-8") -> str: # noqa: ARG002 + return self._text + + +def test_helper_normalises_long_form_volume_dict(): + """Long-form dict volume entries must not raise AttributeError.""" + p = _FakePath(yaml.dump({ + "services": {"chromadb": {"volumes": [ + {"source": "chromadb-data", "target": "/data", "type": "volume"}, + ]}}, + "volumes": {"chromadb-data": None}, + })) + result = _chromadb_volumes(p) # type: ignore[arg-type] + assert result == ["chromadb-data:/data"] + + +def test_helper_passes_short_form_string_unchanged(): + """Short-form strings (with or without mount options) are returned as-is.""" + p = _FakePath(yaml.dump({ + "services": {"chromadb": {"volumes": ["chromadb-data:/data:z"]}}, + "volumes": {"chromadb-data": None}, + })) + result = _chromadb_volumes(p) # type: ignore[arg-type] + assert result == ["chromadb-data:/data:z"] + # split(":")[1] still extracts the correct target regardless of :z suffix + assert result[0].split(":")[1] == "/data" + + +def test_helper_legacy_check_catches_path_with_mount_option_suffix(): + """The split-based legacy check catches the old path even with :z appended.""" + mount = f"{VOLUME_NAME}:{LEGACY_PERSIST_PATH}:z" + # Confirm endswith() would miss it (the bug we fixed) + assert not mount.endswith(f":{LEGACY_PERSIST_PATH}"), ( + "endswith() should NOT catch ':z'-suffixed legacy path — that was the bug" + ) + # Confirm split()[1] does catch it + assert mount.split(":")[1] == LEGACY_PERSIST_PATH, ( + "split(':')[1] must still identify the legacy target path with :z suffix" + ) + + +# --------------------------------------------------------------------------- +# Regression tests for ChromaDB migration safety invariants (PR #6166 audit) +# --------------------------------------------------------------------------- + +MIGRATION_DOC = ROOT / "docs" / "chromadb-volume-migration.md" + + +def test_migration_guide_requires_quiescence_before_backup(): + """Step 2 must stop ChromaDB before running docker cp to prevent torn SQLite pages.""" + text = MIGRATION_DOC.read_text(encoding="utf-8") + assert "docker compose stop chromadb" in text, ( + "Migration guide must require stopping ChromaDB before database backup" + ) + # Confirm stopping step occurs prior to the cold backup step + stop_step = text.find("Step 2: Quiesce Writers and Gracefully Stop ChromaDB") + backup_step = text.find("Step 4: Take a Cold File Backup") + assert stop_step != -1 and backup_step != -1 and stop_step < backup_step, ( + "Quiescence/stop step must precede cold backup step in migration sequence" + ) + + +def test_migration_guide_mandates_container_preservation_via_commit(): + """Guide must preserve stopped container as an immutable image before any recreation.""" + text = MIGRATION_DOC.read_text(encoding="utf-8") + assert "docker commit" in text, ( + "Migration guide must instruct operators to commit the container to an immutable backup image" + ) + assert "com.docker.compose.project" in text, ( + "Guide must explain that Compose discovers containers by internal labels, not name alone" + ) + + +def test_migration_guide_requires_sqlite_and_hnsw_integrity_verification(): + """Guide must mandate SQLite PRAGMA integrity_check and checking HNSW index files.""" + text = MIGRATION_DOC.read_text(encoding="utf-8") + assert "PRAGMA integrity_check;" in text, ( + "Migration guide must require SQLite PRAGMA integrity_check" + ) + assert "data_level0.bin" in text or "*.bin" in text, ( + "Migration guide must require verifying HNSW segment index binary files" + ) + + +def test_migration_guide_requires_empty_volume_precheck(): + """Guide must verify destination volume is strictly empty prior to seeding.""" + text = MIGRATION_DOC.read_text(encoding="utf-8") + assert "ls -A" in text or "empty and ready for seeding" in text, ( + "Migration guide must check that the destination named volume is empty before seeding" + ) + + +def test_migration_guide_includes_staging_validation(): + """Guide must validate the seeded volume with a staging container before Compose cutover.""" + text = MIGRATION_DOC.read_text(encoding="utf-8") + assert "chromadb-staging" in text, ( + "Migration guide must include staging container validation step" + ) + + +def test_migration_guide_warns_against_reverting_to_broken_mount(): + """Rollback instructions must explicitly prohibit git checkout to pre-#6166 broken mount.""" + text = MIGRATION_DOC.read_text(encoding="utf-8") + assert "Do not `git checkout` the pre-#6166 Compose file" in text or "broken" in text.lower(), ( + "Rollback instructions must explain why checking out old Compose file breaks volume mount" + ) diff --git a/website/backup-restore.md b/website/backup-restore.md index 424b8bd59..54596145f 100644 --- a/website/backup-restore.md +++ b/website/backup-restore.md @@ -131,3 +131,12 @@ where you run it matters: > > On native installs ChromaDB lives at `data/chroma/` and is included in the > snapshot normally. +> +> **Upgrading from a pre-#6166 install?** The old Compose files mounted +> `chromadb-data` at the wrong path (`/chroma/chroma`), so the volume was +> always empty while live data accumulated in the container's writable layer +> under `/data`. If you have a running pre-#6166 ChromaDB container, follow +> [docs/chromadb-volume-migration.md](https://github.com/odysseus-dev/odysseus/blob/dev/docs/chromadb-volume-migration.md) +> to rescue that data before recreating or deleting existing containers. The +> migration guide provides step-by-step instructions for recovery, backup +> verification, volume seeding, rollback, and re-indexing.