fix(security): verify deployed trust before dependency work

.github/workflows/ci.yml

@@ -155,6 +155,17 @@ jobs:
         if: steps.docs-check.outputs.docs_only != 'true'
         with:
           python-version: "3.11"
+      # Establish system trust before dependency installers or their hooks run.
+      # Keep one apt pass for FFmpeg and bubblewrap; containment is checked later.
+      - name: Install FFmpeg and bubblewrap and validate system CA trust
+        if: steps.docs-check.outputs.docs_only != 'true'
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y --no-install-recommends ffmpeg bubblewrap ca-certificates
+          python scripts/check_ca_trust.py --system
+          command -v ffmpeg
+          ffmpeg -version | head -n 1
+
       # uv resolves the same versions as pip but installs ~5x faster, and its
       # cache keeps each shard from re-downloading the whole requirement set.
       - uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7  # v10.2.0
@@ -166,6 +177,7 @@ jobs:
           prune-cache: false
       - run: |
           uv pip install --system --require-hashes --only-binary=:all: --no-deps -r requirements-bootstrap.txt
+          python scripts/check_ca_trust.py --python
           uv pip install --system -r requirements.txt
         if: steps.docs-check.outputs.docs_only != 'true'
       - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020  # v7.0.0
@@ -187,16 +199,6 @@ jobs:
         if: steps.docs-check.outputs.docs_only != 'true'
       - run: mkdir -p data  # sqlite DB lives at ./data/app.db
         if: steps.docs-check.outputs.docs_only != 'true'
-      # One apt pass for FFmpeg (media integration tests) and bubblewrap
-      # (containment, verified in the next step).
-      - name: Install FFmpeg and bubblewrap
-        if: steps.docs-check.outputs.docs_only != 'true'
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y --no-install-recommends ffmpeg bubblewrap ca-certificates
-          command -v ffmpeg
-          ffmpeg -version | head -n 1
-
       - name: Validate Python and system CA trust
         if: steps.docs-check.outputs.docs_only != 'true'
         run: python scripts/check_ca_trust.py --system --python

docs/tls-trust.md

@@ -16,27 +16,50 @@ The pinned bootstrap protects package integrity even when the old installer's ce
 On Linux/OpenSSL deployments, inspect both the effective CA file and hashed CA directory, plus the two Python bundles:
 
 ```sh
-python scripts/check_ca_trust.py --system --python --overrides
+python scripts/check_ca_trust.py --system --python --overrides --verbose
 ```
 
 Explicit PEM bundles can also be passed as positional arguments. The check rejects the four published root fingerprints, all sixteen published intermediate certificates (including their cross-signs), the cross-signed G3 issuer in the published test chains, and malformed/empty bundles. Checking intermediates matters because adding one as a custom trust anchor can restore an affected chain even after its roots have been removed.
 
 `--overrides` checks every nonempty `LLM_CA_BUNDLE`, `SSL_CERT_FILE`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`, `PIP_CERT`, `NODE_EXTRA_CA_CERTS` and `GRPC_DEFAULT_SSL_ROOTS_FILE_PATH` setting in the checker's process environment, including configured fallback settings. Requests/pip hashed-directory overrides are inspected too; `--system` covers `SSL_CERT_DIR`. Run with the application's environment and filesystem access. The script does not load `.env` or application configuration files; pass any CA paths configured only there explicitly, for example `python scripts/check_ca_trust.py --system --python /path/to/private-ca.pem`. Missing or invalid configured files fail the check. Selecting only `--overrides` with no configured paths also fails instead of reporting that stores were validated.
 
+`--verbose` reports the Python executable/prefix, user home, effective UID/GID where available, and resolved CA paths, including bundles selected from user-installed packages. Compare these with the running application's identity and interpreter. It does not dump environment variables or credentials.
+
 This is an offline inventory check of known certificate fingerprints. It does not modify stores, reject requests at runtime, model native OS revocation, or detect arbitrary reissued certificates sharing the compromised keys. Chromium's policy separately matches public keys. Check mounted or replaced CA files again after deployment, remove affected trust through the owning platform/client configuration, and restart clients that cache SSL contexts. A passing build cannot validate custom files introduced later.
 
 Both Docker build stages refresh system CA certificates and bootstrap pip/certifi before dependency installation. The build fails if known affected CA certificates remain. Node uses the corrected system store through `--use-openssl-ca`. Python gRPC uses that same PEM file through `GRPC_DEFAULT_SSL_ROOTS_FILE_PATH`; updating certifi alone does not replace gRPC's independent wheel bundle. After dependency installation, the image checks gRPC's selected file as well. An explicit gRPC override must be readable and free of known affected CA certificates: gRPC can otherwise fall back to its affected bundled defaults. Upstream gRPC has [merged a bundle update](https://github.com/grpc/grpc/commit/8b32af3bf498c2ba9667410eaa8c836e700e97e6); as checked on 2026-10-21, its [release backport](https://github.com/grpc/grpc/pull/43700) is still open and the latest published grpcio wheel is 1.84.0. Retain the checked override until a released replacement's actual trust behavior has been verified.
 
 Chromium uses a separate trust store. The Docker image installs a managed `CADistrustedCertificates` policy for the eight compromised intermediate public keys, covering their cross-signs without depending on background revocation updates. It adds no trusted certificate and does not disable certificate verification. The policy and public-certificate provenance are documented in [the browser policy note](../docker/chromium-ca-policy.md). Native browser installations and independently managed profiles require their own platform policy or vendor update; this image policy does not configure host browsers.
 
-Rebuild with a fresh base and dependency layers when remediating an existing deployment:
+To replace only Odysseus in an already running Compose stack, build from the corrected checkout with a distinct local image tag. Use the same Compose files and project name as the existing deployment for every command below:
 
 ```sh
+export ODYSSEUS_IMAGE="odysseus-local:ca-$(git rev-parse --short=12 HEAD)"
 docker compose build --pull --no-cache odysseus
-docker compose up -d odysseus
-docker compose exec odysseus python scripts/check_ca_trust.py --system --python --grpc --overrides
+docker compose up -d --no-deps --pull never --no-build odysseus
+docker image inspect --format '{{.Id}}' "$ODYSSEUS_IMAGE"
+docker inspect --format '{{.Image}}' "$(docker compose ps -q odysseus)"
 ```
 
+The two image IDs must match. Keep the selected `ODYSSEUS_IMAGE` in the deployment configuration for subsequent recreations. Compose can prefer the registry image when both `image` and `build` are present; the explicit pull/build flags keep this replacement on the local build. `--no-deps` leaves the existing sidecars running and is intended for an existing stack. For registry deployments, select a verified corrected immutable tag or digest. A merge to `dev` publishes `:dev`; it does not update `:latest`, which is published from `main`.
+
+Check the standard image as its configured app account, using the image's Python interpreter and the same user-bin PATH prefix as the entrypoint:
+
+```sh
+docker compose exec -T --user 0 odysseus sh -ec '
+  gosu_bin=$(command -v gosu)
+  app_user=$(getent passwd "${PUID:-1000}" | cut -d: -f1)
+  test -n "$app_user"
+  exec "$gosu_bin" "$app_user" env PATH="/app/.local/bin:$PATH" \
+    /usr/local/bin/python /app/scripts/check_ca_trust.py \
+    --system --python --grpc --overrides --verbose
+'
+```
+
+`gosu` selects the account's HOME and drops privileges before Python runs, so the check includes its persisted user packages. The entrypoint itself is not rerun. Plain `docker compose exec ... python` uses the image's default root user and can miss `/app/.local` packages. A custom service user, entrypoint, virtual environment or launch command requires matching that deployment's actual user, HOME and Python interpreter instead of copying the standard command unchanged. Docker exec inherits container-creation environment, not variables exported later by the app or entrypoint: reproduce any additional Python/TLS settings with exec's `--env` options, and pass CA paths configured only in application files explicitly. Confirm the verbose identity and selected paths before accepting the result.
+
+For containers started non-root or without the SETUID/SETGID capabilities needed by `gosu`, use `docker compose exec --user UID:GID --env HOME=/actual/app/home` with the deployment's actual numeric IDs, home, PATH and interpreter instead. Do not add capabilities merely to run this check.
+
 SearXNG is a separate HTTPS client and image; updating the Odysseus image does not patch it. Inspect its effective CA path and update or rebuild that image independently. Also inspect persisted `/app/.local` packages, remote Cookbook environments, and externally configured MCP runtimes. Image replacement does not necessarily replace those environments.
 
 For behavioral validation, all twelve [published test chains](https://emudhra.we-have-a-problem.com/chains.zip), including the AAA cross-signs, must fail certificate verification through the effective stores. Confirm that an ordinary known-good HTTPS request still succeeds. In Chromium, confirm the managed policy is loaded at `chrome://policy` and test the incident endpoints through the actual browser path with certificate verification enabled. The PEM checker cannot inspect Chromium's policy or its active verifier. Package versions and a successful image build alone do not prove that a running deployment selects the corrected store.

scripts/check_ca_trust.py

@@ -135,9 +135,16 @@ def main(argv: list[str] | None = None) -> int:
     parser.add_argument("--python", action="store_true", help="check certifi and pip's separate vendored bundle")
     parser.add_argument("--grpc", action="store_true", help="check gRPC's selected default CA file")
     parser.add_argument("--overrides", action="store_true", help="check configured LLM/HTTPX/Requests/pip/Node/gRPC CA overrides")
+    parser.add_argument("--verbose", action="store_true", help="show the runtime identity, Python executable and selected CA paths")
     args = parser.parse_args(argv)
     if not (args.bundles or args.system or args.python or args.grpc or args.overrides):
         parser.error("select --system, --python, --grpc, --overrides, or explicit PEM bundle paths")
+    if args.verbose:
+        print(f"Python executable: {sys.executable}")
+        print(f"Python prefix: {sys.prefix}")
+        print(f"User home: {Path.home()}")
+        if hasattr(os, "geteuid"):
+            print(f"Effective user: uid={os.geteuid()} gid={os.getegid()}")
     try:
         paths = list(args.bundles)
         if args.system:
@@ -154,6 +161,8 @@ def main(argv: list[str] | None = None) -> int:
             raise ValueError("no CA bundle files selected; also select --system, --python, or an explicit path")
         errors = []
         for path in paths:
+            if args.verbose:
+                print(f"CA bundle: {path}")
             try:
                 check_bundle(path)
             except (OSError, ValueError, ssl.SSLError) as exc:

tests/test_ca_trust.py

@@ -10,6 +10,7 @@ from pathlib import Path
 import shutil
 import ssl
 import subprocess
+import sys
 import tempfile
 from types import SimpleNamespace
 import unittest
@@ -44,6 +45,43 @@ class CATrustTests(unittest.TestCase):
     def test_unaffected_emsign_root_remains_usable(self):
         checker.check_bundle(self.fixtures / "emsign-g1.pem")
 
+    def test_verbose_python_check_reports_and_rejects_affected_user_bundle(self):
+        # A fresh interpreter must see persisted user packages, independently
+        # of pytest's imported certifi/pip modules and any active test venv.
+        with tempfile.TemporaryDirectory() as tmp:
+            home = Path(tmp)
+            env = dict(os.environ, HOME=str(home), PYTHONUSERBASE=str(home / ".local"))
+            env.pop("PYTHONNOUSERSITE", None)
+            env.pop("PYTHONPATH", None)
+            python = getattr(sys, "_base_executable", sys.executable)
+            site_result = subprocess.run(
+                [python, "-m", "site", "--user-site"], env=env, capture_output=True, text=True,
+            )
+            self.assertEqual(site_result.returncode, 0, site_result.stderr)
+            user_site = Path(site_result.stdout.strip())
+            for package in ("certifi", "pip", "pip/_vendor", "pip/_vendor/certifi"):
+                directory = user_site / package
+                directory.mkdir(parents=True, exist_ok=True)
+                (directory / "__init__.py").write_text(
+                    "from pathlib import Path\ndef where(): return str(Path(__file__).with_name('cacert.pem'))\n"
+                )
+            application_bundle = user_site / "certifi/cacert.pem"
+            pip_bundle = user_site / "pip/_vendor/certifi/cacert.pem"
+            for bundle in (application_bundle, pip_bundle):
+                bundle.write_text(PUBLIC_CERTIFICATES["emsign-g1.pem"])
+            command = [python, str(ROOT / "scripts/check_ca_trust.py"), "--python", "--verbose"]
+            clean = subprocess.run(command, env=env, capture_output=True, text=True)
+            self.assertEqual(clean.returncode, 0, clean.stderr)
+            self.assertIn(str(application_bundle), clean.stdout)
+            self.assertIn(str(pip_bundle), clean.stdout)
+            self.assertIn(str(home), clean.stdout)
+            self.assertIn("Python executable:", clean.stdout)
+            application_bundle.write_text(PUBLIC_CERTIFICATES["emsign-g3.pem"])
+            affected = subprocess.run(command, env=env, capture_output=True, text=True)
+            self.assertEqual(affected.returncode, 1, affected.stderr)
+            self.assertIn(str(application_bundle), affected.stderr)
+            self.assertIn("emSign ECC Root CA - G3", affected.stderr)
+
     def test_compromised_emsign_path_is_rejected(self):
         with self.assertRaisesRegex(ValueError, "emSign ECC Root CA - G3"):
             checker.check_bundle(self.fixtures / "emsign-g3.pem")

tests/test_ci_authoritative_validation.py

@@ -33,3 +33,16 @@ def test_python_tests_are_authoritative():
 
     assert "python -m pytest -q" in python_tests
     assert "continue-on-error:" not in python_tests
+
+
+def test_ci_checks_system_trust_before_project_installation():
+    python_tests = _indented_block(_WORKFLOW.read_text(), "python-tests", 2)
+    system_check = python_tests.index("python scripts/check_ca_trust.py --system\n")
+    system_update = python_tests.index("sudo apt-get install -y --no-install-recommends ffmpeg bubblewrap ca-certificates")
+    bootstrap = python_tests.index("uv pip install --system --require-hashes")
+    python_check = python_tests.index("python scripts/check_ca_trust.py --python\n")
+    application_install = python_tests.index("uv pip install --system -r requirements.txt")
+
+    assert system_update < system_check < bootstrap < python_check < application_install
+    assert system_check < python_tests.index("npm ci")
+    assert system_check < python_tests.index("npx playwright install")