fix(security): refresh client trust and publish validated SearXNG images

Repair SearXNG system and certifi stores, validate native amd64/arm64 runtime images through Compose, and publish the exact verified artifacts through a trusted CI job. Adopt released Python, certifi, grpcio, uv and Docker CLI trust fixes, with strict controls for all fourteen public incident chains.

Part of #6784 and #6790.

Co-authored-by: Léo <leograndcontact@gmail.com>

docs/searxng-trust-image.md

@@ -0,0 +1,50 @@
+# SearXNG image trust correction
+
+SearXNG has independent system and Python certificate stores. Updating Odysseus's Python environment does not update this sidecar. The pinned SearXNG image includes certificate authorities listed in the public eMudhra incident inventory. `docker/searxng/Dockerfile` derives a corrected image from the same immutable application base, without downloading dependencies or changing the SearXNG version.
+
+The build removes the 21 known incident CA certificates by SHA-256 fingerprint from system CA source files and certifi, regenerates the system PEM bundle and hashed CA directory, and checks the generated stores. Only certificates present in the base are removed: the inspected amd64 image loses two system roots and one certifi root. Unaffected certificates remain. The image's existing certifi package version metadata is unchanged; this is an image-owned trust-content correction rather than a package upgrade.
+
+The runtime is Void Linux with its package manager removed. Do not substitute Alpine, Debian or XBPS package-install commands. Removing affected source certificates is necessary because the upstream root entrypoint runs `update-ca-certificates` again at startup.
+
+## Build and validate
+
+Run these commands from the repository root on a Docker host. The build context allowlist excludes credentials and unrelated application files. The repair requires no build-step network; pulling the pinned base still needs registry access.
+
+```sh
+# Reproduce the affected upstream behavior with offline public chain fixtures.
+docker build --network=none --target baseline-validation -f docker/searxng/Dockerfile .
+
+# Check corrected trust and actual upstream entrypoint/loopback health.
+docker build --network=none --target validation -f docker/searxng/Dockerfile .
+
+# Also exercise verified HTTPS through SearXNG's own curl client and a real search.
+# Requires outbound networking and working upstream engines.
+docker build --target live-validation -f docker/searxng/Dockerfile .
+
+# Build the runtime image and select it through the opt-in Compose overlay.
+docker compose -f docker-compose.yml -f docker/searxng/compose.yml build searxng
+docker compose -f docker-compose.yml -f docker/searxng/compose.yml up -d --no-deps searxng
+docker compose -f docker-compose.yml -f docker/searxng/compose.yml ps searxng
+```
+
+The overlay retains the base service's settings migration, persistent configuration, healthcheck, ports and capability limits. All Compose commands managing this local image must include the overlay; using only the base file selects its original image. Validation fixtures and validation scripts are excluded from the runtime stage.
+
+The offline checks use fourteen public incident chains, including two without Certificate Transparency entries, with a frozen verification time, hostname verification and affected positive controls. They independently exercise the system PEM, system hashed directory and certifi bundle. The unpatched baseline must accept 14/14, 14/14 and 7/14 respectively; the corrected image must reject all fourteen in every store. The startup check rechecks system trust after the upstream entrypoint regenerates it. Live validation requires an ordinary verified HTTPS request, certificate-verification failures for all fourteen incident hosts through SearXNG's own curl client after startup, and nonempty SearXNG JSON search results; network and engine availability can cause that optional target to fail.
+
+## Scope and rollout
+
+The `ci / SearXNG image` workflow builds on native amd64 and arm64 runners. Each architecture reproduces the affected upstream stores, checks the repaired image with build-step networking disabled, then starts the actual runtime through the base Compose service's settings wrapper, capabilities and healthcheck. The CI override removes published ports and uses a fresh temporary configuration volume; it does not replace the service's startup path. Live checks run against that existing service without repairing its trust stores. Pull requests perform these checks without registry credentials or publication.
+
+On the canonical repository's `dev`, `main` or initial `release/searxng-ca-trust` branch, successful native checks allow a separate job to publish the exact exported images to `ghcr.io/odysseus-dev/odysseus-searxng`. Explicit workflow runs on those branches can also request publication. The publisher verifies source revisions, source-input checksums and architecture digests before recording the immutable multiarch reference. Tags identify the commit and workflow attempt; no moving `latest` or `dev` tag is changed. Fresh native jobs then pull anonymously and repeat Compose/live validation. Only after both pass does the workflow produce the `searxng-release` artifact with `searxng-release.json` for the coordinated Compose pin update.
+
+For the first publication, a repository administrator must make the new GHCR package public before anonymous verification can succeed. Rerun the failed verification jobs after changing package visibility. The initial release branch permits this bootstrap before the workflow reaches the default branch; it does not merge or deploy the application. Future publication can use the normal trusted-branch push or explicit workflow trigger.
+
+Validate each target architecture on a native builder or an explicitly configured emulation builder. An amd64 success does not establish arm64 support. This patch preserves the upstream default identity and entrypoint; it does not alter host trust, existing running containers or operator-mounted/custom trust stores. Operators must inspect those overrides separately. Removing AAA Certificate Services can also affect unrelated legacy certificate chains anchored solely there; test configured engines and any private integrations before rollout.
+
+After architecture validation and independent review, publish an approved multi-architecture runtime image, record its immutable digest, and update the base and standalone GPU Compose pins together. The opt-in overlay permits local validation before a published reference exists. Recreate affected services after selecting the replacement. Keep the broader incident remediation tracker open for deployment and other runtime work.
+
+## Public fixture provenance
+
+Fingerprint inventory: <https://emudhra.we-have-a-problem.com/roots.json> and <https://emudhra.we-have-a-problem.com/impacted-intermediates.json>. The cross-signed issuer and chain fixtures come from <https://emudhra.we-have-a-problem.com/chains.zip> (archive SHA-256 `7b949956955fc501a30f4d98d2f513d34c0bbbe92e29a968194c96e097ce2345`). Fixtures contain public certificates only. Each fixture certificate's fingerprint is verified before chain checks.
+
+The same incident follow-up updates Odysseus's application bootstrap and independent bundled clients; see [outbound TLS trust](tls-trust.md). Those package updates do not change the derived SearXNG application/package versions. The base-image alignment regression requires its immutable upstream base to match the deployment pins until a verified derived release is selected, with the local overlay tag matching the application date.

docs/tls-trust.md

@@ -2,7 +2,7 @@
 
 Odysseus uses both Python package CA bundles and system trust. HTTPX uses certifi by default; the guarded web/PDF fetcher and webhook delivery explicitly use Python's system SSL context. A configured `LLM_CA_BUNDLE` adds certificates to system trust for model calls. Upgrading certifi alone cannot repair every outbound HTTPS path.
 
-For the October 2026 eMudhra incident, use certifi 2026.10.19 or newer, pip 26.2.2 or newer, and uv 0.13.4 or newer wherever uv is used. Pip has its own vendored certifi and also loads system roots. Update the operating system's certificate package separately. See the [Python security announcement](https://discuss.python.org/t/109576) and [incident details](https://emudhra.we-have-a-problem.com/).
+For the October 2026 eMudhra incident, use certifi 2026.10.23 or newer, pip 26.2.2 or newer, and uv 0.13.5 or newer wherever uv is used. Pip has its own vendored certifi and also loads system roots. Update the operating system's certificate package separately. See the [Python security announcement](https://discuss.python.org/t/109576) and [incident details](https://emudhra.we-have-a-problem.com/).
 
 Before other pip installs, bootstrap the selected environment with the integrity-pinned wheels:
 
@@ -27,7 +27,9 @@ Explicit PEM bundles can also be passed as positional arguments. The check rejec
 
 This is an offline inventory check of known certificate fingerprints. It does not modify stores, reject requests at runtime, model native OS revocation, or detect arbitrary reissued certificates sharing the compromised keys. Chromium's policy separately matches public keys. Check mounted or replaced CA files again after deployment, remove affected trust through the owning platform/client configuration, and restart clients that cache SSL contexts. A passing build cannot validate custom files introduced later.
 
-Both Docker build stages refresh system CA certificates and bootstrap pip/certifi before dependency installation. The build fails if known affected CA certificates remain. Node uses the corrected system store through `--use-openssl-ca`. Python gRPC uses that same PEM file through `GRPC_DEFAULT_SSL_ROOTS_FILE_PATH`; updating certifi alone does not replace gRPC's independent wheel bundle. After dependency installation, the image checks gRPC's selected file as well. An explicit gRPC override must be readable and free of known affected CA certificates: gRPC can otherwise fall back to its affected bundled defaults. Upstream gRPC has [merged a bundle update](https://github.com/grpc/grpc/commit/8b32af3bf498c2ba9667410eaa8c836e700e97e6); as checked on 2026-10-21, its [release backport](https://github.com/grpc/grpc/pull/43700) is still open and the latest published grpcio wheel is 1.84.0. Retain the checked override until a released replacement's actual trust behavior has been verified.
+Both Docker build stages use Python 3.14.9, refresh system CA certificates and bootstrap pip/certifi before dependency installation. The build fails if known affected CA certificates remain. Node uses the corrected system store through `--use-openssl-ca`. Python gRPC uses that same PEM file through `GRPC_DEFAULT_SSL_ROOTS_FILE_PATH`; updating certifi alone does not replace gRPC's independent wheel bundle. After dependency installation, the image checks gRPC's selected file as well. An explicit gRPC override must be readable and free of known affected CA certificates. [grpcio 1.84.1](https://github.com/grpc/grpc/releases/tag/v1.84.1) includes the upstream root-bundle correction and is the application minimum. Retain the checked override as a consistent image trust selection; older persisted Python environments and explicit client roots still need independent checks.
+
+The image also bundles Docker CLI [29.9.2](https://github.com/moby/moby/releases/tag/docker-v29.9.2), built with Go 1.26.10's distrust of the compromised intermediate keys. Cookbook uses this client for optional Docker operations. Default Compose does not mount the host Docker socket; local daemon access requires the explicit [host-Docker overlay](../docker/host-docker.yml), whose usual Unix-socket connection does not use TLS. The client update is maintenance of a shipped tool, not evidence that these local calls were exposed. It does not update the host daemon or other containers' clients and trust stores; assess those separately.
 
 Chromium uses a separate trust store. The Docker image installs a managed `CADistrustedCertificates` policy for the eight compromised intermediate public keys, covering their cross-signs without depending on background revocation updates. It adds no trusted certificate and does not disable certificate verification. The policy and public-certificate provenance are documented in [the browser policy note](../docker/chromium-ca-policy.md). Native browser installations and independently managed profiles require their own platform policy or vendor update; this image policy does not configure host browsers.
 
@@ -62,4 +64,10 @@ For containers started non-root or without the SETUID/SETGID capabilities needed
 
 SearXNG is a separate HTTPS client and image; updating the Odysseus image does not patch it. Inspect its effective CA path and update or rebuild that image independently. Also inspect persisted `/app/.local` packages, remote Cookbook environments, and externally configured MCP runtimes. Image replacement does not necessarily replace those environments.
 
-For behavioral validation, all twelve [published test chains](https://emudhra.we-have-a-problem.com/chains.zip), including the AAA cross-signs, must fail certificate verification through the effective stores. Confirm that an ordinary known-good HTTPS request still succeeds. In Chromium, confirm the managed policy is loaded at `chrome://policy` and test the incident endpoints through the actual browser path with certificate verification enabled. The PEM checker cannot inspect Chromium's policy or its active verifier. Package versions and a successful image build alone do not prove that a running deployment selects the corrected store.
+For behavioral validation, all fourteen [published test chains](https://emudhra.we-have-a-problem.com/chains.zip), including the AAA cross-signs and two chains without Certificate Transparency entries, must fail certificate verification through the effective stores. Confirm that an ordinary known-good HTTPS request still succeeds. In Chromium, confirm the managed policy is loaded at `chrome://policy` and test the incident endpoints through the actual browser path with certificate verification enabled. The PEM checker cannot inspect Chromium's policy or its active verifier. Package versions and a successful image build alone do not prove that a running deployment selects the corrected store.
+
+## Follow-up releases
+
+The maintained bootstrap selects certifi 2026.10.23 with its verified wheel hash; pip remains 26.2.2. The certifi update removes the C1/C3 roots in addition to the earlier incident changes. CI uses uv 0.13.5, which adds verifier-level rejection of the compromised intermediate keys and updated managed CPython builds. Existing virtual environments do not update automatically. See the [certifi change](https://github.com/certifi/python-certifi/commit/b60466c9f2882d9cf088943c650b0faac06c080c), [uv release](https://github.com/astral-sh/uv/releases/tag/0.13.5), and [Python security releases](https://blog.python.org/2026/10/python-3151-3149-31317/). The pinned bootstrap still replaces the selected versions exactly; preserving newer installed versions and private-mirror support remain separate maintenance.
+
+SearXNG remains an independent trust owner. Use the [derived-image validation and rollout instructions](searxng-trust-image.md); updating these application packages does not repair an existing sidecar. Host daemons, native browser policy, optional compiled clients and independently managed integrations require their own effective-client checks.