settled-reach/docs/DEVOPS.md

---
title: "DevOps Procedures"
description: "Build, test, lint, and CI procedures for the Settled Reach project — Makefile targets, worktree setup, server/client builds"
type: architecture
status: active
---

# DevOps Procedures

Operational procedures for building, testing, and running The Settled Reach.

## Repository Layout

```
client/               Godot 4 client (GDScript, scenes, assets)
server/               Rust/bevy_ecs simulation server
tooling/              Build tools, scripts, asset pipelines
tests/                Integration and end-to-end tests (cross-boundary)
governance/           Decision records — decisions/ questions/ rejected/ per domain (pql DQR tree)
.pql/                 pql planning store — git-tracked changelog/ + config.yaml (pql.db is rebuildable)
.config/              Configuration files (linters, formatters, CI)
.cache/               Local caches for testing/linting (gitignored)
docs/                 Design, architecture, briefings, workshops
db/                   Schema + seed data (asset connectors at tooling/db/)
```

Unit tests live inside their respective projects (`server/` uses `#[cfg(test)]` inline + `tests/` directory per D-030). The top-level `tests/` directory is for integration tests that cross the client-server boundary (IPC round-trip, serialization fixtures, divergence tests).

## Prerequisites

| Tool | Version | Purpose |
|------|---------|---------|
| Rust (via rustup) | stable | Server compilation, clippy, rustfmt (auto-installed by `make setup`) |
| Godot | 4.x | Client editor and runtime (auto-installed to `~/bin/` by `make setup`) |
| Python | 3.x | Tooling scripts, db connectors |
| Make | any | Task runner (see below) |
| curl | any | Downloading Godot |
| unzip | any | Extracting Godot |

## Makefile Targets

All development operations go through the top-level `Makefile`. Run `make` with no arguments for a summary.

### Setup

```bash
make setup                        # Install/verify all dev dependencies
GODOT_VERSION=4.4 make setup      # Pin a specific Godot version
```

Downloads and installs Godot to `~/bin/godot4`, installs Rust clippy + rustfmt, and verifies Python/curl/unzip. Skips the download if the correct version is already installed. The `GODOT_VERSION` variable defaults to `4.6` and can be overridden.

#### The `reach` CLI and PATH (T-1261, D-263)

```bash
make install-reach                # Put `reach` on PATH (run by `make setup`)
make reach-repoint                # Re-point `reach` at THIS checkout
```

`reach` is installed with `uv tool install --editable`, which puts the
executable in `~/.local/bin` rather than `.venv/bin`. **That distinction is the
whole point.** A `[project.scripts]` entrypoint alone lands in `.venv/bin`,
which is on PATH only while the venv is activated — and agents and git hooks
never activate it. Installing as a uv tool makes `reach` a bare name in every
context: interactive shell, git hook, agent `Bash` call.

**Always invoke it as the bare word `reach`.** Never `python -m tooling`, never
`.venv/bin/reach`, never an absolute path. Any of those breaks the
`Bash(reach *)` permission rule and prompts every time — the same failure `tea`
had, where the fix was a bare name on PATH — and reintroduces the interpreter
fork between a human shell and a hook.

`--editable` means the checkout *is* the source: edit `tooling/`, run `reach`,
no reinstall. `--python` is pinned to `PYTHON_VERSION` so the tool and `.venv`
share one interpreter; left to itself uv picks the lowest version satisfying
`requires-python`, which silently forks the two environments.

**Re-pointing.** uv records the source path at install time. An install made
from a worktree keeps resolving there after the worktree is deleted — `reach`
then still runs, but from a path that no longer exists or, worse, from a stale
copy, so edits in the main checkout appear to do nothing. There is no error
message for this. Run `make reach-repoint` from the checkout you want it to
follow.

Activating `.venv` is still needed for running the test scripts directly; it is
not needed for `reach`.

**Agent permissions.** `.claude/settings.json` carries `Bash(reach)` and
`Bash(reach *)` — two entries, because a rule ending in ` *` does not match the
bare word, which is why `pql`, `make` and `cargo test` each have a pair too.
One command with subcommands means one rule covers the whole tool surface; that
is the friction Q-124 was filed about, where ten hand-written
`Bash(tooling/…)` entries each covered a single script.

Note that permission rules prefix-match the **whole command string**, so an
env-prefixed call — `SR_REPO_ROOT=… reach check client-version` — does *not*
match and will prompt. That is accepted rather than worked around: an
environment override is a genuine departure from the normal invocation, and the
ordinary form is what needs to be frictionless. Tests that need overrides
should pass them through the subprocess environment rather than the command
string, as `tooling/test_check.py` does.

### Build

```bash
make build              # Build both client and server
make build-server       # cargo build in server/
make build-client       # Client builds are editor-managed (prints guidance)
```

### Run

```bash
make server             # cargo run in server/
make client             # Launch Godot with client/ project
```

The server must be running before the client connects (subprocess launch will be automated later per D-020).

### Test

```bash
make test               # Run all tests (test-server + test-client)
make test-server        # Rust tests via tests/run-rust (cargo nextest, JSON summary)
make test-client        # Godot tests via tests/run-godot (gdUnit4 headless, JSON summary)
make test-tooling       # Tooling gate: planet-gen determinism guard + import_economics --dry-run (T-1066)
```

The IPC test layers (D-030) have dedicated targets:

```bash
make test-ipc-fixtures      # Layer 1: serialization round-trip fixtures
make test-ipc-protocol      # Layer 2: mock LocalBridge protocol tests
make test-ipc-integration   # Layer 3: real subprocess round-trip (+ benchmark when ready)
make test-ipc-benchmark     # IPC latency benchmark (blocked: #555/#556 handshake)
```

Each `tests/run-*` script outputs a JSON summary to stdout and streams progress to stderr:
```json
{"suite":"rust","total":42,"passed":42,"failed":0,"duration_ms":1230}
```

All scripts accept `--filter <name>` to run a subset of tests. They are whitelistable for agent use (no TTY prompts, no interactive input).

Server tests use Rust's built-in test framework with `#[cfg(test)]` inline tests and `tests/` integration tests (D-030). Client tests use gdUnit4 (D-030).

### Cross-Encoder Fixtures

```bash
make fixtures           # Regenerate Rust->GDScript fixtures (server/tests/gen_fixtures.rs)
make fixtures-client    # Generate GDScript->Rust fixtures + verify Rust decoder (#475)
```

The bidirectional protocol is validated by two sets of committed fixtures:
- **Rust encodes, GDScript decodes:** `client/tests/fixtures/msgpack/` (generated by `make fixtures`)
- **GDScript encodes, Rust decodes:** `server/tests/fixtures/gdscript/` (generated by `make fixtures-client`)

Regenerate both after any protocol change. Commit the updated fixtures alongside the code change.

**Troubleshooting fixture failures:**

- **`make fixtures-client` fails with encode errors:** Check that `client/addons/messagepack/messagepack.gd` is up to date. The script exits non-zero on any encode failure.
- **`gdscript_generated_fixtures_deserialize` fails:** Fixtures in `server/tests/fixtures/gdscript/` are stale or corrupted. Re-run `make fixtures-client` and commit the updated files.
- **Fixture staleness in `make pre-pr`:** Protocol changed but fixtures were not regenerated. Run `make fixtures && make fixtures-client`, then commit both `client/tests/fixtures/` and `server/tests/fixtures/gdscript/`.

### Golden File Management

```bash
make golden-diff        # Show diff if golden file output has changed
make golden-update      # Regenerate golden file and stage for commit
```

The golden file (`server/tests/golden/proof_room_tick_10.json`) is a committed snapshot of ObserverSnapshot output after a deterministic 10-tick replay. It catches unintentional changes to simulation output.

**Workflow after intentional simulation changes:**

1. Run `make golden-diff` to see what changed
2. Review the diff — confirm changes are expected
3. Run `make golden-update` to regenerate and stage the new golden file
4. Commit the updated golden file alongside your simulation change

`golden-diff` exits 1 if the golden file has changed (useful in scripts). `golden-update` regenerates the file and runs `git add` but does not commit — the developer reviews and commits manually.

### Lint

```bash
make lint               # Run all linters
make lint-server        # clippy (deny warnings) + rustfmt --check
make lint-client        # gdlint/gdformat
```

### CI (Local)

Run the full CI pipeline locally before pushing:

```bash
make ci                 # Both pipelines
make ci-server          # lint-server → build-server → test-server
make ci-client          # lint-client → build-client → test-client
```

CI targets chain lint → build → test sequentially. A failure in any stage stops the pipeline.

### Pre-PR Checks

Before pushing a PR, run:

```bash
make pre-pr
```

This runs all checks in order: lint → build → test → content validation → fixture staleness. Total runtime ~2.5 minutes (incremental build), under 3 minutes clean.

For branch-specific checks:

```bash
make pre-pr-server      # Server changes: lint, build, test, fixture staleness
make pre-pr-client      # Client changes: lint, build, test
make pre-pr-content     # Content changes: schema + cross-reference validation
```

If `pre-pr-fixtures` fails, your protocol changes require fixture regeneration:

```bash
make fixtures
git add client/tests/fixtures/
git commit -m "chore(fixtures): regenerate for protocol v8"
```

The fixture staleness check is a **blocker** (exit 1) — stale fixtures cause false positive client tests.

### Clean

```bash
make clean              # Remove build artifacts and .cache/ contents
```

### Content Validation

```bash
reach validate content    # Validate content YAML against JSON schemas
make check-fact-ids       # Check fact_id references against knowledge catalogs
```

### Gauntlet Checklists

```bash
make checklist-validate   # Validate checklist YAML against schema (standalone)
make checklist-generate   # Validate + print per-room condition summary
```

Checklists live at `content/gauntlet/rooms/{room_id}/checklist.yaml` (per-room) and `content/gauntlet/cross_room_checks.yaml` (cross-room). Each condition is evaluable from an `ObserverSnapshot`.

7 condition types: `player_near`, `player_facing`, `entity_present`, `entity_absent`, `expected_monologue`, `expected_dialogue`, `expected_interaction_verb`.

Schema: `content/_schema/checklist.schema.json`. The checklist format feeds into #503 (client auto-checklist progress tracking).

`check-fact-ids` operates in two modes:
- **Advisory** — when knowledge catalogs (`content/global/knowledge/*.yaml`) have no fact definitions yet: lists referenced fact_ids and exits cleanly.
- **Enforcing** — when catalogs are populated: fails on any `fact_id` reference that doesn't match a canonical definition.

## Asset Pipeline — Generator-Driven DB (#855, #856, #857)

`server/data/systems.db` is a **read-only canonical snapshot** produced by a single
generator.  It is committed to the repo so the client can ship it, but it is never
the source of truth.  Direct edits are forbidden — they are silently overwritten by
the next regeneration.

### Generator

`import_economics` (`tooling/economy-db/import_economics.py`) is the sole
generator.  As its first step it runs the Rust `generate_brands` binary
(via `core.process.cargo_binary`) to refresh `generated_brands.toml`, then imports
economics data and the atlas index (names-only city pool; geometry tables stay
empty for the Phase 4 server cascade).  The former atlas geometry generator
(`generate_atlas`) was retired in #951 (D-223).

Run it with:

```bash
make regen-db
```

### Meta table stamp

After every successful non-dry-run, the generator writes a row to the `meta` table in
`systems.db` recording the SHA-1 of its source files and the schema file.  The source
registry is the `GENERATOR_SOURCES` dict in `tooling/generator_sources.py`.

```bash
make check-systems-db    # Verify the stamp is fresh (exit 1 = stale)
```

### Making a DB change

1. Edit source files (TOML, JSON, `markers.json`).
2. `make regen-db`
3. `git add server/data/systems.db`
4. Commit with `chore(db): regen systems.db — <reason>`

For schema changes, also update `server/data/systems-schema.sql` and add migration DDL
to `MIGRATION_SQL` in `import_economics.py`.

See `.claude/rules/asset-pipeline.md` for the full rule set.

## Client version mirror

`project.yaml`'s `version:` is the source of truth, but the client cannot read that
file at runtime — an exported build has no repo root above `res://`. The value is
therefore mirrored into `client/project.godot` as `application/config/version`, which
Godot bakes into the exported PCK, and read through `client/scripts/build_version.gd`.

This is not cosmetic. The Atlas disk cache (D-255) keys its **only** invalidation
signal on that version, so a version the client cannot read means a cache that can
never be invalidated — see T-1241, and T-1239 for what a stale canvas cache actually
costs.

```bash
reach check client-version   # exit 1 = the two files disagree
```

The pre-push hook runs this unconditionally (drift persists on `main` once
introduced, so gating it on "were those files touched in this push" would let an
existing drift ride along). **Bump both files together.**

## Canvas-generation version pairing

Changing *how* a canvas is generated is only half a change. The other half is bumping
`project.yaml`'s version — otherwise every warm Atlas cache keeps serving canvases
built by code that no longer exists. That pairing was never enforced and broke five
times (0.4.2 lake_margin_q, 0.4.3 coast_warp_px, 0.4.4 the extent inversion, 0.4.5 the
Global sentinel, 0.4.6 one-course-per-river), each bumped only after the fact. The last
one took eight days to find (T-1239) because the failure is invisible to its author: it
reproduces only where a warm cache exists.

```bash
reach check canvas-version   # exit 1 = generation changed, version didn't
```

The path registry is `tooling/canvas_sources.py` — globbed, not hand-listed, so a
module added in a future split is covered the moment it exists. It **deliberately
over-includes**: a false positive costs one version bump and one round of cache misses,
a false negative costs another week of a wrong map.

The registry includes itself, which closes the narrowing hole — removing a path *and*
changing that path in one push still trips the gate, because the registry file is in the
set. The cost is that editing the registry requires a version bump.

There is **no override flag**, on purpose. It would be reached for exactly when someone
is certain their change is harmless, which is the state of mind that produced all five
regressions. If you are sure, bump anyway — it costs one cache miss.

Units for the gate (including "a comment edit that quotes version numbers is not a
bump") run in `make test-tooling`.

## Pre-commit and Pre-push Hooks

Git hooks are stored in `.config/hooks/` (version-controlled). Activate them with:

```bash
make setup          # Includes hook installation
make install-hooks  # Just hooks (also makes them executable)
```

Or manually:

```bash
git config core.hooksPath .config/hooks
```

### Pre-commit checks (`.config/hooks/pre-commit`)

| Check | Script | Behavior |
|-------|--------|----------|
| fact_id validation | `reach check fact-ids` | Warns if catalogs are stubs; fails on unknown fact_ids when populated |
| Decision records | `pql decisions validate` | Blocks on malformed decision records (warns if pql not on PATH) |
| Planning changelog | `pql plan export --stage` | Flushes ticket mutations to `.pql/changelog/` and stages them into the commit (warn-only on failure) |
| cargo audit | `cargo audit` | Only when `Cargo.toml`/`Cargo.lock` is staged; blocks on advisories (warns if cargo-audit not installed) |

### Pre-push checks (`.config/hooks/pre-push`)

There is **no CI** — the pre-push gate is the only automatic verification, so it is
deliberately comprehensive.  Checks are scoped to what actually changed vs the remote
(`origin/<branch>`, falling back to `origin/main` for first pushes): `client/` changes
gate the GDScript checks, `server/` the Rust checks, `tooling/` + `pyproject.toml` the
Python checks.

| Check | Runs when | Behavior |
|-------|-----------|----------|
| GDScript parse (headless Godot) | `client/` changed | Blocks on any SCRIPT ERROR (skipped if no `client/.godot/` import) |
| gdlint | `client/` changed | **Advisory** until the codebase is clean |
| gdformat --check | `client/` changed | **Advisory** until the codebase is clean |
| `cargo fmt --check` | `server/` changed | Blocks |
| `cargo clippy --all-targets -- -D warnings` | `server/` changed | Blocks (skipped if no `server/target/` — cold worktree) |
| `cargo test` | `server/` changed | Blocks — the only automatic correctness gate (skipped if no `server/target/`) |
| `cargo deny check` | `server/` changed | Blocks (only if cargo-deny installed and `server/deny.toml` exists) |
| `ruff check tooling/` | `tooling/` changed | Blocks |
| `make test-tooling` | `tooling/` changed | Blocks — sim determinism guard + economics dry-run (T-1066) |
| JSON syntax (`python3 -m json.tool`) | any changed `*.json` | Blocks on syntax errors |
| systems.db stamp | `server/data/systems.db` in push | Blocks on stale stamp (#857); missing meta table warns only |
| Clerk review (D-221) | **disabled** (#965) | Force-run with `SR_RUN_CLERK=1` |

### Post-merge / post-checkout / post-rewrite

These hooks replay the pql planning changelog (`pql plan import` / `rebuild`) and
re-sync decisions from `governance/*.md` so the planning DB stays in step after
merges, checkouts, and history rewrites.

The `core.hooksPath` setting uses a relative path (`.config/hooks`) that resolves per worktree, so it works correctly across all worktrees in the repository.

To bypass hooks in an emergency:

```bash
git commit --no-verify -m "fix: emergency hotfix"
git push --no-verify
```

## Configuration Files

The `.config/` directory currently holds exactly one thing: the version-controlled
git hooks in `.config/hooks/` (pre-commit, pre-push, post-merge, post-checkout,
post-rewrite), activated via `git config core.hooksPath .config/hooks`
(`make install-hooks`).

Linter and formatter configuration lives with the code it governs, not in `.config/`:
ruff is configured in the root `pyproject.toml` (`[tool.ruff]`), Rust uses cargo
defaults plus `server/deny.toml`, and the client uses gdlint/gdformat defaults.
Project-specific config (e.g. `server/Cargo.toml`, `client/project.godot`) stays in
those directories. `.config/` is reserved for future cross-cutting configuration
that has no better home.

## Cache Directory

`.cache/` is gitignored and used for:

- Test result caches
- Linter caches
- Build artifact caches (if configured)
- Coverage reports

Agents and CI jobs can write freely to `.cache/` without polluting the working tree. `make clean` clears it.

## Testing Architecture (D-030)

Three-layer testing strategy:

1. **Unit tests** — Inside `server/` (Rust `#[cfg(test)]`) and `client/` (gdUnit4). Test individual systems in isolation.
2. **Integration tests** — Inside `server/tests/` (Rust) and `tests/` (cross-boundary). Test system interactions, IPC serialization round-trips.
3. **Fixture-based tests** — IPC serialization fixture files in `tests/` for protocol regression testing. Known-good MessagePack payloads verified against both sides.

Key components:
- **CauseChain** (production ECS component) — Tracks causal attribution for testable observation sequences (D-030).
- **Deterministic replay** — Server simulation is deterministic given the same seed + input sequence. Replay logs enable regression testing (#201, critical).

### Real-rendering test exception: `tests/run-visual` (`tests/visual_capture.gd`)

`tests/run-godot` hardcodes `--headless`, whose dummy driver produces no usable GPU
texture output (`SubViewport.get_texture().get_image()` returns unusable data). Any
test that needs real composited pixels lives in `tests/run-visual`
(`client/tests/visual_capture.gd`) instead, which boots Godot with a real
`--rendering-driver opengl3` and asserts on an actual `get_viewport().get_texture().
get_image()` capture — this is the project's one real-driver exception, and every
"did anything draw at all" real-pixel check lives here now, not in a standalone gdUnit
file with its own headless-skip guard.

`client/tests/test_atlas_window_overlay_draw_smoke.gd` (T-1153 live round 4) used to be
a second, parallel real-driver path — a standalone gdUnit suite that rendered
`AtlasWindowOverlay` into its own ad hoc `SubViewport`, self-detecting the dummy driver
(`DisplayServer.get_name() == "headless"`) and skipping under `tests/run-godot`. It
retired with the rest of the `AtlasWindowViewer`/continuous-zoom cluster (T-1182, D-255)
and its 2 real-pixel scenarios (single-window draw, tile-mosaic draw) folded into this
harness (T-1157): the stepped ladder has no separate "single window" vs "tile mosaic"
draw path anymore (D-255(a): "one viewer one path"), so the fold's two natural
equivalents are a fixed-rung canvas capture (`atlas_GJ380c_District` — LINEAR-filtered,
`StepCanvasTerrainLayer._filter_for_rung()`'s non-orbital branch) and the Global-rung
capture (`atlas_GJ380c_Global` — NEAREST-filtered, the orbital branch, a differently-shaped
canvas/footprint). Both are ordinary `tests/visual.json` golden scenarios driven through
production navigation (`visual_scenarios.gd`'s `_setup_atlas_golden_shot`) against a real
`--test-mode` server (`SR_LIVE=1`) — real derived terrain, real composited pixels, no
synthetic stub canvas — closing the "did anything draw at all" gap without a second,
parallel real-driver mechanism.

```bash
# Any single atlas golden, real driver, real server:
make screenshot SCENARIO=atlas_GJ380c_District
make screenshot SCENARIO=atlas_GJ380c_Global

# The full golden suite (xvfb-wrapped, all scenarios incl. atlas_*):
make test-visual
```

## Planning store (pql)

Tickets and decisions live in **pql**, not in the old SQLite wrapper scripts. Decisions
are markdown-sourced under `governance/{decisions,questions,rejected}/`; tickets live in
`.pql/pql.db`, rebuilt from the git-tracked `.pql/changelog/`. Never use the `sqlite3`
CLI — it crashes in Claude Code (std::bad_alloc).

```bash
pql ticket list --status in_progress         # query tickets (T-NNN ids)
pql ticket status T-440 done                  # mutate — flushed to the changelog
pql decisions list --type confirmed           # query decisions
pql decisions show D-010 --with-tickets
make decisions-sync                           # parse governance/*.md into pql.db
make decisions-validate                       # malformed-record gate
```

Markdown records + the changelog are the source of truth; `pql.db` is a rebuildable
index. The pre-commit hook runs `pql decisions validate` and exports ticket mutations to
the changelog; post-merge/checkout/rewrite hooks replay it. Full reference:
`.claude/rules/ticket-cli.md`.

## Commit Conventions

See the `/git-commit` skill (`.claude/skills/git-commit/`) for full details. Summary:

- Conventional commits: `type(scope): summary`
- Types: `feat`, `fix`, `refactor`, `chore`, `docs`, `data`, `loc`
- Scopes match project subsystems: `client`, `server`, `engine`, `simulation`, `ui`, `audio`, `meta`, etc.
- Imperative mood, lowercase, no period, max 72 chars
- CHANGELOG.md updated after each commit group