settled-reach/docs/DEVOPS.md

---
title: "DevOps Procedures"
description: "Build, test, lint, and CI procedures for the Settled Reach project — Makefile targets, worktree setup, server/client builds"
type: architecture
status: active
---

# DevOps Procedures

Operational procedures for building, testing, and running The Settled Reach.

## Repository Layout

```
client/               Godot 4 client (GDScript, scenes, assets)
server/               Rust/bevy_ecs simulation server
tooling/              Build tools, scripts, asset pipelines
tests/                Integration and end-to-end tests (cross-boundary)
governance/           Decision records — decisions/ questions/ rejected/ per domain (pql DQR tree)
.pql/                 pql planning store — git-tracked changelog/ + config.yaml (pql.db is rebuildable)
.config/              Configuration files (linters, formatters, CI)
.cache/               Local caches for testing/linting (gitignored)
docs/                 Design, architecture, briefings, workshops
db/                   Schema + seed data (asset connectors at tooling/db/)
```

Unit tests live inside their respective projects (`server/` uses `#[cfg(test)]` inline + `tests/` directory per D-030). The top-level `tests/` directory is for integration tests that cross the client-server boundary (IPC round-trip, serialization fixtures, divergence tests).

## Prerequisites

| Tool | Version | Purpose |
|------|---------|---------|
| Rust (via rustup) | stable | Server compilation, clippy, rustfmt (auto-installed by `make setup`) |
| Godot | 4.x | Client editor and runtime (auto-installed to `~/bin/` by `make setup`) |
| Python | 3.x | Tooling scripts, db connectors |
| Make | any | Task runner (see below) |
| curl | any | Downloading Godot |
| unzip | any | Extracting Godot |

## Makefile Targets

All development operations go through the top-level `Makefile`. Run `make` with no arguments for a summary.

### Setup

```bash
make setup                        # Install/verify all dev dependencies
GODOT_VERSION=4.4 make setup      # Pin a specific Godot version
```

Downloads and installs Godot to `~/bin/godot4`, installs Rust clippy + rustfmt, and verifies Python/curl/unzip. Skips the download if the correct version is already installed. The `GODOT_VERSION` variable defaults to `4.6` and can be overridden.

### Build

```bash
make build              # Build both client and server
make build-server       # cargo build in server/
make build-client       # Client builds are editor-managed (prints guidance)
```

### Run

```bash
make server             # cargo run in server/
make client             # Launch Godot with client/ project
```

The server must be running before the client connects (subprocess launch will be automated later per D-020).

### Test

```bash
make test               # Run all tests (test-server + test-client)
make test-server        # Rust tests via tests/run-rust (cargo nextest, JSON summary)
make test-client        # Godot tests via tests/run-godot (gdUnit4 headless, JSON summary)
make test-tooling       # Tooling gate: planet-gen determinism guard + import_economics --dry-run (T-1066)
```

The IPC test layers (D-030) have dedicated targets:

```bash
make test-ipc-fixtures      # Layer 1: serialization round-trip fixtures
make test-ipc-protocol      # Layer 2: mock LocalBridge protocol tests
make test-ipc-integration   # Layer 3: real subprocess round-trip (+ benchmark when ready)
make test-ipc-benchmark     # IPC latency benchmark (blocked: #555/#556 handshake)
```

Each `tests/run-*` script outputs a JSON summary to stdout and streams progress to stderr:
```json
{"suite":"rust","total":42,"passed":42,"failed":0,"duration_ms":1230}
```

All scripts accept `--filter <name>` to run a subset of tests. They are whitelistable for agent use (no TTY prompts, no interactive input).

Server tests use Rust's built-in test framework with `#[cfg(test)]` inline tests and `tests/` integration tests (D-030). Client tests use gdUnit4 (D-030).

### Cross-Encoder Fixtures

```bash
make fixtures           # Regenerate Rust->GDScript fixtures (server/tests/gen_fixtures.rs)
make fixtures-client    # Generate GDScript->Rust fixtures + verify Rust decoder (#475)
```

The bidirectional protocol is validated by two sets of committed fixtures:
- **Rust encodes, GDScript decodes:** `client/tests/fixtures/msgpack/` (generated by `make fixtures`)
- **GDScript encodes, Rust decodes:** `server/tests/fixtures/gdscript/` (generated by `make fixtures-client`)

Regenerate both after any protocol change. Commit the updated fixtures alongside the code change.

**Troubleshooting fixture failures:**

- **`make fixtures-client` fails with encode errors:** Check that `client/addons/messagepack/messagepack.gd` is up to date. The script exits non-zero on any encode failure.
- **`gdscript_generated_fixtures_deserialize` fails:** Fixtures in `server/tests/fixtures/gdscript/` are stale or corrupted. Re-run `make fixtures-client` and commit the updated files.
- **Fixture staleness in `make pre-pr`:** Protocol changed but fixtures were not regenerated. Run `make fixtures && make fixtures-client`, then commit both `client/tests/fixtures/` and `server/tests/fixtures/gdscript/`.

### Golden File Management

```bash
make golden-diff        # Show diff if golden file output has changed
make golden-update      # Regenerate golden file and stage for commit
```

The golden file (`server/tests/golden/proof_room_tick_10.json`) is a committed snapshot of ObserverSnapshot output after a deterministic 10-tick replay. It catches unintentional changes to simulation output.

**Workflow after intentional simulation changes:**

1. Run `make golden-diff` to see what changed
2. Review the diff — confirm changes are expected
3. Run `make golden-update` to regenerate and stage the new golden file
4. Commit the updated golden file alongside your simulation change

`golden-diff` exits 1 if the golden file has changed (useful in scripts). `golden-update` regenerates the file and runs `git add` but does not commit — the developer reviews and commits manually.

### Lint

```bash
make lint               # Run all linters
make lint-server        # clippy (deny warnings) + rustfmt --check
make lint-client        # gdlint/gdformat
```

### CI (Local)

Run the full CI pipeline locally before pushing:

```bash
make ci                 # Both pipelines
make ci-server          # lint-server → build-server → test-server
make ci-client          # lint-client → build-client → test-client
```

CI targets chain lint → build → test sequentially. A failure in any stage stops the pipeline.

### Pre-PR Checks

Before pushing a PR, run:

```bash
make pre-pr
```

This runs all checks in order: lint → build → test → content validation → fixture staleness. Total runtime ~2.5 minutes (incremental build), under 3 minutes clean.

For branch-specific checks:

```bash
make pre-pr-server      # Server changes: lint, build, test, fixture staleness
make pre-pr-client      # Client changes: lint, build, test
make pre-pr-content     # Content changes: schema + cross-reference validation
```

If `pre-pr-fixtures` fails, your protocol changes require fixture regeneration:

```bash
make fixtures
git add client/tests/fixtures/
git commit -m "chore(fixtures): regenerate for protocol v8"
```

The fixture staleness check is a **blocker** (exit 1) — stale fixtures cause false positive client tests.

### Clean

```bash
make clean              # Remove build artifacts and .cache/ contents
```

### Content Validation

```bash
make validate-content     # Validate content YAML against JSON schemas
make check-fact-ids       # Check fact_id references against knowledge catalogs
```

### Gauntlet Checklists

```bash
make checklist-validate   # Validate checklist YAML against schema (standalone)
make checklist-generate   # Validate + print per-room condition summary
```

Checklists live at `content/gauntlet/rooms/{room_id}/checklist.yaml` (per-room) and `content/gauntlet/cross_room_checks.yaml` (cross-room). Each condition is evaluable from an `ObserverSnapshot`.

7 condition types: `player_near`, `player_facing`, `entity_present`, `entity_absent`, `expected_monologue`, `expected_dialogue`, `expected_interaction_verb`.

Schema: `content/_schema/checklist.schema.json`. The checklist format feeds into #503 (client auto-checklist progress tracking).

`check-fact-ids` operates in two modes:
- **Advisory** — when knowledge catalogs (`content/global/knowledge/*.yaml`) have no fact definitions yet: lists referenced fact_ids and exits cleanly.
- **Enforcing** — when catalogs are populated: fails on any `fact_id` reference that doesn't match a canonical definition.

## Asset Pipeline — Generator-Driven DB (#855, #856, #857)

`server/data/systems.db` is a **read-only canonical snapshot** produced by a single
generator.  It is committed to the repo so the client can ship it, but it is never
the source of truth.  Direct edits are forbidden — they are silently overwritten by
the next regeneration.

### Generator

`import_economics` (`tooling/economy-db/import_economics.py`) is the sole
generator.  As its first step it shells out to the Rust `generate_brands` binary
(via `tooling/generate-brands`) to refresh `generated_brands.toml`, then imports
economics data and the atlas index (names-only city pool; geometry tables stay
empty for the Phase 4 server cascade).  The former atlas geometry generator
(`generate_atlas`) was retired in #951 (D-223).

Run it with:

```bash
make regen-db
```

### Meta table stamp

After every successful non-dry-run, the generator writes a row to the `meta` table in
`systems.db` recording the SHA-1 of its source files and the schema file.  The source
registry is the `GENERATOR_SOURCES` dict in `tooling/check-systems-db-stamp`.

```bash
make check-systems-db    # Verify the stamp is fresh (exit 1 = stale)
```

### Making a DB change

1. Edit source files (TOML, JSON, `markers.json`).
2. `make regen-db`
3. `git add server/data/systems.db`
4. Commit with `chore(db): regen systems.db — <reason>`

For schema changes, also update `server/data/systems-schema.sql` and add migration DDL
to `MIGRATION_SQL` in `import_economics.py`.

See `.claude/rules/asset-pipeline.md` for the full rule set.

## Client version mirror

`project.yaml`'s `version:` is the source of truth, but the client cannot read that
file at runtime — an exported build has no repo root above `res://`. The value is
therefore mirrored into `client/project.godot` as `application/config/version`, which
Godot bakes into the exported PCK, and read through `client/scripts/build_version.gd`.

This is not cosmetic. The Atlas disk cache (D-255) keys its **only** invalidation
signal on that version, so a version the client cannot read means a cache that can
never be invalidated — see T-1241, and T-1239 for what a stale canvas cache actually
costs.

```bash
make check-client-version    # exit 1 = the two files disagree
```

The pre-push hook runs this unconditionally (drift persists on `main` once
introduced, so gating it on "were those files touched in this push" would let an
existing drift ride along). **Bump both files together.**

## Canvas-generation version pairing

Changing *how* a canvas is generated is only half a change. The other half is bumping
`project.yaml`'s version — otherwise every warm Atlas cache keeps serving canvases
built by code that no longer exists. That pairing was never enforced and broke five
times (0.4.2 lake_margin_q, 0.4.3 coast_warp_px, 0.4.4 the extent inversion, 0.4.5 the
Global sentinel, 0.4.6 one-course-per-river), each bumped only after the fact. The last
one took eight days to find (T-1239) because the failure is invisible to its author: it
reproduces only where a warm cache exists.

```bash
make check-canvas-version    # exit 1 = generation changed, version didn't
```

The path registry is `tooling/canvas_sources.py` — globbed, not hand-listed, so a
module added in a future split is covered the moment it exists. It **deliberately
over-includes**: a false positive costs one version bump and one round of cache misses,
a false negative costs another week of a wrong map.

The registry includes itself, which closes the narrowing hole — removing a path *and*
changing that path in one push still trips the gate, because the registry file is in the
set. The cost is that editing the registry requires a version bump.

There is **no override flag**, on purpose. It would be reached for exactly when someone
is certain their change is harmless, which is the state of mind that produced all five
regressions. If you are sure, bump anyway — it costs one cache miss.

Units for the gate (including "a comment edit that quotes version numbers is not a
bump") run in `make test-tooling`.

## Pre-commit and Pre-push Hooks

Git hooks are stored in `.config/hooks/` (version-controlled). Activate them with:

```bash
make setup          # Includes hook installation
make install-hooks  # Just hooks (also makes them executable)
```

Or manually:

```bash
git config core.hooksPath .config/hooks
```

### Pre-commit checks (`.config/hooks/pre-commit`)

| Check | Script | Behavior |
|-------|--------|----------|
| fact_id validation | `tooling/check-fact-ids` | Warns if catalogs are stubs; fails on unknown fact_ids when populated |
| Decision records | `pql decisions validate` | Blocks on malformed decision records (warns if pql not on PATH) |
| Planning changelog | `pql plan export --stage` | Flushes ticket mutations to `.pql/changelog/` and stages them into the commit (warn-only on failure) |
| cargo audit | `cargo audit` | Only when `Cargo.toml`/`Cargo.lock` is staged; blocks on advisories (warns if cargo-audit not installed) |

### Pre-push checks (`.config/hooks/pre-push`)

There is **no CI** — the pre-push gate is the only automatic verification, so it is
deliberately comprehensive.  Checks are scoped to what actually changed vs the remote
(`origin/<branch>`, falling back to `origin/main` for first pushes): `client/` changes
gate the GDScript checks, `server/` the Rust checks, `tooling/` + `pyproject.toml` the
Python checks.

| Check | Runs when | Behavior |
|-------|-----------|----------|
| GDScript parse (headless Godot) | `client/` changed | Blocks on any SCRIPT ERROR (skipped if no `client/.godot/` import) |
| gdlint | `client/` changed | **Advisory** until the codebase is clean |
| gdformat --check | `client/` changed | **Advisory** until the codebase is clean |
| `cargo fmt --check` | `server/` changed | Blocks |
| `cargo clippy --all-targets -- -D warnings` | `server/` changed | Blocks (skipped if no `server/target/` — cold worktree) |
| `cargo test` | `server/` changed | Blocks — the only automatic correctness gate (skipped if no `server/target/`) |
| `cargo deny check` | `server/` changed | Blocks (only if cargo-deny installed and `server/deny.toml` exists) |
| `ruff check tooling/` | `tooling/` changed | Blocks |
| `make test-tooling` | `tooling/` changed | Blocks — sim determinism guard + economics dry-run (T-1066) |
| JSON syntax (`python3 -m json.tool`) | any changed `*.json` | Blocks on syntax errors |
| systems.db stamp | `server/data/systems.db` in push | Blocks on stale stamp (#857); missing meta table warns only |
| Clerk review (D-221) | **disabled** (#965) | Force-run with `SR_RUN_CLERK=1` |

### Post-merge / post-checkout / post-rewrite

These hooks replay the pql planning changelog (`pql plan import` / `rebuild`) and
re-sync decisions from `governance/*.md` so the planning DB stays in step after
merges, checkouts, and history rewrites.

The `core.hooksPath` setting uses a relative path (`.config/hooks`) that resolves per worktree, so it works correctly across all worktrees in the repository.

To bypass hooks in an emergency:

```bash
git commit --no-verify -m "fix: emergency hotfix"
git push --no-verify
```

## Configuration Files

The `.config/` directory currently holds exactly one thing: the version-controlled
git hooks in `.config/hooks/` (pre-commit, pre-push, post-merge, post-checkout,
post-rewrite), activated via `git config core.hooksPath .config/hooks`
(`make install-hooks`).

Linter and formatter configuration lives with the code it governs, not in `.config/`:
ruff is configured in the root `pyproject.toml` (`[tool.ruff]`), Rust uses cargo
defaults plus `server/deny.toml`, and the client uses gdlint/gdformat defaults.
Project-specific config (e.g. `server/Cargo.toml`, `client/project.godot`) stays in
those directories. `.config/` is reserved for future cross-cutting configuration
that has no better home.

## Cache Directory

`.cache/` is gitignored and used for:

- Test result caches
- Linter caches
- Build artifact caches (if configured)
- Coverage reports

Agents and CI jobs can write freely to `.cache/` without polluting the working tree. `make clean` clears it.

## Testing Architecture (D-030)

Three-layer testing strategy:

1. **Unit tests** — Inside `server/` (Rust `#[cfg(test)]`) and `client/` (gdUnit4). Test individual systems in isolation.
2. **Integration tests** — Inside `server/tests/` (Rust) and `tests/` (cross-boundary). Test system interactions, IPC serialization round-trips.
3. **Fixture-based tests** — IPC serialization fixture files in `tests/` for protocol regression testing. Known-good MessagePack payloads verified against both sides.

Key components:
- **CauseChain** (production ECS component) — Tracks causal attribution for testable observation sequences (D-030).
- **Deterministic replay** — Server simulation is deterministic given the same seed + input sequence. Replay logs enable regression testing (#201, critical).

### Real-rendering test exception: `tests/run-visual` (`tests/visual_capture.gd`)

`tests/run-godot` hardcodes `--headless`, whose dummy driver produces no usable GPU
texture output (`SubViewport.get_texture().get_image()` returns unusable data). Any
test that needs real composited pixels lives in `tests/run-visual`
(`client/tests/visual_capture.gd`) instead, which boots Godot with a real
`--rendering-driver opengl3` and asserts on an actual `get_viewport().get_texture().
get_image()` capture — this is the project's one real-driver exception, and every
"did anything draw at all" real-pixel check lives here now, not in a standalone gdUnit
file with its own headless-skip guard.

`client/tests/test_atlas_window_overlay_draw_smoke.gd` (T-1153 live round 4) used to be
a second, parallel real-driver path — a standalone gdUnit suite that rendered
`AtlasWindowOverlay` into its own ad hoc `SubViewport`, self-detecting the dummy driver
(`DisplayServer.get_name() == "headless"`) and skipping under `tests/run-godot`. It
retired with the rest of the `AtlasWindowViewer`/continuous-zoom cluster (T-1182, D-255)
and its 2 real-pixel scenarios (single-window draw, tile-mosaic draw) folded into this
harness (T-1157): the stepped ladder has no separate "single window" vs "tile mosaic"
draw path anymore (D-255(a): "one viewer one path"), so the fold's two natural
equivalents are a fixed-rung canvas capture (`atlas_GJ380c_District` — LINEAR-filtered,
`StepCanvasTerrainLayer._filter_for_rung()`'s non-orbital branch) and the Global-rung
capture (`atlas_GJ380c_Global` — NEAREST-filtered, the orbital branch, a differently-shaped
canvas/footprint). Both are ordinary `tests/visual.json` golden scenarios driven through
production navigation (`visual_scenarios.gd`'s `_setup_atlas_golden_shot`) against a real
`--test-mode` server (`SR_LIVE=1`) — real derived terrain, real composited pixels, no
synthetic stub canvas — closing the "did anything draw at all" gap without a second,
parallel real-driver mechanism.

```bash
# Any single atlas golden, real driver, real server:
make screenshot SCENARIO=atlas_GJ380c_District
make screenshot SCENARIO=atlas_GJ380c_Global

# The full golden suite (xvfb-wrapped, all scenarios incl. atlas_*):
make test-visual
```

## Planning store (pql)

Tickets and decisions live in **pql**, not in the old SQLite wrapper scripts. Decisions
are markdown-sourced under `governance/{decisions,questions,rejected}/`; tickets live in
`.pql/pql.db`, rebuilt from the git-tracked `.pql/changelog/`. Never use the `sqlite3`
CLI — it crashes in Claude Code (std::bad_alloc).

```bash
pql ticket list --status in_progress         # query tickets (T-NNN ids)
pql ticket status T-440 done                  # mutate — flushed to the changelog
pql decisions list --type confirmed           # query decisions
pql decisions show D-010 --with-tickets
make decisions-sync                           # parse governance/*.md into pql.db
make decisions-validate                       # malformed-record gate
```

Markdown records + the changelog are the source of truth; `pql.db` is a rebuildable
index. The pre-commit hook runs `pql decisions validate` and exports ticket mutations to
the changelog; post-merge/checkout/rewrite hooks replay it. Full reference:
`.claude/rules/ticket-cli.md`.

## Commit Conventions

See the `/git-commit` skill (`.claude/skills/git-commit/`) for full details. Summary:

- Conventional commits: `type(scope): summary`
- Types: `feat`, `fix`, `refactor`, `chore`, `docs`, `data`, `loc`
- Scopes match project subsystems: `client`, `server`, `engine`, `simulation`, `ui`, `audio`, `meta`, etc.
- Imperative mood, lowercase, no period, max 72 chars
- CHANGELOG.md updated after each commit group