settled-reach/docs/DEVOPS.md

title, description, type, status

title	description	type	status
DevOps Procedures	Build, test, lint, and CI procedures for the Settled Reach project — Makefile targets, worktree setup, server/client builds	architecture	active

DevOps Procedures

Operational procedures for building, testing, and running The Settled Reach.

Repository Layout

client/               Godot 4 client (GDScript, scenes, assets)
server/               Rust/bevy_ecs simulation server
tooling/              Build tools, scripts, asset pipelines
tests/                Integration and end-to-end tests (cross-boundary)
governance/           Decision records — decisions/ questions/ rejected/ per domain (pql DQR tree)
.pql/                 pql planning store — git-tracked changelog/ + config.yaml (pql.db is rebuildable)
.config/              Configuration files (linters, formatters, CI)
.cache/               Local caches for testing/linting (gitignored)
docs/                 Design, architecture, briefings, workshops
db/                   Schema + seed data (asset connectors at tooling/db/)

Unit tests live inside their respective projects (server/ uses #[cfg(test)] inline + tests/ directory per D-030). The top-level tests/ directory is for integration tests that cross the client-server boundary (IPC round-trip, serialization fixtures, divergence tests).

Prerequisites

Tool	Version	Purpose
Rust (via rustup)	stable	Server compilation, clippy, rustfmt (auto-installed by make setup)
Godot	4.x	Client editor and runtime (auto-installed to ~/bin/ by make setup)
Python	3.x	Tooling scripts, db connectors
Make	any	Task runner (see below)
curl	any	Downloading Godot
unzip	any	Extracting Godot

Makefile Targets

All development operations go through the top-level Makefile. Run make with no arguments for a summary.

Setup

make setup                        # Install/verify all dev dependencies
GODOT_VERSION=4.4 make setup      # Pin a specific Godot version

Downloads and installs Godot to ~/bin/godot4, installs Rust clippy + rustfmt, and verifies Python/curl/unzip. Skips the download if the correct version is already installed. The GODOT_VERSION variable defaults to 4.6 and can be overridden.

The reach CLI and PATH (T-1261, D-263)

make install-reach                # Put `reach` on PATH (run by `make setup`)
make reach-repoint                # Re-point `reach` at THIS checkout

reach is installed with uv tool install --editable, which puts the executable in ~/.local/bin rather than .venv/bin. That distinction is the whole point. A [project.scripts] entrypoint alone lands in .venv/bin, which is on PATH only while the venv is activated — and agents and git hooks never activate it. Installing as a uv tool makes reach a bare name in every context: interactive shell, git hook, agent Bash call.

Always invoke it as the bare word reach. Never python -m tooling, never .venv/bin/reach, never an absolute path. Any of those breaks the Bash(reach *) permission rule and prompts every time — the same failure tea had, where the fix was a bare name on PATH — and reintroduces the interpreter fork between a human shell and a hook.

--editable means the checkout is the source: edit tooling/, run reach, no reinstall. --python is pinned to PYTHON_VERSION so the tool and .venv share one interpreter; left to itself uv picks the lowest version satisfying requires-python, which silently forks the two environments.

Re-pointing. uv records the source path at install time. An install made from a worktree keeps resolving there after the worktree is deleted — reach then still runs, but from a path that no longer exists or, worse, from a stale copy, so edits in the main checkout appear to do nothing. There is no error message for this. Run make reach-repoint from the checkout you want it to follow.

Activating .venv is still needed for running the test scripts directly; it is not needed for reach.

Agent permissions. .claude/settings.json carries Bash(reach) and Bash(reach *) — two entries, because a rule ending in * does not match the bare word, which is why pql, make and cargo test each have a pair too. One command with subcommands means one rule covers the whole tool surface; that is the friction Q-124 was filed about, where ten hand-written Bash(tooling/…) entries each covered a single script.

Note that permission rules prefix-match the whole command string, so an env-prefixed call — SR_REPO_ROOT=… reach check client-version — does not match and will prompt. That is accepted rather than worked around: an environment override is a genuine departure from the normal invocation, and the ordinary form is what needs to be frictionless. Tests that need overrides should pass them through the subprocess environment rather than the command string, as tooling/test_check.py does.

Build

make build              # Build both client and server
make build-server       # cargo build in server/
make build-client       # Client builds are editor-managed (prints guidance)

Run

make server             # cargo run in server/
make client             # Launch Godot with client/ project

The server must be running before the client connects (subprocess launch will be automated later per D-020).

Test

make test               # Run all tests (test-server + test-client)
make test-server        # Rust tests via tests/run-rust (cargo nextest, JSON summary)
make test-client        # Godot tests via tests/run-godot (gdUnit4 headless, JSON summary)
make test-tooling       # Tooling gate: planet-gen determinism guard + import_economics --dry-run (T-1066)

The IPC test layers (D-030) have dedicated targets:

make test-ipc-fixtures      # Layer 1: serialization round-trip fixtures
make test-ipc-protocol      # Layer 2: mock LocalBridge protocol tests
make test-ipc-integration   # Layer 3: real subprocess round-trip (+ benchmark when ready)
make test-ipc-benchmark     # IPC latency benchmark (blocked: #555/#556 handshake)

Each tests/run-* script outputs a JSON summary to stdout and streams progress to stderr:

{"suite":"rust","total":42,"passed":42,"failed":0,"duration_ms":1230}

All scripts accept --filter <name> to run a subset of tests. They are whitelistable for agent use (no TTY prompts, no interactive input).

Server tests use Rust's built-in test framework with #[cfg(test)] inline tests and tests/ integration tests (D-030). Client tests use gdUnit4 (D-030).

Cross-Encoder Fixtures

make fixtures           # Regenerate Rust->GDScript fixtures (server/tests/gen_fixtures.rs)
make fixtures-client    # Generate GDScript->Rust fixtures + verify Rust decoder (#475)

The bidirectional protocol is validated by two sets of committed fixtures:

• Rust encodes, GDScript decodes: client/tests/fixtures/msgpack/ (generated by make fixtures)
• GDScript encodes, Rust decodes: server/tests/fixtures/gdscript/ (generated by make fixtures-client)

Regenerate both after any protocol change. Commit the updated fixtures alongside the code change.

Troubleshooting fixture failures:

• make fixtures-client fails with encode errors: Check that client/addons/messagepack/messagepack.gd is up to date. The script exits non-zero on any encode failure.
• gdscript_generated_fixtures_deserialize fails: Fixtures in server/tests/fixtures/gdscript/ are stale or corrupted. Re-run make fixtures-client and commit the updated files.
• Fixture staleness in make pre-pr: Protocol changed but fixtures were not regenerated. Run make fixtures && make fixtures-client, then commit both client/tests/fixtures/ and server/tests/fixtures/gdscript/.

Golden File Management

make golden-diff        # Show diff if golden file output has changed
make golden-update      # Regenerate golden file and stage for commit

The golden file (server/tests/golden/proof_room_tick_10.json) is a committed snapshot of ObserverSnapshot output after a deterministic 10-tick replay. It catches unintentional changes to simulation output.

Workflow after intentional simulation changes:

1. Run make golden-diff to see what changed
2. Review the diff — confirm changes are expected
3. Run make golden-update to regenerate and stage the new golden file
4. Commit the updated golden file alongside your simulation change

golden-diff exits 1 if the golden file has changed (useful in scripts). golden-update regenerates the file and runs git add but does not commit — the developer reviews and commits manually.

Lint

make lint               # Run all linters
make lint-server        # clippy (deny warnings) + rustfmt --check
make lint-client        # gdlint/gdformat

CI (Local)

Run the full CI pipeline locally before pushing:

make ci                 # Both pipelines
make ci-server          # lint-server → build-server → test-server
make ci-client          # lint-client → build-client → test-client

CI targets chain lint → build → test sequentially. A failure in any stage stops the pipeline.

Pre-PR Checks

Before pushing a PR, run:

make pre-pr

This runs all checks in order: lint → build → test → content validation → fixture staleness. Total runtime ~2.5 minutes (incremental build), under 3 minutes clean.

For branch-specific checks:

make pre-pr-server      # Server changes: lint, build, test, fixture staleness
make pre-pr-client      # Client changes: lint, build, test
make pre-pr-content     # Content changes: schema + cross-reference validation

If pre-pr-fixtures fails, your protocol changes require fixture regeneration:

make fixtures
git add client/tests/fixtures/
git commit -m "chore(fixtures): regenerate for protocol v8"

The fixture staleness check is a blocker (exit 1) — stale fixtures cause false positive client tests.

Clean

make clean              # Remove build artifacts and .cache/ contents

Content Validation

make validate-content     # Validate content YAML against JSON schemas
make check-fact-ids       # Check fact_id references against knowledge catalogs

Gauntlet Checklists

make checklist-validate   # Validate checklist YAML against schema (standalone)
make checklist-generate   # Validate + print per-room condition summary

Checklists live at content/gauntlet/rooms/{room_id}/checklist.yaml (per-room) and content/gauntlet/cross_room_checks.yaml (cross-room). Each condition is evaluable from an ObserverSnapshot.

7 condition types: player_near, player_facing, entity_present, entity_absent, expected_monologue, expected_dialogue, expected_interaction_verb.

Schema: content/_schema/checklist.schema.json. The checklist format feeds into #503 (client auto-checklist progress tracking).

check-fact-ids operates in two modes:

• Advisory — when knowledge catalogs (content/global/knowledge/*.yaml) have no fact definitions yet: lists referenced fact_ids and exits cleanly.
• Enforcing — when catalogs are populated: fails on any fact_id reference that doesn't match a canonical definition.

Asset Pipeline — Generator-Driven DB (#855, #856, #857)

server/data/systems.db is a read-only canonical snapshot produced by a single generator. It is committed to the repo so the client can ship it, but it is never the source of truth. Direct edits are forbidden — they are silently overwritten by the next regeneration.

Generator

import_economics (tooling/economy-db/import_economics.py) is the sole generator. As its first step it shells out to the Rust generate_brands binary (via tooling/generate-brands) to refresh generated_brands.toml, then imports economics data and the atlas index (names-only city pool; geometry tables stay empty for the Phase 4 server cascade). The former atlas geometry generator (generate_atlas) was retired in #951 (D-223).

Run it with:

make regen-db

Meta table stamp

After every successful non-dry-run, the generator writes a row to the meta table in systems.db recording the SHA-1 of its source files and the schema file. The source registry is the GENERATOR_SOURCES dict in tooling/generator_sources.py.

make check-systems-db    # Verify the stamp is fresh (exit 1 = stale)

Making a DB change

1. Edit source files (TOML, JSON, markers.json).
2. make regen-db
3. git add server/data/systems.db
4. Commit with chore(db): regen systems.db — <reason>

For schema changes, also update server/data/systems-schema.sql and add migration DDL to MIGRATION_SQL in import_economics.py.

See .claude/rules/asset-pipeline.md for the full rule set.

Client version mirror

project.yaml's version: is the source of truth, but the client cannot read that file at runtime — an exported build has no repo root above res://. The value is therefore mirrored into client/project.godot as application/config/version, which Godot bakes into the exported PCK, and read through client/scripts/build_version.gd.

This is not cosmetic. The Atlas disk cache (D-255) keys its only invalidation signal on that version, so a version the client cannot read means a cache that can never be invalidated — see T-1241, and T-1239 for what a stale canvas cache actually costs.

reach check client-version   # exit 1 = the two files disagree

The pre-push hook runs this unconditionally (drift persists on main once introduced, so gating it on "were those files touched in this push" would let an existing drift ride along). Bump both files together.

Canvas-generation version pairing

Changing how a canvas is generated is only half a change. The other half is bumping project.yaml's version — otherwise every warm Atlas cache keeps serving canvases built by code that no longer exists. That pairing was never enforced and broke five times (0.4.2 lake_margin_q, 0.4.3 coast_warp_px, 0.4.4 the extent inversion, 0.4.5 the Global sentinel, 0.4.6 one-course-per-river), each bumped only after the fact. The last one took eight days to find (T-1239) because the failure is invisible to its author: it reproduces only where a warm cache exists.

reach check canvas-version   # exit 1 = generation changed, version didn't

The path registry is tooling/canvas_sources.py — globbed, not hand-listed, so a module added in a future split is covered the moment it exists. It deliberately over-includes: a false positive costs one version bump and one round of cache misses, a false negative costs another week of a wrong map.

The registry includes itself, which closes the narrowing hole — removing a path and changing that path in one push still trips the gate, because the registry file is in the set. The cost is that editing the registry requires a version bump.

There is no override flag, on purpose. It would be reached for exactly when someone is certain their change is harmless, which is the state of mind that produced all five regressions. If you are sure, bump anyway — it costs one cache miss.

Units for the gate (including "a comment edit that quotes version numbers is not a bump") run in make test-tooling.

Pre-commit and Pre-push Hooks

Git hooks are stored in .config/hooks/ (version-controlled). Activate them with:

make setup          # Includes hook installation
make install-hooks  # Just hooks (also makes them executable)

Or manually:

git config core.hooksPath .config/hooks

Pre-commit checks (.config/hooks/pre-commit)

Check	Script	Behavior
fact_id validation	reach check fact-ids	Warns if catalogs are stubs; fails on unknown fact_ids when populated
Decision records	pql decisions validate	Blocks on malformed decision records (warns if pql not on PATH)
Planning changelog	pql plan export --stage	Flushes ticket mutations to .pql/changelog/ and stages them into the commit (warn-only on failure)
cargo audit	cargo audit	Only when Cargo.toml/Cargo.lock is staged; blocks on advisories (warns if cargo-audit not installed)

Pre-push checks (.config/hooks/pre-push)

There is no CI — the pre-push gate is the only automatic verification, so it is deliberately comprehensive. Checks are scoped to what actually changed vs the remote (origin/<branch>, falling back to origin/main for first pushes): client/ changes gate the GDScript checks, server/ the Rust checks, tooling/ + pyproject.toml the Python checks.

Check	Runs when	Behavior
GDScript parse (headless Godot)	client/ changed	Blocks on any SCRIPT ERROR (skipped if no client/.godot/ import)
gdlint	client/ changed	Advisory until the codebase is clean
gdformat --check	client/ changed	Advisory until the codebase is clean
cargo fmt --check	server/ changed	Blocks
cargo clippy --all-targets -- -D warnings	server/ changed	Blocks (skipped if no server/target/ — cold worktree)
cargo test	server/ changed	Blocks — the only automatic correctness gate (skipped if no server/target/)
cargo deny check	server/ changed	Blocks (only if cargo-deny installed and server/deny.toml exists)
ruff check tooling/	tooling/ changed	Blocks
make test-tooling	tooling/ changed	Blocks — sim determinism guard + economics dry-run (T-1066)
JSON syntax (python3 -m json.tool)	any changed *.json	Blocks on syntax errors
systems.db stamp	server/data/systems.db in push	Blocks on stale stamp (#857); missing meta table warns only
Clerk review (D-221)	disabled (#965)	Force-run with SR_RUN_CLERK=1

Post-merge / post-checkout / post-rewrite

These hooks replay the pql planning changelog (pql plan import / rebuild) and re-sync decisions from governance/*.md so the planning DB stays in step after merges, checkouts, and history rewrites.

The core.hooksPath setting uses a relative path (.config/hooks) that resolves per worktree, so it works correctly across all worktrees in the repository.

To bypass hooks in an emergency:

git commit --no-verify -m "fix: emergency hotfix"
git push --no-verify

Configuration Files

The .config/ directory currently holds exactly one thing: the version-controlled git hooks in .config/hooks/ (pre-commit, pre-push, post-merge, post-checkout, post-rewrite), activated via git config core.hooksPath .config/hooks (make install-hooks).

Linter and formatter configuration lives with the code it governs, not in .config/: ruff is configured in the root pyproject.toml ([tool.ruff]), Rust uses cargo defaults plus server/deny.toml, and the client uses gdlint/gdformat defaults. Project-specific config (e.g. server/Cargo.toml, client/project.godot) stays in those directories. .config/ is reserved for future cross-cutting configuration that has no better home.

Cache Directory

.cache/ is gitignored and used for:

• Test result caches
• Linter caches
• Build artifact caches (if configured)
• Coverage reports

Agents and CI jobs can write freely to .cache/ without polluting the working tree. make clean clears it.

Testing Architecture (D-030)

Three-layer testing strategy:

1. Unit tests — Inside server/ (Rust #[cfg(test)]) and client/ (gdUnit4). Test individual systems in isolation.
2. Integration tests — Inside server/tests/ (Rust) and tests/ (cross-boundary). Test system interactions, IPC serialization round-trips.
3. Fixture-based tests — IPC serialization fixture files in tests/ for protocol regression testing. Known-good MessagePack payloads verified against both sides.

Key components:

• CauseChain (production ECS component) — Tracks causal attribution for testable observation sequences (D-030).
• Deterministic replay — Server simulation is deterministic given the same seed + input sequence. Replay logs enable regression testing (#201, critical).

Real-rendering test exception: tests/run-visual (tests/visual_capture.gd)

tests/run-godot hardcodes --headless, whose dummy driver produces no usable GPU texture output (SubViewport.get_texture().get_image() returns unusable data). Any test that needs real composited pixels lives in tests/run-visual (client/tests/visual_capture.gd) instead, which boots Godot with a real --rendering-driver opengl3 and asserts on an actual get_viewport().get_texture(). get_image() capture — this is the project's one real-driver exception, and every "did anything draw at all" real-pixel check lives here now, not in a standalone gdUnit file with its own headless-skip guard.

client/tests/test_atlas_window_overlay_draw_smoke.gd (T-1153 live round 4) used to be a second, parallel real-driver path — a standalone gdUnit suite that rendered AtlasWindowOverlay into its own ad hoc SubViewport, self-detecting the dummy driver (DisplayServer.get_name() == "headless") and skipping under tests/run-godot. It retired with the rest of the AtlasWindowViewer/continuous-zoom cluster (T-1182, D-255) and its 2 real-pixel scenarios (single-window draw, tile-mosaic draw) folded into this harness (T-1157): the stepped ladder has no separate "single window" vs "tile mosaic" draw path anymore (D-255(a): "one viewer one path"), so the fold's two natural equivalents are a fixed-rung canvas capture (atlas_GJ380c_District — LINEAR-filtered, StepCanvasTerrainLayer._filter_for_rung()'s non-orbital branch) and the Global-rung capture (atlas_GJ380c_Global — NEAREST-filtered, the orbital branch, a differently-shaped canvas/footprint). Both are ordinary tests/visual.json golden scenarios driven through production navigation (visual_scenarios.gd's _setup_atlas_golden_shot) against a real --test-mode server (SR_LIVE=1) — real derived terrain, real composited pixels, no synthetic stub canvas — closing the "did anything draw at all" gap without a second, parallel real-driver mechanism.

# Any single atlas golden, real driver, real server:
make screenshot SCENARIO=atlas_GJ380c_District
make screenshot SCENARIO=atlas_GJ380c_Global

# The full golden suite (xvfb-wrapped, all scenarios incl. atlas_*):
make test-visual

Planning store (pql)

Tickets and decisions live in pql, not in the old SQLite wrapper scripts. Decisions are markdown-sourced under governance/{decisions,questions,rejected}/; tickets live in .pql/pql.db, rebuilt from the git-tracked .pql/changelog/. Never use the sqlite3 CLI — it crashes in Claude Code (std::bad_alloc).

pql ticket list --status in_progress         # query tickets (T-NNN ids)
pql ticket status T-440 done                  # mutate — flushed to the changelog
pql decisions list --type confirmed           # query decisions
pql decisions show D-010 --with-tickets
make decisions-sync                           # parse governance/*.md into pql.db
make decisions-validate                       # malformed-record gate

Markdown records + the changelog are the source of truth; pql.db is a rebuildable index. The pre-commit hook runs pql decisions validate and exports ticket mutations to the changelog; post-merge/checkout/rewrite hooks replay it. Full reference: .claude/rules/ticket-cli.md.

Commit Conventions

See the /git-commit skill (.claude/skills/git-commit/) for full details. Summary:

• Conventional commits: type(scope): summary
• Types: feat, fix, refactor, chore, docs, data, loc
• Scopes match project subsystems: client, server, engine, simulation, ui, audio, meta, etc.
• Imperative mood, lowercase, no period, max 72 chars
• CHANGELOG.md updated after each commit group