diff --git a/.claude/rules/asset-pipeline.md b/.claude/rules/asset-pipeline.md new file mode 100644 index 000000000..c29b851d4 --- /dev/null +++ b/.claude/rules/asset-pipeline.md @@ -0,0 +1,121 @@ +# Asset Pipeline — Source-Canonical Rule + +`server/data/systems.db` is a **read-only, deterministic snapshot** produced by the +generator pipeline. It is checked in to the repo as a build artefact so the Godot +client can ship it without a build step, but **it is never the source of truth**. + +--- + +## The Golden Rule + +> **Edit sources, not the DB.** + +If you need to change economics data, modify the TOML/JSON source files. +If you need to change atlas markers, modify the `markers.json` files. +Never run `UPDATE` or `INSERT` directly on `server/data/systems.db` outside of a +migration — those changes will be silently overwritten by the next `make regen-db`. + +--- + +## What produces systems.db + +Three generators write to `systems.db` in sequence: + +| Generator | Command | Source files | +|-----------|---------|--------------| +| `generate_brands` | `tooling/generate-brands` | `server/src/bin/generate_brands/main.rs` | +| `import_economics` | `python3 tooling/economy-db/import_economics.py` | `tooling/economy-db/import_economics.py` | +| `generate_atlas` | `python3 tooling/planet-gen/generate_atlas.py --seed 42` | `tooling/planet-gen/generate_atlas.py` | + +`make regen-db` runs all three in the correct order. + +--- + +## The meta table stamp (#855, #856) + +After every successful non-dry-run, each generator writes a row to the `meta` table: + +```sql +CREATE TABLE meta ( + generator_name TEXT PRIMARY KEY, -- 'import_economics' | 'generate_atlas' | 'generate_brands' + schema_version TEXT NOT NULL, -- SHA-1 of server/data/systems-schema.sql at generation time + generator_sha TEXT NOT NULL, -- SHA-1 of the generator source file(s) + generated_at TEXT NOT NULL DEFAULT (datetime('now')) +); +``` + +The `generator_sha` is the SHA-1 of the generator source file content. If the source +changes and `regen-db` is not re-run, the stamped SHA will differ from the current file +SHA — this is what the pre-push hook detects. + +--- + +## How to make a DB change + +### Normal data changes (economics, atlas markers) + +1. Edit the source files (TOML, JSON, markers.json). +2. Run `make regen-db`. +3. Run `make check-systems-db` to confirm the stamp is fresh. +4. Stage and commit: + ```bash + git add server/data/systems.db + git commit -m "chore(db): regen systems.db — " + ``` + +### Schema changes (new tables or columns) + +1. Add the DDL to `server/data/systems-schema.sql`. +2. Add migration SQL to `MIGRATION_SQL` in `import_economics.py` if the change + affects existing DBs (idempotent `CREATE TABLE IF NOT EXISTS` or `ALTER TABLE`). +3. Run `make regen-db`. +4. Stage `server/data/systems-schema.sql` and `server/data/systems.db` together. + +--- + +## Pre-push hook (#857) + +`.config/hooks/pre-push` (installed via `make install-hooks`) checks that whenever +`server/data/systems.db` is in the push, its meta stamp matches the current generator +source SHAs. If not, the push is rejected with: + +``` +systems.db is stale — run `make regen-db` before pushing. + Stale generators: ['import_economics'] +``` + +Fix: run `make regen-db`, stage `server/data/systems.db`, amend or add a commit. +Or use `/pr-push` — it detects stale generator sources and reruns `make regen-db` +automatically before pushing. + +The check script is `tooling/check-systems-db-stamp`. Run it interactively with +`make check-systems-db` or `python3 tooling/check-systems-db-stamp --verbose`. + +--- + +## /pr-push integration (#858) + +The `/pr-push` skill checks whether any generator source files are modified on the +branch. If they are, it automatically runs `make regen-db` and stages the updated +`server/data/systems.db` before pushing — preventing pre-push hook rejections on +branches that modify generators without regenerating. + +--- + +## Why direct DB edits are forbidden + +Two branches that both commit `server/data/systems.db` changes produce a binary +merge conflict. Git cannot diff or merge binary SQLite files. Sprint 36 hit this +exact class of problem. The meta stamp + pre-push hook is the systematic fix: + +- Regeneration is deterministic (same sources → same DB, byte-for-byte) +- Only one branch modifies generator sources at a time (per team scope rules) +- The pre-push hook is a hard blocker before the binary conflict can land + +--- + +## Future: savegame migration lineage + +The `meta.schema_version` field records the schema SHA at generation time. When the +savegame system is built (Phase 5+), a save file can record which systems.db snapshot +it derives from, enabling forward migration without branching the DB file itself. diff --git a/.claude/skills/sprint-start/SKILL.md b/.claude/skills/sprint-start/SKILL.md index 29f501407..ff54f77aa 100644 --- a/.claude/skills/sprint-start/SKILL.md +++ b/.claude/skills/sprint-start/SKILL.md @@ -423,6 +423,9 @@ Task( 1. Read the sprint briefing: docs/sprints/sprint-{N}/{team}.md 2. Read the decision files referenced in the briefing. + If your work touches systems.db sources (markers.json, TOML files, + or generator code), read .claude/rules/asset-pipeline.md before + modifying anything. 3. Check TaskList for available work. 4. Claim an unblocked task (TaskUpdate with owner: your name), mark it in_progress, and implement it. diff --git a/CLAUDE.md b/CLAUDE.md index e2edfb52c..316f47472 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,6 +25,12 @@ Full annotated tree: `.claude/rules/project-structure.md` See [docs/DEVOPS.md](docs/DEVOPS.md) for build, test, lint, and CI procedures. All development operations go through the top-level `Makefile` — run `make` for a summary of targets. +### Asset pipeline + +`server/data/systems.db` is a read-only canonical snapshot produced by the generator +pipeline — never edit it directly. To regenerate: `make regen-db`. Full rules in +`.claude/rules/asset-pipeline.md`. + ## Development Cascade — First Things First Development follows a strict cascade. Each phase has a concrete deliverable. **Do NOT discuss, design, or implement detail from a later phase while an earlier phase is incomplete.** If you encounter references to later-phase detail (room grammar, NPC bundles, heritage tokens, etc.) in documents or decisions, either ignore them silently and stay at the correct level, or flag that the reference is dragging attention to the wrong scope level and suggest it be rephrased or moved. @@ -89,7 +95,7 @@ The ticketing database (`settledreach.db`) is accessed via `SR_DB_PATH` env var | Sprints | `tooling/db/sprint status`, `start-work`, `prepare` | `/sprint-start` skill | | SQL queries | `tooling/db/sqlite-query "SELECT ..."` | — | | SQL writes | `tooling/db/sqlite-exec "UPDATE ..."` | — | -| Decisions | `tooling/db/decision next`, `claim`, `check-dupes` | — | +| Decisions | `tooling/db/decision show`, `next`, `claim`, `check-dupes` | — | ### Testing preferences diff --git a/docs/DEVOPS.md b/docs/DEVOPS.md index 383aa2e7c..4cd94d3c2 100644 --- a/docs/DEVOPS.md +++ b/docs/DEVOPS.md @@ -207,13 +207,55 @@ Schema: `content/_schema/checklist.schema.json`. The checklist format feeds into - **Advisory** — when knowledge catalogs (`content/global/knowledge/*.yaml`) have no fact definitions yet: lists referenced fact_ids and exits cleanly. - **Enforcing** — when catalogs are populated: fails on any `fact_id` reference that doesn't match a canonical definition. -## Pre-commit Hooks +## Asset Pipeline — Generator-Driven DB (#855, #856, #857) + +`server/data/systems.db` is a **read-only canonical snapshot** produced by three +generators. It is committed to the repo so the client can ship it, but it is never +the source of truth. Direct edits are forbidden — they are silently overwritten by +the next regeneration. + +### Generators + +| Generator | Source | Runs via | +|-----------|--------|----------| +| `generate_brands` | `server/src/bin/generate_brands/main.rs` | `tooling/generate-brands` | +| `import_economics` | `tooling/economy-db/import_economics.py` | `python3 tooling/economy-db/import_economics.py` | +| `generate_atlas` | `tooling/planet-gen/generate_atlas.py` | `python3 tooling/planet-gen/generate_atlas.py --seed 42` | + +Run all three at once with: + +```bash +make regen-db +``` + +### Meta table stamp + +After every successful non-dry-run, each generator writes a row to the `meta` table in +`systems.db` recording the SHA-1 of its source file(s) and the schema file. + +```bash +make check-systems-db # Verify the stamp is fresh (exit 1 = stale) +``` + +### Making a DB change + +1. Edit source files (TOML, JSON, `markers.json`). +2. `make regen-db` +3. `git add server/data/systems.db` +4. Commit with `chore(db): regen systems.db — ` + +For schema changes, also update `server/data/systems-schema.sql` and add migration DDL +to `MIGRATION_SQL` in `import_economics.py`. + +See `.claude/rules/asset-pipeline.md` for the full rule set. + +## Pre-commit and Pre-push Hooks Git hooks are stored in `.config/hooks/` (version-controlled). Activate them with: ```bash make setup # Includes hook installation -make setup-hooks # Just hooks +make install-hooks # Just hooks (also makes them executable) ``` Or manually: @@ -224,9 +266,14 @@ git config core.hooksPath .config/hooks Active checks: -| Check | Script | Behavior | -|-------|--------|----------| -| fact_id validation | `tooling/check-fact-ids` | Warns if catalogs are stubs; fails on unknown fact_ids when populated | +| Hook | Check | Script | Behavior | +|------|-------|--------|----------| +| pre-commit | fact_id validation | `tooling/check-fact-ids` | Warns if catalogs are stubs; fails on unknown fact_ids when populated | +| pre-push | GDScript parse | internal | Fails on any SCRIPT ERROR | +| pre-push | Rust lint | internal | fmt + clippy | +| pre-push | Python lint | internal | ruff | +| pre-push | JSON syntax | internal | python3 -m json.tool | +| pre-push | systems.db stamp | `tooling/check-systems-db-stamp` | Rejects stale DB when pushed (#857) | The `core.hooksPath` setting uses a relative path (`.config/hooks`) that resolves per worktree, so it works correctly across all worktrees in the repository.