settled-reach/.claude/rules/asset-pipeline.md

Asset Pipeline — Source-Canonical Rule

server/data/systems.db is a read-only, deterministic snapshot produced by the generator pipeline. It is checked in to the repo as a build artefact so the Godot client can ship it without a build step, but it is never the source of truth.

---

The Golden Rule

Edit sources, not the DB.

If you need to change economics data, modify the TOML/JSON source files. If you need to change the atlas city-name pool, modify the names-only markers.json files (D-223 — they carry no geometry or population). Never run UPDATE or INSERT directly on server/data/systems.db outside of a migration — those changes will be silently overwritten by the next make regen-db.

---

What produces systems.db

One generator writes to systems.db: import_economics (python3 tooling/economy-db/import_economics.py, run via make regen-db). The former atlas geometry generator (generate_atlas) was retired in #951 (D-223); import_economics now also owns the atlas index — it loads the names-only markers.json city pool into atlas_city_names and empties the geometry tables (the server cascade fills them, Phase 4).

import_economics shells out to the Rust generate_brands binary as its first step to refresh wiki/economics/corporations/generated_brands.toml, then reads the TOML and imports brand data into the DB. The Rust binary is a subroutine of the Python importer, not an independent generator — changes to its source invalidate the import_economics meta stamp even though the Python file itself didn't change.

The full set of source files contributing to the meta stamp SHA (the Python importer entrypoint + its economy_import module package, the Rust brand binary sources, tooling/schema_version.py, the authored economics TOMLs, and the registry itself) is defined once in tooling/generator_sources.py (T-1067) — imported by both the importer's stamp writer and check-systems-db-stamp, and listed via python3 tooling/generator_sources.py --list.

The surviving planet-gen importers (import_heightmaps, import_province_boundaries) are one-time build imports baked into the committed DB — not part of make regen-db, and intentionally not stamped.

---

The meta table stamp (T-855, T-856)

After every successful non-dry-run, the generator writes a row to the meta table:

CREATE TABLE meta (
    generator_name TEXT PRIMARY KEY,   -- 'import_economics' (sole generator since #951/D-223)
    schema_version TEXT NOT NULL,      -- monotonic semver string (e.g. "1.0.0") — see T-888
    schema_sha TEXT,                   -- SHA-1 of server/data/systems-schema.sql (tamper detection)
    generator_sha TEXT NOT NULL,       -- SHA-1 of the generator source file(s)
    generated_at TEXT NOT NULL DEFAULT (datetime('now'))
);

schema_version is a monotonic semver string (e.g. "1.0.0"), not a hash. It is defined as the SCHEMA_VERSION constant in tooling/schema_version.py and must be bumped manually whenever the schema changes in a backwards-incompatible way. Unlike a SHA-1 hash, semver strings are orderable — this enables savegame migration lineage in Phase 5+: a save file can record which schema version it derives from and determine exactly which migrations to apply (T-888). The old SHA-1 is preserved in schema_sha for tamper detection alongside the semver.

The generator_sha is the SHA-1 of the concatenated bytes of the generator's source files (sorted by path, so order is deterministic). If any source file changes and make regen-db is not re-run, the stamped SHA will differ from the recomputed current SHA — this is what the pre-push hook detects.

What's deterministic: the stored SHA (same sources → same recorded SHA). What's NOT deterministic: the DB binary itself. meta.generated_at uses datetime('now'), SQLite rowid/autoincrement values drift across runs, and transaction ordering can reshape freelist pages — two consecutive make regen-db calls produce byte-different SQLite files even with identical inputs. This is fine: the freshness guarantee comes from the stamp, not from bytewise DB equality.

---

How to make a DB change

Normal data changes (economics, atlas markers)

1. Edit the source files (TOML, JSON, markers.json).
2. Run make regen-db.
3. Run make check-systems-db to confirm the stamp is fresh.
4. Stage and commit:

   git add server/data/systems.db
   git commit -m "chore(db): regen systems.db — <what changed>"
   

Schema changes (new tables or columns)

1. Add the DDL to server/data/systems-schema.sql.
2. Add migration SQL to MIGRATION_SQL in import_economics.py if the change affects existing DBs (idempotent CREATE TABLE IF NOT EXISTS or ALTER TABLE).
3. Run make regen-db.
4. Stage server/data/systems-schema.sql and server/data/systems.db together.

---

Pre-push hook (T-857)

.config/hooks/pre-push (installed via make install-hooks) checks that whenever server/data/systems.db is in the push, its meta stamp matches the current generator source SHAs. If not, the push is rejected with:

systems.db is stale — run `make regen-db` before pushing.
  Stale generators: ['import_economics']

Fix: run make regen-db, stage server/data/systems.db, amend or add a commit. Or use /pr-process — it detects stale generator sources and reruns make regen-db automatically before pushing.

The check script is tooling/check-systems-db-stamp. Run it interactively with make check-systems-db or python3 tooling/check-systems-db-stamp --verbose. The GENERATOR_SOURCES dict in tooling/generator_sources.py is the single registry (T-1067) — the check script and the importer's stamp writer both import it, and the /pr-process skill derives its source-file watch list from python3 tooling/generator_sources.py --list. When you add a new generator or source file, register it there and nowhere else.

---

/pr-process integration (T-858)

The /pr-process skill checks whether any generator source files are modified on the branch. If they are, it automatically runs make regen-db and stages the updated server/data/systems.db before pushing — preventing pre-push hook rejections on branches that modify generators without regenerating.

---

Why direct DB edits are forbidden

Two branches that both commit server/data/systems.db changes produce a binary merge conflict. Git cannot diff or merge binary SQLite files. Sprint 36 hit this exact class of problem. The meta stamp + pre-push hook is the systematic fix:

• The stamp is deterministic (same generator source → same recorded SHA)
• Only one branch modifies generator sources at a time (per team scope rules)
• The pre-push hook is a hard blocker before the binary conflict can land

The migration escape hatch

The rule above says "never run UPDATE or INSERT directly on systems.db outside of a migration." Here's what a legitimate migration looks like, and what isn't one:

Sanctioned path: the MIGRATION_SQL block in import_economics.py. That string is executed at the top of every import run (inside the same transaction that clears + reimports data) and contains idempotent CREATE TABLE IF NOT EXISTS / CREATE INDEX IF NOT EXISTS statements, plus ALTER TABLE additions handled via the COLUMN_MIGRATIONS list. When you need a new table, column, or index on systems.db, add it there. It'll run on the next make regen-db and the meta stamp will flip because import_economics.py changed.

Also legitimate: edits to server/data/systems-schema.sql (the canonical DDL used by fresh builds) paired with matching entries in MIGRATION_SQL for existing DBs. The stamp's schema_version field records the schema file's SHA at generation time — change the schema, commit both files together, and the stamp picks it up automatically.

NOT legitimate and forbidden:

• Running any raw SQL (e.g. the sqlite3 CLI) against systems.db by hand. Any changes you make are silently reverted by the next regen-db run — your edits die, not the pipeline's.
• One-off patch scripts that open systems.db and modify rows.
• Editing the DB file with a SQLite GUI.
• Committing systems.db alone, without the corresponding source change that would explain the diff on regen.

If you think you need an exception, the right move is to make the source change explicit instead: either edit the wiki TOMLs / JSONs that feed the generators, or edit MIGRATION_SQL / systems-schema.sql directly. There is no hand-edit path that survives regen.

---

Savegame migration lineage (Phase 5+)

meta.schema_version now stores a monotonic semver string (T-888). When the savegame system is built (Phase 5+), a save file records its schema_version string; the loader can determine which migrations to apply by comparing that version to the current one. meta.schema_sha retains the old SHA-1 for tamper detection.

When to bump SCHEMA_VERSION: edit the SCHEMA_VERSION = "1.0.0" constant in tooling/schema_version.py whenever a schema change is backwards-incompatible (column removed, type changed, FK constraint added, table dropped). Additive changes (new nullable columns, new tables, new indexes) do not require a bump.