settled-reach/.claude/rules/asset-pipeline.md

Asset Pipeline — Source-Canonical Rule

server/data/systems.db is a read-only, deterministic snapshot produced by the generator pipeline. It is checked in to the repo as a build artefact so the Godot client can ship it without a build step, but it is never the source of truth.

---

The Golden Rule

Edit sources, not the DB.

If you need to change economics data, modify the TOML/JSON source files. If you need to change atlas markers, modify the markers.json files. Never run UPDATE or INSERT directly on server/data/systems.db outside of a migration — those changes will be silently overwritten by the next make regen-db.

---

What produces systems.db

Two generators write to systems.db:

Generator	Command	Source files (all contribute to the meta stamp SHA)
import_economics	python3 tooling/economy-db/import_economics.py	tooling/economy-db/import_economics.py + the Rust brand binary sources it invokes: server/src/bin/generate_brands/main.rs, server/src/bin/generate_brands/names.rs, tooling/generate-brands
generate_atlas	python3 tooling/planet-gen/generate_atlas.py --seed 42	tooling/planet-gen/generate_atlas.py

import_economics shells out to the Rust generate_brands binary as its first step to refresh wiki/economics/corporations/generated_brands.toml, then reads the TOML and imports brand data into the DB. The Rust binary is a subroutine of the Python importer, not an independent generator — changes to its source invalidate the import_economics meta stamp even though the Python file itself didn't change.

make regen-db runs both in the correct order (economics first, atlas second).

---

The meta table stamp (#855, #856)

After every successful non-dry-run, each generator writes a row to the meta table:

CREATE TABLE meta (
    generator_name TEXT PRIMARY KEY,   -- 'import_economics' | 'generate_atlas'
    schema_version TEXT NOT NULL,      -- SHA-1 of server/data/systems-schema.sql at generation time
    generator_sha TEXT NOT NULL,       -- SHA-1 of the generator source file(s)
    generated_at TEXT NOT NULL DEFAULT (datetime('now'))
);

The generator_sha is the SHA-1 of the concatenated bytes of the generator's source files (sorted by path, so order is deterministic). If any source file changes and make regen-db is not re-run, the stamped SHA will differ from the recomputed current SHA — this is what the pre-push hook detects.

What's deterministic: the stored SHA (same sources → same recorded SHA). What's NOT deterministic: the DB binary itself. meta.generated_at uses datetime('now'), SQLite rowid/autoincrement values drift across runs, and transaction ordering can reshape freelist pages — two consecutive make regen-db calls produce byte-different SQLite files even with identical inputs. This is fine: the freshness guarantee comes from the stamp, not from bytewise DB equality.

---

How to make a DB change

Normal data changes (economics, atlas markers)

1. Edit the source files (TOML, JSON, markers.json).
2. Run make regen-db.
3. Run make check-systems-db to confirm the stamp is fresh.
4. Stage and commit:

   git add server/data/systems.db
   git commit -m "chore(db): regen systems.db — <what changed>"
   

Schema changes (new tables or columns)

1. Add the DDL to server/data/systems-schema.sql.
2. Add migration SQL to MIGRATION_SQL in import_economics.py if the change affects existing DBs (idempotent CREATE TABLE IF NOT EXISTS or ALTER TABLE).
3. Run make regen-db.
4. Stage server/data/systems-schema.sql and server/data/systems.db together.

---

Pre-push hook (#857)

.config/hooks/pre-push (installed via make install-hooks) checks that whenever server/data/systems.db is in the push, its meta stamp matches the current generator source SHAs. If not, the push is rejected with:

systems.db is stale — run `make regen-db` before pushing.
  Stale generators: ['import_economics']

Fix: run make regen-db, stage server/data/systems.db, amend or add a commit. Or use /pr-push — it detects stale generator sources and reruns make regen-db automatically before pushing.

The check script is tooling/check-systems-db-stamp. Run it interactively with make check-systems-db or python3 tooling/check-systems-db-stamp --verbose. The GENERATOR_SOURCES dict at the top of that script is the single registry — when you add a new generator or source file, update it there and mirror the change in the /pr-push skill's source-file watch list.

---

/pr-push integration (#858)

The /pr-push skill checks whether any generator source files are modified on the branch. If they are, it automatically runs make regen-db and stages the updated server/data/systems.db before pushing — preventing pre-push hook rejections on branches that modify generators without regenerating.

---

Why direct DB edits are forbidden

Two branches that both commit server/data/systems.db changes produce a binary merge conflict. Git cannot diff or merge binary SQLite files. Sprint 36 hit this exact class of problem. The meta stamp + pre-push hook is the systematic fix:

• The stamp is deterministic (same generator source → same recorded SHA)
• Only one branch modifies generator sources at a time (per team scope rules)
• The pre-push hook is a hard blocker before the binary conflict can land

The migration escape hatch

The rule above says "never run UPDATE or INSERT directly on systems.db outside of a migration." Here's what a legitimate migration looks like, and what isn't one:

Sanctioned path: the MIGRATION_SQL block in import_economics.py. That string is executed at the top of every import run (inside the same transaction that clears + reimports data) and contains idempotent CREATE TABLE IF NOT EXISTS / CREATE INDEX IF NOT EXISTS statements, plus ALTER TABLE additions handled via the COLUMN_MIGRATIONS list. When you need a new table, column, or index on systems.db, add it there. It'll run on the next make regen-db and the meta stamp will flip because import_economics.py changed.

Also legitimate: edits to server/data/systems-schema.sql (the canonical DDL used by fresh builds) paired with matching entries in MIGRATION_SQL for existing DBs. The stamp's schema_version field records the schema file's SHA at generation time — change the schema, commit both files together, and the stamp picks it up automatically.

NOT legitimate and forbidden:

• Running tooling/db/sqlite-exec (or any raw SQL) against systems.db by hand. Any changes you make are silently reverted by the next regen-db run — your edits die, not the pipeline's.
• One-off patch scripts that open systems.db and modify rows.
• Editing the DB file with a SQLite GUI.
• Committing systems.db alone, without the corresponding source change that would explain the diff on regen.

If you think you need an exception, the right move is to make the source change explicit instead: either edit the wiki TOMLs / JSONs that feed the generators, or edit MIGRATION_SQL / systems-schema.sql directly. There is no hand-edit path that survives regen.

---

Future: savegame migration lineage

The meta.schema_version field records the schema SHA at generation time. When the savegame system is built (Phase 5+), a save file can record which systems.db snapshot it derives from, enabling forward migration without branching the DB file itself.