settled-reach/.claude/rules/asset-pipeline.md

# Asset Pipeline — Source-Canonical Rule

`server/data/systems.db` is a **read-only, deterministic snapshot** produced by the
generator pipeline.  It is checked in to the repo as a build artefact so the Godot
client can ship it without a build step, but **it is never the source of truth**.

---

## The Golden Rule

> **Edit sources, not the DB.**

If you need to change economics data, modify the TOML/JSON source files.
If you need to change the atlas city-name pool, modify the names-only
`markers.json` files (D-223 — they carry no geometry or population).
Never run `UPDATE` or `INSERT` directly on `server/data/systems.db` outside of a
migration — those changes will be silently overwritten by the next `make regen-db`.

---

## What produces systems.db

**One generator** writes to `systems.db`: `import_economics`
(`python3 tooling/economy-db/import_economics.py`, run via `make regen-db`).
The former atlas geometry generator (`generate_atlas`) was retired in #951
(D-223); `import_economics` now also owns the atlas index — it loads the
names-only `markers.json` city pool into `atlas_city_names` and empties the
geometry tables (the server cascade fills them, Phase 4).

`import_economics` shells out to the Rust `generate_brands` binary as its first
step to refresh `wiki/economics/corporations/generated_brands.toml`, then reads
the TOML and imports brand data into the DB.  The Rust binary is a subroutine
of the Python importer, not an independent generator — changes to its source
invalidate the `import_economics` meta stamp even though the Python file
itself didn't change.

The full set of source files contributing to the meta stamp SHA (the Python
importer entrypoint + its `economy_import` module package, the Rust brand
binary sources, `tooling/schema_version.py`, the authored economics TOMLs, and
the registry itself) is defined once in `tooling/generator_sources.py` (T-1067)
— imported by both the importer's stamp writer and `check-systems-db-stamp`,
and listed via `python3 tooling/generator_sources.py --list`.

The surviving planet-gen importers (`import_heightmaps`,
`import_province_boundaries`) are one-time build imports baked into the
committed DB — not part of `make regen-db`, and intentionally not stamped.

---

## The meta table stamp (T-855, T-856)

After every successful non-dry-run, the generator writes a row to the `meta` table:

```sql
CREATE TABLE meta (
    generator_name TEXT PRIMARY KEY,   -- 'import_economics' (sole generator since #951/D-223)
    schema_version TEXT NOT NULL,      -- monotonic semver string (e.g. "1.0.0") — see T-888
    schema_sha TEXT,                   -- SHA-1 of server/data/systems-schema.sql (tamper detection)
    generator_sha TEXT NOT NULL,       -- SHA-1 of the generator source file(s)
    generated_at TEXT NOT NULL DEFAULT (datetime('now'))
);
```

`schema_version` is a **monotonic semver string** (e.g. `"1.0.0"`), not a hash.
It is defined as the `SCHEMA_VERSION` constant in `tooling/schema_version.py`
and must be bumped manually whenever the schema changes in a backwards-incompatible way.
Unlike a SHA-1 hash, semver strings are orderable — this enables savegame migration
lineage in Phase 5+: a save file can record which schema version it derives from and
determine exactly which migrations to apply (T-888).  The old SHA-1 is preserved in
`schema_sha` for tamper detection alongside the semver.

The `generator_sha` is the SHA-1 of the concatenated bytes of the generator's
source files (sorted by path, so order is deterministic).  If any source file
changes and `make regen-db` is not re-run, the stamped SHA will differ from the
recomputed current SHA — this is what the pre-push hook detects.

**What's deterministic:** the stored SHA (same sources → same recorded SHA).
**What's NOT deterministic:** the DB binary itself.  `meta.generated_at` uses
`datetime('now')`, SQLite `rowid`/`autoincrement` values drift across runs, and
transaction ordering can reshape freelist pages — two consecutive `make regen-db`
calls produce byte-different SQLite files even with identical inputs.  This is
fine: the freshness guarantee comes from the stamp, not from bytewise DB equality.

---

## How to make a DB change

### Normal data changes (economics, atlas markers)

1. Edit the source files (TOML, JSON, markers.json).
2. Run `make regen-db`.
3. Run `make check-systems-db` to confirm the stamp is fresh.
4. Stage and commit:
   ```bash
   git add server/data/systems.db
   git commit -m "chore(db): regen systems.db — <what changed>"
   ```

### Schema changes (new tables or columns)

1. Add the DDL to `server/data/systems-schema.sql`.
2. Add migration SQL to `MIGRATION_SQL` in `import_economics.py` if the change
   affects existing DBs (idempotent `CREATE TABLE IF NOT EXISTS` or `ALTER TABLE`).
3. Run `make regen-db`.
4. Stage `server/data/systems-schema.sql` and `server/data/systems.db` together.

---

## Pre-push hook (T-857)

`.config/hooks/pre-push` (installed via `make install-hooks`) checks that whenever
`server/data/systems.db` is in the push, its meta stamp matches the current generator
source SHAs.  If not, the push is rejected with:

```
systems.db is stale — run `make regen-db` before pushing.
  Stale generators: ['import_economics']
```

Fix: run `make regen-db`, stage `server/data/systems.db`, amend or add a commit.
Or use `/pr-process` — it detects stale generator sources and reruns `make regen-db`
automatically before pushing.

The check script is `tooling/check-systems-db-stamp`.  Run it interactively with
`make check-systems-db` or `python3 tooling/check-systems-db-stamp --verbose`.  The
`GENERATOR_SOURCES` dict in `tooling/generator_sources.py` is the single registry
(T-1067) — the check script and the importer's stamp writer both import it, and
the `/pr-process` skill derives its source-file watch list from
`python3 tooling/generator_sources.py --list`.  When you add a new generator or
source file, register it there and nowhere else.

---

## /pr-process integration (T-858)

The `/pr-process` skill checks whether any generator source files are modified on the
branch.  If they are, it automatically runs `make regen-db` and stages the updated
`server/data/systems.db` before pushing — preventing pre-push hook rejections on
branches that modify generators without regenerating.

---

## Why direct DB edits are forbidden

Two branches that both commit `server/data/systems.db` changes produce a binary
merge conflict.  Git cannot diff or merge binary SQLite files.  Sprint 36 hit this
exact class of problem.  The meta stamp + pre-push hook is the systematic fix:

- The stamp is deterministic (same generator source → same recorded SHA)
- Only one branch modifies generator sources at a time (per team scope rules)
- The pre-push hook is a hard blocker before the binary conflict can land

## The migration escape hatch

The rule above says "never run UPDATE or INSERT directly on systems.db outside
of a migration."  Here's what a legitimate migration looks like, and what isn't
one:

**Sanctioned path: the `MIGRATION_SQL` block in `import_economics.py`.**  That
string is executed at the top of every import run (inside the same transaction
that clears + reimports data) and contains idempotent `CREATE TABLE IF NOT
EXISTS` / `CREATE INDEX IF NOT EXISTS` statements, plus `ALTER TABLE` additions
handled via the `COLUMN_MIGRATIONS` list.  When you need a new table, column,
or index on systems.db, add it there.  It'll run on the next `make regen-db`
and the meta stamp will flip because `import_economics.py` changed.

**Also legitimate:** edits to `server/data/systems-schema.sql` (the canonical
DDL used by fresh builds) paired with matching entries in `MIGRATION_SQL` for
existing DBs.  The stamp's `schema_version` field records the schema file's
SHA at generation time — change the schema, commit both files together, and
the stamp picks it up automatically.

**NOT legitimate and forbidden:**

- Running any raw SQL (e.g. the `sqlite3` CLI) against `systems.db` by
  hand.  Any changes you make are silently reverted by the next `regen-db` run
  — your edits die, not the pipeline's.
- One-off patch scripts that open `systems.db` and modify rows.
- Editing the DB file with a SQLite GUI.
- Committing `systems.db` alone, without the corresponding source change that
  would explain the diff on regen.

If you think you need an exception, the right move is to make the source
change explicit instead: either edit the wiki TOMLs / JSONs that feed the
generators, or edit `MIGRATION_SQL` / `systems-schema.sql` directly.  There is
no hand-edit path that survives regen.

---

## Savegame migration lineage (Phase 5+)

`meta.schema_version` now stores a monotonic semver string (T-888).  When the savegame
system is built (Phase 5+), a save file records its `schema_version` string; the
loader can determine which migrations to apply by comparing that version to the
current one.  `meta.schema_sha` retains the old SHA-1 for tamper detection.

**When to bump `SCHEMA_VERSION`:** edit the `SCHEMA_VERSION = "1.0.0"` constant in
`tooling/schema_version.py` whenever a schema change is backwards-incompatible
(column removed, type changed, FK constraint added, table dropped).  Additive changes
(new nullable columns, new tables, new indexes) do not require a bump.