chore(ci): merge brand pipeline into import_economics + harden review findings

Addresses all blocking + minor items from PR #136 review.

Architectural change (T2/H3 — the review's main complaint):

  generate_brands was previously a separate Rust binary that produced a TOML
  artifact, with its stamp written "on behalf" by import_economics.py at the
  end of its own run.  Reviewers flagged the invisible coupling: two sources
  of truth in a system designed to have one, and no way to tell from the
  stamp that one "generator" was really a subroutine of the other.

  import_economics now invokes tooling/generate-brands as the first step of
  its main() flow, before opening its own DB connection.  The TOML artefact
  is still produced and still committed (useful for diff-review of brand
  changes), but there's now one pipeline owner.  The meta table carries two
  rows (import_economics, generate_atlas) not three; the Rust binary's
  source SHA folds into import_economics' stamp via IMPORT_ECONOMICS_SOURCES.
  A MIGRATION_SQL DELETE cleans up pre-merge DBs that still have the
  orphan generate_brands row.

Other review items addressed in-line:

  H1  generate_atlas._write_stamp no longer commits — transaction ownership
      stays with the caller (matches import_economics pattern).  Stamp +
      atlas data now commit atomically; a failed stamp rolls back the
      atlas data rather than leaving a stamp-missing-data intermediate.

  H2  _file_sha1 (in both import_economics, generate_atlas,
      check-systems-db-stamp) raises FileNotFoundError on missing sources
      instead of silently contributing an empty-bytes hash.  A ghost-SHA
      convergence could otherwise produce vacuous "fresh" passes.

  H4  pre-push no-meta-table warning rephrased — was "run after next
      regeneration", now "run now if this DB was generated by you".

  T1  asset-pipeline.md determinism claim softened: the stamp is
      deterministic (same source → same recorded SHA), the DB binary is
      not (generated_at + SQLite rowids/freelist churn).

  T3  asset-pipeline.md gains a "migration escape hatch" section naming
      MIGRATION_SQL in import_economics.py as the only sanctioned path
      for direct writes, and forbidding hand-run sqlite-exec / one-off
      patch scripts / SQLite-GUI edits.

  T4  Makefile regen-db now runs as a single shell with `set -e`.  A
      failure in one generator halts the pipeline immediately, preventing
      the "stale data, fresh stamp" state where a later step stamped a
      DB whose earlier step had failed.  import_economics' exit code 2
      (coverage gate warning) remains explicitly tolerated.

  T5  pre-push stamp check now runs on a branch's first push too —
      compares against origin/main instead of origin/$BRANCH, closing
      the gap where a new branch could ship a stale DB via the first push.

  T6  check-systems-db-stamp fails closed on unknown generator_names in
      meta — a future branch adding a new generator without registering
      it in GENERATOR_SOURCES will now be rejected, not silently skipped.

  T7  /pr-push watch list gains a mutual cross-reference comment with
      GENERATOR_SOURCES in check-systems-db-stamp, plus the missing
      names.rs source file, so the two lists cannot silently drift.

Follow-up tickets created:

  #887 T8  decisions-orphan-tickets CLI — surfaces tickets whose
           decision_ref points at a non-existent D-record.
  #888 T9  meta.schema_version monotonic semver — for savegame migration
           lineage in Phase 5+ (SHA comparison can't be ordered).

Verified:

  make regen-db end-to-end — OK
  make check-systems-db    — OK, 2 generator(s) up to date
  STALE detection          — OK, verified by touching generate_atlas.py
  /pr-push watch list      — OK, flags this branch's changed sources
  decision show D-159      — OK, structured output with tickets + refs

Refs: #855 #856 #857 #858 #859 PR #136

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-04-22 08:38:41 +02:00
co-authored by Claude Opus 4.6
parent 07ca1c9fc1
commit 3d9dd7d909
8 changed files with 253 additions and 66 deletions
+63 -11
View File
@@ -19,15 +19,21 @@ migration — those changes will be silently overwritten by the next `make regen
## What produces systems.db
Three generators write to `systems.db` in sequence:
Two generators write to `systems.db`:
| Generator | Command | Source files |
| Generator | Command | Source files (all contribute to the meta stamp SHA) |
|-----------|---------|--------------|
| `generate_brands` | `tooling/generate-brands` | `server/src/bin/generate_brands/main.rs` |
| `import_economics` | `python3 tooling/economy-db/import_economics.py` | `tooling/economy-db/import_economics.py` |
| `import_economics` | `python3 tooling/economy-db/import_economics.py` | `tooling/economy-db/import_economics.py` + the Rust brand binary sources it invokes: `server/src/bin/generate_brands/main.rs`, `server/src/bin/generate_brands/names.rs`, `tooling/generate-brands` |
| `generate_atlas` | `python3 tooling/planet-gen/generate_atlas.py --seed 42` | `tooling/planet-gen/generate_atlas.py` |
`make regen-db` runs all three in the correct order.
`import_economics` shells out to the Rust `generate_brands` binary as its first
step to refresh `wiki/economics/corporations/generated_brands.toml`, then reads
the TOML and imports brand data into the DB. The Rust binary is a subroutine
of the Python importer, not an independent generator — changes to its source
invalidate the `import_economics` meta stamp even though the Python file
itself didn't change.
`make regen-db` runs both in the correct order (economics first, atlas second).
---
@@ -37,16 +43,24 @@ After every successful non-dry-run, each generator writes a row to the `meta` ta
```sql
CREATE TABLE meta (
generator_name TEXT PRIMARY KEY, -- 'import_economics' | 'generate_atlas' | 'generate_brands'
generator_name TEXT PRIMARY KEY, -- 'import_economics' | 'generate_atlas'
schema_version TEXT NOT NULL, -- SHA-1 of server/data/systems-schema.sql at generation time
generator_sha TEXT NOT NULL, -- SHA-1 of the generator source file(s)
generated_at TEXT NOT NULL DEFAULT (datetime('now'))
);
```
The `generator_sha` is the SHA-1 of the generator source file content. If the source
changes and `regen-db` is not re-run, the stamped SHA will differ from the current file
SHA — this is what the pre-push hook detects.
The `generator_sha` is the SHA-1 of the concatenated bytes of the generator's
source files (sorted by path, so order is deterministic). If any source file
changes and `make regen-db` is not re-run, the stamped SHA will differ from the
recomputed current SHA — this is what the pre-push hook detects.
**What's deterministic:** the stored SHA (same sources → same recorded SHA).
**What's NOT deterministic:** the DB binary itself. `meta.generated_at` uses
`datetime('now')`, SQLite `rowid`/`autoincrement` values drift across runs, and
transaction ordering can reshape freelist pages — two consecutive `make regen-db`
calls produce byte-different SQLite files even with identical inputs. This is
fine: the freshness guarantee comes from the stamp, not from bytewise DB equality.
---
@@ -89,7 +103,10 @@ Or use `/pr-push` — it detects stale generator sources and reruns `make regen-
automatically before pushing.
The check script is `tooling/check-systems-db-stamp`. Run it interactively with
`make check-systems-db` or `python3 tooling/check-systems-db-stamp --verbose`.
`make check-systems-db` or `python3 tooling/check-systems-db-stamp --verbose`. The
`GENERATOR_SOURCES` dict at the top of that script is the single registry — when
you add a new generator or source file, update it there and mirror the change in
the `/pr-push` skill's source-file watch list.
---
@@ -108,10 +125,45 @@ Two branches that both commit `server/data/systems.db` changes produce a binary
merge conflict. Git cannot diff or merge binary SQLite files. Sprint 36 hit this
exact class of problem. The meta stamp + pre-push hook is the systematic fix:
- Regeneration is deterministic (same sources → same DB, byte-for-byte)
- The stamp is deterministic (same generator source → same recorded SHA)
- Only one branch modifies generator sources at a time (per team scope rules)
- The pre-push hook is a hard blocker before the binary conflict can land
## The migration escape hatch
The rule above says "never run UPDATE or INSERT directly on systems.db outside
of a migration." Here's what a legitimate migration looks like, and what isn't
one:
**Sanctioned path: the `MIGRATION_SQL` block in `import_economics.py`.** That
string is executed at the top of every import run (inside the same transaction
that clears + reimports data) and contains idempotent `CREATE TABLE IF NOT
EXISTS` / `CREATE INDEX IF NOT EXISTS` statements, plus `ALTER TABLE` additions
handled via the `COLUMN_MIGRATIONS` list. When you need a new table, column,
or index on systems.db, add it there. It'll run on the next `make regen-db`
and the meta stamp will flip because `import_economics.py` changed.
**Also legitimate:** edits to `server/data/systems-schema.sql` (the canonical
DDL used by fresh builds) paired with matching entries in `MIGRATION_SQL` for
existing DBs. The stamp's `schema_version` field records the schema file's
SHA at generation time — change the schema, commit both files together, and
the stamp picks it up automatically.
**NOT legitimate and forbidden:**
- Running `tooling/db/sqlite-exec` (or any raw SQL) against `systems.db` by
hand. Any changes you make are silently reverted by the next `regen-db` run
— your edits die, not the pipeline's.
- One-off patch scripts that open `systems.db` and modify rows.
- Editing the DB file with a SQLite GUI.
- Committing `systems.db` alone, without the corresponding source change that
would explain the diff on regen.
If you think you need an exception, the right move is to make the source
change explicit instead: either edit the wiki TOMLs / JSONs that feed the
generators, or edit `MIGRATION_SQL` / `systems-schema.sql` directly. There is
no hand-edit path that survives regen.
---
## Future: savegame migration lineage
+7 -1
View File
@@ -218,13 +218,19 @@ If clean, continue.
### 4a. Regen systems.db if generator sources or data changed (#858)
Check whether any file in the **source-file watch list** was modified on this branch
versus `origin/main`. This list covers generator code AND the data files that feed them:
versus `origin/main`. This list covers generator code AND the data files that feed them.
The generator-source paths below **must stay in sync** with `GENERATOR_SOURCES` in
`tooling/check-systems-db-stamp` (PR #136 review T7) — if you add a new source file
to the stamp, add it here too, and vice versa. Drift between the two lists reintroduces
exactly the silent-stale-DB class of bug this skill exists to prevent.
```bash
git diff --name-only origin/main...HEAD -- \
tooling/economy-db/import_economics.py \
tooling/planet-gen/generate_atlas.py \
server/src/bin/generate_brands/main.rs \
server/src/bin/generate_brands/names.rs \
tooling/generate-brands \
server/data/systems-schema.sql \
wiki/star-systems/ \