chore(ci): merge brand pipeline into import_economics + harden review findings
Addresses all blocking + minor items from PR #136 review. Architectural change (T2/H3 — the review's main complaint): generate_brands was previously a separate Rust binary that produced a TOML artifact, with its stamp written "on behalf" by import_economics.py at the end of its own run. Reviewers flagged the invisible coupling: two sources of truth in a system designed to have one, and no way to tell from the stamp that one "generator" was really a subroutine of the other. import_economics now invokes tooling/generate-brands as the first step of its main() flow, before opening its own DB connection. The TOML artefact is still produced and still committed (useful for diff-review of brand changes), but there's now one pipeline owner. The meta table carries two rows (import_economics, generate_atlas) not three; the Rust binary's source SHA folds into import_economics' stamp via IMPORT_ECONOMICS_SOURCES. A MIGRATION_SQL DELETE cleans up pre-merge DBs that still have the orphan generate_brands row. Other review items addressed in-line: H1 generate_atlas._write_stamp no longer commits — transaction ownership stays with the caller (matches import_economics pattern). Stamp + atlas data now commit atomically; a failed stamp rolls back the atlas data rather than leaving a stamp-missing-data intermediate. H2 _file_sha1 (in both import_economics, generate_atlas, check-systems-db-stamp) raises FileNotFoundError on missing sources instead of silently contributing an empty-bytes hash. A ghost-SHA convergence could otherwise produce vacuous "fresh" passes. H4 pre-push no-meta-table warning rephrased — was "run after next regeneration", now "run now if this DB was generated by you". T1 asset-pipeline.md determinism claim softened: the stamp is deterministic (same source → same recorded SHA), the DB binary is not (generated_at + SQLite rowids/freelist churn). T3 asset-pipeline.md gains a "migration escape hatch" section naming MIGRATION_SQL in import_economics.py as the only sanctioned path for direct writes, and forbidding hand-run sqlite-exec / one-off patch scripts / SQLite-GUI edits. T4 Makefile regen-db now runs as a single shell with `set -e`. A failure in one generator halts the pipeline immediately, preventing the "stale data, fresh stamp" state where a later step stamped a DB whose earlier step had failed. import_economics' exit code 2 (coverage gate warning) remains explicitly tolerated. T5 pre-push stamp check now runs on a branch's first push too — compares against origin/main instead of origin/$BRANCH, closing the gap where a new branch could ship a stale DB via the first push. T6 check-systems-db-stamp fails closed on unknown generator_names in meta — a future branch adding a new generator without registering it in GENERATOR_SOURCES will now be rejected, not silently skipped. T7 /pr-push watch list gains a mutual cross-reference comment with GENERATOR_SOURCES in check-systems-db-stamp, plus the missing names.rs source file, so the two lists cannot silently drift. Follow-up tickets created: #887 T8 decisions-orphan-tickets CLI — surfaces tickets whose decision_ref points at a non-existent D-record. #888 T9 meta.schema_version monotonic semver — for savegame migration lineage in Phase 5+ (SHA comparison can't be ordered). Verified: make regen-db end-to-end — OK make check-systems-db — OK, 2 generator(s) up to date STALE detection — OK, verified by touching generate_atlas.py /pr-push watch list — OK, flags this branch's changed sources decision show D-159 — OK, structured output with tickets + refs Refs: #855 #856 #857 #858 #859 PR #136 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -19,15 +19,21 @@ migration — those changes will be silently overwritten by the next `make regen
|
||||
|
||||
## What produces systems.db
|
||||
|
||||
Three generators write to `systems.db` in sequence:
|
||||
Two generators write to `systems.db`:
|
||||
|
||||
| Generator | Command | Source files |
|
||||
| Generator | Command | Source files (all contribute to the meta stamp SHA) |
|
||||
|-----------|---------|--------------|
|
||||
| `generate_brands` | `tooling/generate-brands` | `server/src/bin/generate_brands/main.rs` |
|
||||
| `import_economics` | `python3 tooling/economy-db/import_economics.py` | `tooling/economy-db/import_economics.py` |
|
||||
| `import_economics` | `python3 tooling/economy-db/import_economics.py` | `tooling/economy-db/import_economics.py` + the Rust brand binary sources it invokes: `server/src/bin/generate_brands/main.rs`, `server/src/bin/generate_brands/names.rs`, `tooling/generate-brands` |
|
||||
| `generate_atlas` | `python3 tooling/planet-gen/generate_atlas.py --seed 42` | `tooling/planet-gen/generate_atlas.py` |
|
||||
|
||||
`make regen-db` runs all three in the correct order.
|
||||
`import_economics` shells out to the Rust `generate_brands` binary as its first
|
||||
step to refresh `wiki/economics/corporations/generated_brands.toml`, then reads
|
||||
the TOML and imports brand data into the DB. The Rust binary is a subroutine
|
||||
of the Python importer, not an independent generator — changes to its source
|
||||
invalidate the `import_economics` meta stamp even though the Python file
|
||||
itself didn't change.
|
||||
|
||||
`make regen-db` runs both in the correct order (economics first, atlas second).
|
||||
|
||||
---
|
||||
|
||||
@@ -37,16 +43,24 @@ After every successful non-dry-run, each generator writes a row to the `meta` ta
|
||||
|
||||
```sql
|
||||
CREATE TABLE meta (
|
||||
generator_name TEXT PRIMARY KEY, -- 'import_economics' | 'generate_atlas' | 'generate_brands'
|
||||
generator_name TEXT PRIMARY KEY, -- 'import_economics' | 'generate_atlas'
|
||||
schema_version TEXT NOT NULL, -- SHA-1 of server/data/systems-schema.sql at generation time
|
||||
generator_sha TEXT NOT NULL, -- SHA-1 of the generator source file(s)
|
||||
generated_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
);
|
||||
```
|
||||
|
||||
The `generator_sha` is the SHA-1 of the generator source file content. If the source
|
||||
changes and `regen-db` is not re-run, the stamped SHA will differ from the current file
|
||||
SHA — this is what the pre-push hook detects.
|
||||
The `generator_sha` is the SHA-1 of the concatenated bytes of the generator's
|
||||
source files (sorted by path, so order is deterministic). If any source file
|
||||
changes and `make regen-db` is not re-run, the stamped SHA will differ from the
|
||||
recomputed current SHA — this is what the pre-push hook detects.
|
||||
|
||||
**What's deterministic:** the stored SHA (same sources → same recorded SHA).
|
||||
**What's NOT deterministic:** the DB binary itself. `meta.generated_at` uses
|
||||
`datetime('now')`, SQLite `rowid`/`autoincrement` values drift across runs, and
|
||||
transaction ordering can reshape freelist pages — two consecutive `make regen-db`
|
||||
calls produce byte-different SQLite files even with identical inputs. This is
|
||||
fine: the freshness guarantee comes from the stamp, not from bytewise DB equality.
|
||||
|
||||
---
|
||||
|
||||
@@ -89,7 +103,10 @@ Or use `/pr-push` — it detects stale generator sources and reruns `make regen-
|
||||
automatically before pushing.
|
||||
|
||||
The check script is `tooling/check-systems-db-stamp`. Run it interactively with
|
||||
`make check-systems-db` or `python3 tooling/check-systems-db-stamp --verbose`.
|
||||
`make check-systems-db` or `python3 tooling/check-systems-db-stamp --verbose`. The
|
||||
`GENERATOR_SOURCES` dict at the top of that script is the single registry — when
|
||||
you add a new generator or source file, update it there and mirror the change in
|
||||
the `/pr-push` skill's source-file watch list.
|
||||
|
||||
---
|
||||
|
||||
@@ -108,10 +125,45 @@ Two branches that both commit `server/data/systems.db` changes produce a binary
|
||||
merge conflict. Git cannot diff or merge binary SQLite files. Sprint 36 hit this
|
||||
exact class of problem. The meta stamp + pre-push hook is the systematic fix:
|
||||
|
||||
- Regeneration is deterministic (same sources → same DB, byte-for-byte)
|
||||
- The stamp is deterministic (same generator source → same recorded SHA)
|
||||
- Only one branch modifies generator sources at a time (per team scope rules)
|
||||
- The pre-push hook is a hard blocker before the binary conflict can land
|
||||
|
||||
## The migration escape hatch
|
||||
|
||||
The rule above says "never run UPDATE or INSERT directly on systems.db outside
|
||||
of a migration." Here's what a legitimate migration looks like, and what isn't
|
||||
one:
|
||||
|
||||
**Sanctioned path: the `MIGRATION_SQL` block in `import_economics.py`.** That
|
||||
string is executed at the top of every import run (inside the same transaction
|
||||
that clears + reimports data) and contains idempotent `CREATE TABLE IF NOT
|
||||
EXISTS` / `CREATE INDEX IF NOT EXISTS` statements, plus `ALTER TABLE` additions
|
||||
handled via the `COLUMN_MIGRATIONS` list. When you need a new table, column,
|
||||
or index on systems.db, add it there. It'll run on the next `make regen-db`
|
||||
and the meta stamp will flip because `import_economics.py` changed.
|
||||
|
||||
**Also legitimate:** edits to `server/data/systems-schema.sql` (the canonical
|
||||
DDL used by fresh builds) paired with matching entries in `MIGRATION_SQL` for
|
||||
existing DBs. The stamp's `schema_version` field records the schema file's
|
||||
SHA at generation time — change the schema, commit both files together, and
|
||||
the stamp picks it up automatically.
|
||||
|
||||
**NOT legitimate and forbidden:**
|
||||
|
||||
- Running `tooling/db/sqlite-exec` (or any raw SQL) against `systems.db` by
|
||||
hand. Any changes you make are silently reverted by the next `regen-db` run
|
||||
— your edits die, not the pipeline's.
|
||||
- One-off patch scripts that open `systems.db` and modify rows.
|
||||
- Editing the DB file with a SQLite GUI.
|
||||
- Committing `systems.db` alone, without the corresponding source change that
|
||||
would explain the diff on regen.
|
||||
|
||||
If you think you need an exception, the right move is to make the source
|
||||
change explicit instead: either edit the wiki TOMLs / JSONs that feed the
|
||||
generators, or edit `MIGRATION_SQL` / `systems-schema.sql` directly. There is
|
||||
no hand-edit path that survives regen.
|
||||
|
||||
---
|
||||
|
||||
## Future: savegame migration lineage
|
||||
|
||||
@@ -218,13 +218,19 @@ If clean, continue.
|
||||
### 4a. Regen systems.db if generator sources or data changed (#858)
|
||||
|
||||
Check whether any file in the **source-file watch list** was modified on this branch
|
||||
versus `origin/main`. This list covers generator code AND the data files that feed them:
|
||||
versus `origin/main`. This list covers generator code AND the data files that feed them.
|
||||
|
||||
The generator-source paths below **must stay in sync** with `GENERATOR_SOURCES` in
|
||||
`tooling/check-systems-db-stamp` (PR #136 review T7) — if you add a new source file
|
||||
to the stamp, add it here too, and vice versa. Drift between the two lists reintroduces
|
||||
exactly the silent-stale-DB class of bug this skill exists to prevent.
|
||||
|
||||
```bash
|
||||
git diff --name-only origin/main...HEAD -- \
|
||||
tooling/economy-db/import_economics.py \
|
||||
tooling/planet-gen/generate_atlas.py \
|
||||
server/src/bin/generate_brands/main.rs \
|
||||
server/src/bin/generate_brands/names.rs \
|
||||
tooling/generate-brands \
|
||||
server/data/systems-schema.sql \
|
||||
wiki/star-systems/ \
|
||||
|
||||
Reference in New Issue
Block a user