jpmschweitzer/settled-reach
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(tooling): T-1289 — economy-db becomes reach ledger

Browse Source 
The sole generator of systems.db moves to tooling/domains/ledger/ and is
now `reach ledger import`. economy_import/ keeps its name (Rust comments in
server/src cite it); the entrypoint becomes service.py; schema_version.py
moves with the importer, which is where the version is defined.

The stamp survived the move, which is the thing that had to hold:

- generated_brands.toml is byte-identical (sha256 e748531…) before and after
- `reach check systems-db-stamp` reported STALE after the move (the registry
  saw it) and OK after the regen
- the dry-run carries every count and warning of the baseline transcript,
  and exit 2 — imported and stamped, coverage gate unmet — still reaches the
  caller through @command

`make regen-db` survives as a one-line delegate, per D-263's muscle-memory
clause: about fifty files name it, including the headers of generated wiki
TOMLs and the remedies the push gate prints. `make economy-db` is retired;
it ran `reach generate brands` before the import, which the import already
does as its first step.

economy_import.errors is reconciled as DOMAINS.md asked. ImportAborted stays
as internal rollback control flow and never reaches a caller; the service
converts it to a ReachError carrying the remedy.

regenerate_brands caught cargo_binary's ReachError, printed it and raised
ImportAborted, dropping the remedy. It runs before the import transaction
opens, so there is nothing to roll back — it now propagates.

Both sys.path bootstraps are gone; they existed only because the directory
was hyphenated. The step labels ran [1/10]…[10/13]…[17/19]; one 24-step
counter now drives the event phase and progress.

Stale pointers fixed on the way: MIGRATION_SQL has lived in
economy_import/migration.py since T-1067, but the asset-pipeline rule,
DEVOPS and the schema comments still sent readers to import_economics.py;
the rule and DEVOPS also still named the check scripts T-1281 retired.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
jpmschweitzer
2026-09-23 16:18:59 +02:00
co-authored by Claude Opus 5.5
parent 668772075c
commit 23538d640f
35 changed files with 720 additions and 572 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/rules/asset-pipeline.md
+11 -11
View File
@@ -21,7 +21,7 @@ migration — those changes will be silently overwritten by the next `make regen
## What produces systems.db
**One generator** writes to `systems.db`: `import_economics`
(`python3 tooling/economy-db/import_economics.py`, run via `make regen-db`).
(`reach ledger import`, run via `make regen-db`).
The former atlas geometry generator (`generate_atlas`) was retired in #951
(D-223); `import_economics` now also owns the atlas index — it loads the
names-only `markers.json` city pool into `atlas_city_names` and empties the
@@ -36,9 +36,9 @@ itself didn't change.
The full set of source files contributing to the meta stamp SHA (the Python
importer entrypoint + its `economy_import` module package, the Rust brand
binary sources, `tooling/schema_version.py`, the authored economics TOMLs, and
binary sources, `tooling/domains/ledger/schema_version.py`, the authored economics TOMLs, and
the registry itself) is defined once in `tooling/generator_sources.py` (T-1067)
— imported by both the importer's stamp writer and `check-systems-db-stamp`,
— imported by both the importer's stamp writer and `reach check systems-db-stamp`,
and listed via `python3 tooling/generator_sources.py --list`.
The surviving `reach atlas planet` importers (`import-heightmaps`,
@@ -62,7 +62,7 @@ CREATE TABLE meta (
```
`schema_version` is a **monotonic semver string** (e.g. `"1.0.0"`), not a hash.
It is defined as the `SCHEMA_VERSION` constant in `tooling/schema_version.py`
It is defined as the `SCHEMA_VERSION` constant in `tooling/domains/ledger/schema_version.py`
and must be bumped manually whenever the schema changes in a backwards-incompatible way.
Unlike a SHA-1 hash, semver strings are orderable — this enables savegame migration
lineage in Phase 5+: a save file can record which schema version it derives from and
@@ -89,7 +89,7 @@ fine: the freshness guarantee comes from the stamp, not from bytewise DB equalit
1. Edit the source files (TOML, JSON, markers.json).
2. Run `make regen-db`.
3. Run `make check-systems-db` to confirm the stamp is fresh.
3. Run `reach check systems-db-stamp` to confirm the stamp is fresh.
4. Stage and commit:
```bash
git add server/data/systems.db
@@ -99,7 +99,7 @@ fine: the freshness guarantee comes from the stamp, not from bytewise DB equalit
### Schema changes (new tables or columns)
1. Add the DDL to `server/data/systems-schema.sql`.
2. Add migration SQL to `MIGRATION_SQL` in `import_economics.py` if the change
2. Add migration SQL to `MIGRATION_SQL` in `economy_import/migration.py` if the change
affects existing DBs (idempotent `CREATE TABLE IF NOT EXISTS` or `ALTER TABLE`).
3. Run `make regen-db`.
4. Stage `server/data/systems-schema.sql` and `server/data/systems.db` together.
@@ -121,8 +121,8 @@ Fix: run `make regen-db`, stage `server/data/systems.db`, amend or add a commit.
Or use `/pr-process` — it detects stale generator sources and reruns `make regen-db`
automatically before pushing.
The check script is `tooling/check-systems-db-stamp`. Run it interactively with
`make check-systems-db` or `python3 tooling/check-systems-db-stamp --verbose`. The
The check is `reach check systems-db-stamp` (`tooling/domains/check/`), run by
the push gate and by hand the same way. The
`GENERATOR_SOURCES` dict in `tooling/generator_sources.py` is the single registry
(T-1067) — the check script and the importer's stamp writer both import it, and
the `/pr-process` skill derives its source-file watch list from
@@ -156,13 +156,13 @@ The rule above says "never run UPDATE or INSERT directly on systems.db outside
of a migration." Here's what a legitimate migration looks like, and what isn't
one:
**Sanctioned path: the `MIGRATION_SQL` block in `import_economics.py`.** That
**Sanctioned path: the `MIGRATION_SQL` block in `tooling/domains/ledger/economy_import/migration.py`.** That
string is executed at the top of every import run (inside the same transaction
that clears + reimports data) and contains idempotent `CREATE TABLE IF NOT
EXISTS` / `CREATE INDEX IF NOT EXISTS` statements, plus `ALTER TABLE` additions
handled via the `COLUMN_MIGRATIONS` list. When you need a new table, column,
or index on systems.db, add it there. It'll run on the next `make regen-db`
and the meta stamp will flip because `import_economics.py` changed.
and the meta stamp will flip because `migration.py` is a stamped source.
**Also legitimate:** edits to `server/data/systems-schema.sql` (the canonical
DDL used by fresh builds) paired with matching entries in `MIGRATION_SQL` for
@@ -195,6 +195,6 @@ loader can determine which migrations to apply by comparing that version to the
current one. `meta.schema_sha` retains the old SHA-1 for tamper detection.
**When to bump `SCHEMA_VERSION`:** edit the `SCHEMA_VERSION = "1.0.0"` constant in
`tooling/schema_version.py` whenever a schema change is backwards-incompatible
`tooling/domains/ledger/schema_version.py` whenever a schema change is backwards-incompatible
(column removed, type changed, FK constraint added, table dropped). Additive changes
(new nullable columns, new tables, new indexes) do not require a bump.
.claude/skills/atlas/SKILL.md
+1 -1
View File
@@ -217,7 +217,7 @@ See `.claude/rules/asset-pipeline.md` (the "Golden Rule": edit sources, not
`systems.db`) and D-223. The bodies/stations catalog this skill authors is
**atlas-CLI-owned** and survives `make regen-db`: `import_economics` only
*enriches* existing body rows (radius, axial tilt, biosphere class — see
`tooling/economy-db/economy_import/bodies.py`), it never deletes or
`tooling/domains/ledger/economy_import/bodies.py`), it never deletes or
regenerates the catalog. What `import_economics` *does* clear on every regen
is the separate `atlas_*` geometry index tables (`atlas_cities`,
`atlas_roads`, etc., D-223) — those hold cascade-computed geometry, not the
.gitignore
+1 -1
View File
@@ -100,7 +100,7 @@ Thumbs.db
*.swp
*.swo
# Generated economics pipeline artifacts (re-created by make economy-db)
# Generated economics pipeline artifacts (re-created by make regen-db)
wiki/economics/corporations/generated_brands.toml
wiki/economics/corporations/generated_corporations.toml
.pql/changelog/ticket_history/2026-09.sql
+58
View File
@@ -345,3 +345,61 @@ nothing prints but `console`, no `subprocess` outside `core/process`.
DONE 2026-09-23. reach atlas planet: ten verbs over the moved package; tooling/test_planet_router.py (in make test-tooling) fails if a router option drifts from its module''s argparse. Finishing the 2026-09-02 half-move found: (1) lazy in-function imports and all of sol_data/ still used bare sibling names that only resolved via sys.path hacks — generate/batch/sol-import would have failed on first globe render under reach; qualified, hacks removed. (2) 247 print() + a per-8KB-block stdout progress writer routed through console (report verbs -> out, progress -> event, throttled download progress). (3) Every error exit now raises ReachError with fix=. (4) TWO CHECKS THAT COULD NOT FAIL: batch --verify-determinism and import-provinces both exited 0 on detected failures; both now raise. (5) sol-import --body is action=append but the router took one value — now repeatable. (6) test_conformance only walked one level, so the nested group''s ten verbs were never checked; now recursive, proven against a mutant. Stray PNGs from the 2026-09-03 runaway test run (8 modified reliefmaps, 6 new heightmaps) parked in .cache/t1288-stray-pngs/, reliefmaps restored from HEAD. NOTE: the reliefmaps differed from the committed bake, i.e. the current renderer no longer reproduces committed reliefmap bytes — not investigated here.', NULL, '2026-09-23 14:07:42', '2026-09-23 14:07:42.675', '2026-09-23 14:07:42.675', NULL, 'b6c6d1e211eada353630e3946da96310', 2) ON CONFLICT(hash) DO NOTHING;
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G65NSJ4RG6SJ12TQE78S6TKR', 'status', 'backlog', 'done', NULL, '2026-09-23 14:07:43', '2026-09-23 14:07:43.036', '2026-09-23 14:07:43.036', NULL, 'e3329ba1f64d0a63c635d4a9049d56b6', 2) ON CONFLICT(hash) DO NOTHING;
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G65T2N1WAT491WY8SV736J3G', 'description', '`tooling/economy-db/` (17 files) + `schema_version.py` become `reach ledger`.
The name is deliberate and is the user''s call (D-263 amendment): reach verbs
mirror the game''s implant-app presentation patterns. `ledger` is the player-facing
component that will aggregate economics — browse markets, wealth, transactions —
the way `atlas` aggregates topography. The tooling verb takes the same name so the
CLI and the UI do not diverge into two vocabularies for one subject.
THIS IS THE HIGHEST-RISK PORT IN THE EPIC and the reason E5 (T-1252) exists as a
separate epic: `import_economics` is the SOLE generator of `server/data/systems.db`,
and its meta stamp is computed over a source-file registry. Two things bite:
1. `tooling/generator_sources.py` lists every stamped path. Moving a file without
updating the registry either breaks the import (missing path) or silently
changes the SHA. T-1286 already had to swap the retired `generate-brands`
wrapper for `core/process.py` there and regen. Expect the same.
2. `economy-db` is hyphenated, so it is NOT importable as a package. `brands.py`
currently bootstraps `sys.path` with REPO_ROOT to reach `tooling.core` — see
the comment there. That bootstrap should DISAPPEAR in this port, not be
copied; it exists only because the directory could not be imported. Overlaps
T-1272 (hyphen sweep) — do them together or in that order.
Every port of this domain ends with `make regen-db` + `reach check systems-db-stamp`
+ a commit of `server/data/systems.db`. `generated_brands.toml` must come back
byte-identical; if it does not, the port changed generator behaviour and that is
a bug, not a regen.
Standard port acceptance as on T-1281/T-1286.', '`tooling/economy-db/` (17 files) + `schema_version.py` become `reach ledger`.
The name is deliberate and is the user''s call (D-263 amendment): reach verbs
mirror the game''s implant-app presentation patterns. `ledger` is the player-facing
component that will aggregate economics — browse markets, wealth, transactions —
the way `atlas` aggregates topography. The tooling verb takes the same name so the
CLI and the UI do not diverge into two vocabularies for one subject.
THIS IS THE HIGHEST-RISK PORT IN THE EPIC and the reason E5 (T-1252) exists as a
separate epic: `import_economics` is the SOLE generator of `server/data/systems.db`,
and its meta stamp is computed over a source-file registry. Two things bite:
1. `tooling/generator_sources.py` lists every stamped path. Moving a file without
updating the registry either breaks the import (missing path) or silently
changes the SHA. T-1286 already had to swap the retired `generate-brands`
wrapper for `core/process.py` there and regen. Expect the same.
2. `economy-db` is hyphenated, so it is NOT importable as a package. `brands.py`
currently bootstraps `sys.path` with REPO_ROOT to reach `tooling.core` — see
the comment there. That bootstrap should DISAPPEAR in this port, not be
copied; it exists only because the directory could not be imported. Overlaps
T-1272 (hyphen sweep) — do them together or in that order.
Every port of this domain ends with `make regen-db` + `reach check systems-db-stamp`
+ a commit of `server/data/systems.db`. `generated_brands.toml` must come back
byte-identical; if it does not, the port changed generator behaviour and that is
a bug, not a regen.
Standard port acceptance as on T-1281/T-1286.
DONE 2026-09-23. tooling/economy-db/ + schema_version.py -> tooling/domains/ledger/ (economy_import/ kept by name, since server/src Rust comments cite economy_import/*.py; import_economics.py became service.py). reach ledger import [--db] [--dry-run] [--strict-specialization]. Evidence: generated_brands.toml sha256 identical before/after (e748531…); stamp gate reported STALE after the move and OK after regen; dry-run transcript carries every count and warning of the baseline; exit 2 (coverage gap, data committed) proven to relay through @command with a probe. Make: regen-db kept as a one-line delegate per D-263''s muscle-memory clause (~50 files name it, incl. generated TOML headers and gate remedies); economy-db retired (it ran generate brands, which the import already does first). Reconciled economy_import.errors (DOMAINS.md ask): ImportAborted stays internal rollback control flow, converted to ReachError at the service boundary. Also: regenerate_brands used to swallow cargo_binary''s ReachError into ImportAborted, dropping its remedy — now propagates (it runs before the transaction, nothing to roll back). Both sys.path bootstraps removed. Step labels were inconsistent ([1/10]..[10/13]..[17/19]); one 24-step counter now drives phase + progress. Stale doc pointers fixed while there: MIGRATION_SQL had lived in economy_import/migration.py since T-1067 but rules/DEVOPS still said import_economics.py; asset-pipeline rule and DEVOPS still named the check-systems-db script/target T-1281 retired.', NULL, '2026-09-23 14:18:49', '2026-09-23 14:18:49.681', '2026-09-23 14:18:49.681', NULL, '13300d7ea5bd79ae4857cd59399531a6', 2) ON CONFLICT(hash) DO NOTHING;
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G65T2N1WAT491WY8SV736J3G', 'status', 'backlog', 'done', NULL, '2026-09-23 14:18:50', '2026-09-23 14:18:50.093', '2026-09-23 14:18:50.093', NULL, '63d24743b229b1544d62104c4e68ca66', 2) ON CONFLICT(hash) DO NOTHING;
.pql/changelog/tickets/2026-09.sql
+60
View File
@@ -501,3 +501,63 @@ nothing prints but `console`, no `subprocess` outside `core/process`.
DONE 2026-09-23. reach atlas planet: ten verbs over the moved package; tooling/test_planet_router.py (in make test-tooling) fails if a router option drifts from its module''s argparse. Finishing the 2026-09-02 half-move found: (1) lazy in-function imports and all of sol_data/ still used bare sibling names that only resolved via sys.path hacks — generate/batch/sol-import would have failed on first globe render under reach; qualified, hacks removed. (2) 247 print() + a per-8KB-block stdout progress writer routed through console (report verbs -> out, progress -> event, throttled download progress). (3) Every error exit now raises ReachError with fix=. (4) TWO CHECKS THAT COULD NOT FAIL: batch --verify-determinism and import-provinces both exited 0 on detected failures; both now raise. (5) sol-import --body is action=append but the router took one value — now repeatable. (6) test_conformance only walked one level, so the nested group''s ten verbs were never checked; now recursive, proven against a mutant. Stray PNGs from the 2026-09-03 runaway test run (8 modified reliefmaps, 6 new heightmaps) parked in .cache/t1288-stray-pngs/, reliefmaps restored from HEAD. NOTE: the reliefmaps differed from the committed bake, i.e. the current renderer no longer reproduces committed reliefmap bytes — not investigated here.', 'done', 'high', NULL, 'tooling', 'D-263', '2026-09-02 15:57:45.382', '2026-09-23 14:07:43.036', NULL, '2a78f8f8646d915600ed55c5dc5d0993', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06GCX60QRMD7KJ1TVWTR7FWK28', 'bug', '06FBPPMZNNEV052DBYYY3A897C', 'Planet renderer no longer reproduces committed reliefmap.png bytes (8 bodies differ on re-render)', 'Found during T-1288. A 2026-09-03 accidental generate run re-rendered bodies in GJ-1002, GJ-1116B, GJ-139, GJ-147, GJ-216A, GJ-234A, GJ-273, GJ-282B: every reliefmap.png came back a few hundred bytes different from HEAD, while GJ1002b''s heightmap.png came back byte-identical (so the simulation is deterministic — make test-tooling confirms — and the drift is in the reliefmap RENDER path or its PNG encoding). Copies of the re-rendered files are in .cache/t1288-stray-pngs/. Question to answer first: is this a visible change (renderer code evolved since the April bake) or encoder-level noise (zlib/Pillow version, metadata)? Pixel-diff one pair before anything else. If visible: decide whether committed reliefmaps should be re-baked. If noise: make the encode deterministic so a re-render is a no-op in git.', 'backlog', 'low', NULL, 'tooling', NULL, '2026-09-23 14:07:49.189', '2026-09-23 14:07:49.189', NULL, '215478973901c77cdd0e6b5c446ff19e', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G65T2N1WAT491WY8SV736J3G', 'task', '06G1S3D0M1TQW0GMFBBPQZG3ZM', 'Port ledger — the economics pipeline under its UI name', '`tooling/economy-db/` (17 files) + `schema_version.py` become `reach ledger`.
The name is deliberate and is the user''s call (D-263 amendment): reach verbs
mirror the game''s implant-app presentation patterns. `ledger` is the player-facing
component that will aggregate economics — browse markets, wealth, transactions —
the way `atlas` aggregates topography. The tooling verb takes the same name so the
CLI and the UI do not diverge into two vocabularies for one subject.
THIS IS THE HIGHEST-RISK PORT IN THE EPIC and the reason E5 (T-1252) exists as a
separate epic: `import_economics` is the SOLE generator of `server/data/systems.db`,
and its meta stamp is computed over a source-file registry. Two things bite:
1. `tooling/generator_sources.py` lists every stamped path. Moving a file without
updating the registry either breaks the import (missing path) or silently
changes the SHA. T-1286 already had to swap the retired `generate-brands`
wrapper for `core/process.py` there and regen. Expect the same.
2. `economy-db` is hyphenated, so it is NOT importable as a package. `brands.py`
currently bootstraps `sys.path` with REPO_ROOT to reach `tooling.core` — see
the comment there. That bootstrap should DISAPPEAR in this port, not be
copied; it exists only because the directory could not be imported. Overlaps
T-1272 (hyphen sweep) — do them together or in that order.
Every port of this domain ends with `make regen-db` + `reach check systems-db-stamp`
+ a commit of `server/data/systems.db`. `generated_brands.toml` must come back
byte-identical; if it does not, the port changed generator behaviour and that is
a bug, not a regen.
Standard port acceptance as on T-1281/T-1286.
DONE 2026-09-23. tooling/economy-db/ + schema_version.py -> tooling/domains/ledger/ (economy_import/ kept by name, since server/src Rust comments cite economy_import/*.py; import_economics.py became service.py). reach ledger import [--db] [--dry-run] [--strict-specialization]. Evidence: generated_brands.toml sha256 identical before/after (e748531…); stamp gate reported STALE after the move and OK after regen; dry-run transcript carries every count and warning of the baseline; exit 2 (coverage gap, data committed) proven to relay through @command with a probe. Make: regen-db kept as a one-line delegate per D-263''s muscle-memory clause (~50 files name it, incl. generated TOML headers and gate remedies); economy-db retired (it ran generate brands, which the import already does first). Reconciled economy_import.errors (DOMAINS.md ask): ImportAborted stays internal rollback control flow, converted to ReachError at the service boundary. Also: regenerate_brands used to swallow cargo_binary''s ReachError into ImportAborted, dropping its remedy — now propagates (it runs before the transaction, nothing to roll back). Both sys.path bootstraps removed. Step labels were inconsistent ([1/10]..[10/13]..[17/19]); one 24-step counter now drives phase + progress. Stale doc pointers fixed while there: MIGRATION_SQL had lived in economy_import/migration.py since T-1067 but rules/DEVOPS still said import_economics.py; asset-pipeline rule and DEVOPS still named the check-systems-db script/target T-1281 retired.', 'backlog', 'high', NULL, 'tooling', 'D-263', '2026-09-02 16:16:28.431', '2026-09-23 14:18:49.681', NULL, '4690f4d637a68bb49840c7a2de30801a', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G65T2N1WAT491WY8SV736J3G', 'task', '06G1S3D0M1TQW0GMFBBPQZG3ZM', 'Port ledger — the economics pipeline under its UI name', '`tooling/economy-db/` (17 files) + `schema_version.py` become `reach ledger`.
The name is deliberate and is the user''s call (D-263 amendment): reach verbs
mirror the game''s implant-app presentation patterns. `ledger` is the player-facing
component that will aggregate economics — browse markets, wealth, transactions —
the way `atlas` aggregates topography. The tooling verb takes the same name so the
CLI and the UI do not diverge into two vocabularies for one subject.
THIS IS THE HIGHEST-RISK PORT IN THE EPIC and the reason E5 (T-1252) exists as a
separate epic: `import_economics` is the SOLE generator of `server/data/systems.db`,
and its meta stamp is computed over a source-file registry. Two things bite:
1. `tooling/generator_sources.py` lists every stamped path. Moving a file without
updating the registry either breaks the import (missing path) or silently
changes the SHA. T-1286 already had to swap the retired `generate-brands`
wrapper for `core/process.py` there and regen. Expect the same.
2. `economy-db` is hyphenated, so it is NOT importable as a package. `brands.py`
currently bootstraps `sys.path` with REPO_ROOT to reach `tooling.core` — see
the comment there. That bootstrap should DISAPPEAR in this port, not be
copied; it exists only because the directory could not be imported. Overlaps
T-1272 (hyphen sweep) — do them together or in that order.
Every port of this domain ends with `make regen-db` + `reach check systems-db-stamp`
+ a commit of `server/data/systems.db`. `generated_brands.toml` must come back
byte-identical; if it does not, the port changed generator behaviour and that is
a bug, not a regen.
Standard port acceptance as on T-1281/T-1286.
DONE 2026-09-23. tooling/economy-db/ + schema_version.py -> tooling/domains/ledger/ (economy_import/ kept by name, since server/src Rust comments cite economy_import/*.py; import_economics.py became service.py). reach ledger import [--db] [--dry-run] [--strict-specialization]. Evidence: generated_brands.toml sha256 identical before/after (e748531…); stamp gate reported STALE after the move and OK after regen; dry-run transcript carries every count and warning of the baseline; exit 2 (coverage gap, data committed) proven to relay through @command with a probe. Make: regen-db kept as a one-line delegate per D-263''s muscle-memory clause (~50 files name it, incl. generated TOML headers and gate remedies); economy-db retired (it ran generate brands, which the import already does first). Reconciled economy_import.errors (DOMAINS.md ask): ImportAborted stays internal rollback control flow, converted to ReachError at the service boundary. Also: regenerate_brands used to swallow cargo_binary''s ReachError into ImportAborted, dropping its remedy — now propagates (it runs before the transaction, nothing to roll back). Both sys.path bootstraps removed. Step labels were inconsistent ([1/10]..[10/13]..[17/19]); one 24-step counter now drives phase + progress. Stale doc pointers fixed while there: MIGRATION_SQL had lived in economy_import/migration.py since T-1067 but rules/DEVOPS still said import_economics.py; asset-pipeline rule and DEVOPS still named the check-systems-db script/target T-1281 retired.', 'done', 'high', NULL, 'tooling', 'D-263', '2026-09-02 16:16:28.431', '2026-09-23 14:18:50.093', NULL, '579a9be46ac9f07e3b7a687fbe4772fe', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
Makefile
+20 -34
View File
@@ -4,7 +4,7 @@ GODOT := $(shell command -v godot4 2>/dev/null || command -v godot 2>/dev/null)
install-reach reach-repoint \
decisions-sync decisions-active decisions-validate \
setup-hooks install-hooks \
audit deny economy-db regen-db \
audit deny regen-db \
pre-pr pre-pr-lint pre-pr-build pre-pr-test pre-pr-validate pre-pr-fixtures \
pre-pr-server pre-pr-client pre-pr-content \
fixtures-client fixtures-gauntlet golden-diff golden-update \
@@ -57,8 +57,7 @@ help:
@echo " make deny Run cargo deny check (license/ban policy)"
@echo " make star-map-data Regenerate client/data/star_map_data.json from systems.db + wiki"
@echo " make check-star-map Assert star_map_data.json is up to date (part of pre-pr-client)"
@echo " make economy-db Import economics data into systems.db (TOML/JSON → SQLite)"
@echo " make regen-db Regenerate systems.db from all sources + stamp meta table (#855)"
@echo " make regen-db Regenerate systems.db + stamp meta table (= reach ledger import)"
@echo " make install-hooks Install pre-push + pre-commit git hooks (once per clone)"
@echo " make fixtures-client Generate GDScript->Rust cross-encoder fixtures (#475)"
@echo " make golden-diff Show diff if golden file output has changed"
@@ -280,7 +279,7 @@ VENV_PY := $(shell test -x .venv/bin/python && echo .venv/bin/python || echo pyt
# 1. atlas planet determinism guard (#963): simulating the same body twice must
# be bit-identical (guards the expensive 271-body heightmap bake).
# Exit 2 = no testable body found — warn, don't block.
# 2. import_economics --dry-run against the committed DB: full parse +
# 2. reach ledger import --dry-run against the committed DB: full parse +
# structural/coverage validation, no writes. Exit 2 = coverage-gate
# warning (D-175) — tolerated, matching regen-db's treatment.
test-tooling:
@@ -333,20 +332,20 @@ test-tooling:
@mkdir -p .cache
@$(VENV_PY) tooling/test_check.py 2> .cache/test-tooling-check.log || \
{ echo " FAIL: check gates — log follows:"; cat .cache/test-tooling-check.log; exit 1; }
@echo " [test-tooling] economy_import.traits validation units (T-995/PR #173 H2)..."
@echo " [test-tooling] ledger traits validation units (T-995/PR #173 H2)..."
@mkdir -p .cache
@python3 tooling/economy-db/test_traits.py 2> .cache/test-tooling-traits.log || \
@$(VENV_PY) tooling/test_ledger_traits.py 2> .cache/test-tooling-traits.log || \
{ echo " FAIL: traits validation units — log follows:"; cat .cache/test-tooling-traits.log; exit 1; }
@echo " [test-tooling] atlas_city_names/atlas_feature_names idempotency (T-964)..."
@python3 tooling/economy-db/test_atlas_idempotency.py 2> .cache/test-tooling-atlas-idempotency.log || \
@$(VENV_PY) tooling/test_ledger_atlas_idempotency.py 2> .cache/test-tooling-atlas-idempotency.log || \
{ echo " FAIL: atlas name-pool idempotency — log follows:"; cat .cache/test-tooling-atlas-idempotency.log; exit 1; }
@echo " [test-tooling] import_economics --dry-run (committed DB)..."
@rc=0; python3 tooling/economy-db/import_economics.py --dry-run \
@echo " [test-tooling] reach ledger import --dry-run (committed DB)..."
@rc=0; reach ledger import --dry-run \
> .cache/test-tooling-dryrun.log 2>&1 || rc=$$?; \
if [ $$rc -eq 2 ]; then \
echo " WARNING: coverage gate warning (exit 2) — not blocking (matches regen-db)"; \
elif [ $$rc -ne 0 ]; then \
echo " FAIL: import_economics --dry-run exited $$rc — log follows:"; \
echo " FAIL: reach ledger import --dry-run exited $$rc — log follows:"; \
cat .cache/test-tooling-dryrun.log; exit $$rc; \
fi
@echo " test-tooling: PASS"
@@ -450,33 +449,20 @@ ci-client: lint-client build-client test-client
# --- Database ---
economy-db: ## Import economics data (commodities, chains, gate links) into systems.db
@echo " Generating minor brands (D-189 #829)..."
@reach generate brands
@python3 tooling/economy-db/import_economics.py
# Survives for muscle memory (D-263: "delegates to reach in one line and says
# so") — `make regen-db` is named in ~50 files, including the headers of the
# generated wiki TOMLs and the remedies the push gate prints. The behaviour is
# `reach ledger import`; this only tolerates its exit 2, which means imported
# AND stamped with the D-175 coverage gate unmet — a content gap, not a failure.
regen-db: ## Regenerate systems.db and stamp it — delegates to `reach ledger import`
@ec=0; reach ledger import || ec=$$?; [ $$ec -eq 0 ] || [ $$ec -eq 2 ] || exit $$ec
regen-db: ## Regenerate systems.db from all sources and stamp meta table (#855, #856)
@# Run as a single shell so `set -e` covers all steps. Without this
@# each recipe line was a fresh shell and a failure in step 1 did not
@# halt step 2, which could produce stale data with a fresh stamp
@# (PR #136 review T4). import_economics' exit code 2 is a valid
@# coverage-gate-warning state (DB and stamp committed), not an error,
@# so it's explicitly tolerated. Any other non-zero exit halts the
@# pipeline immediately. The atlas city/road/river geometry generator
@# was retired in #951 (D-223); import_economics now owns the atlas
@# index — it loads the names-only pool into atlas_city_names and empties
@# the geometry tables (the server cascade fills them, Phase 4).
@set -e; \
echo " [regen-db] Importing economics data (runs generate_brands internally)..."; \
ec=0; python3 tooling/economy-db/import_economics.py || ec=$$?; \
if [ $$ec -ne 0 ] && [ $$ec -ne 2 ]; then exit $$ec; fi; \
echo ""; \
echo " regen-db complete — systems.db is up to date and stamped."; \
echo " Stage it with: git add server/data/systems.db"
# The five gate targets that lived here are RETIRED, not wrapped (D-263,
# The gate targets that lived here are RETIRED, not wrapped (D-263,
# T-1281). They are tooling, and tooling has one door:
#
# make economy-db -> reach ledger import (T-1289; it also ran
# `reach generate brands` first, which the
# import already does as its own first step)
# make check-systems-db -> reach check systems-db-stamp
# make check-client-version -> reach check client-version
# make check-canvas-version -> reach check canvas-version
docs/DEVOPS.md
+6 -6
View File
@@ -239,14 +239,14 @@ make clean # Remove build artifacts and .cache/ contents
```bash
reach validate content # Validate content YAML against JSON schemas
make check-fact-ids # Check fact_id references against knowledge catalogs
reach check fact-ids # Check fact_id references against knowledge catalogs
```
### Gauntlet Checklists
```bash
make checklist-validate # Validate checklist YAML against schema (standalone)
make checklist-generate # Validate + print per-room condition summary
reach validate checklist --check # Validate checklist YAML against schema (standalone)
reach validate checklist # Validate + print per-room condition summary
```
Checklists live at `content/gauntlet/rooms/{room_id}/checklist.yaml` (per-room) and `content/gauntlet/cross_room_checks.yaml` (cross-room). Each condition is evaluable from an `ObserverSnapshot`.
@@ -268,7 +268,7 @@ the next regeneration.
### Generator
`import_economics` (`tooling/economy-db/import_economics.py`) is the sole
`import_economics` (`reach ledger import`, `tooling/domains/ledger/`) is the sole
generator. As its first step it runs the Rust `generate_brands` binary
(via `core.process.cargo_binary`) to refresh `generated_brands.toml`, then imports
economics data and the atlas index (names-only city pool; geometry tables stay
@@ -288,7 +288,7 @@ After every successful non-dry-run, the generator writes a row to the `meta` tab
registry is the `GENERATOR_SOURCES` dict in `tooling/generator_sources.py`.
```bash
make check-systems-db # Verify the stamp is fresh (exit 1 = stale)
reach check systems-db-stamp # Verify the stamp is fresh (exit 1 = stale)
```
### Making a DB change
@@ -299,7 +299,7 @@ make check-systems-db # Verify the stamp is fresh (exit 1 = stale)
4. Commit with `chore(db): regen systems.db — <reason>`
For schema changes, also update `server/data/systems-schema.sql` and add migration DDL
to `MIGRATION_SQL` in `import_economics.py`.
to `MIGRATION_SQL` in `tooling/domains/ledger/economy_import/migration.py`.
See `.claude/rules/asset-pipeline.md` for the full rule set.
docs/gemma-architecture-flavor-methodology.md
+3 -3
View File
@@ -183,7 +183,7 @@ Gemma **never** sets a numeric or eligibility field. This is not a scoping
choice made for simplicity — it's forced by the shape of the guardrails: the
catalog-wide CI checks (`V-TT-01`: ≥5 eligible templates per `BulkClass`
post-gate; `V-TT-02`: no template >60% of its eligible pool's weight,
`tooling/economy-db/economy_import/traits.py`) are properties of the **whole
`tooling/domains/ledger/economy_import/traits.py`) are properties of the **whole
catalog**, not of any one template in isolation. A per-template extraction
pass, looking at one system's prose, has no way to know whether adding this
candidate at `base_weight = 9000` would push some `BulkClass`'s weighted pool
@@ -244,7 +244,7 @@ automatically at bake time — see §7).
5. **CI guardrails as a post-authoring gate, not a pass/fail on Gemma's
output.** Once Miri has assigned `base_weight`/`weight_mods`/gates,
`make regen-db` re-runs `populate_trait_templates()`
(`tooling/economy-db/economy_import/traits.py`), which re-validates
(`tooling/domains/ledger/economy_import/traits.py`), which re-validates
`V-TT-01` and `V-TT-02` across the **whole** catalog, not just the new
entry. A proposal can clear 1-4 and still get rebalanced or rejected at
bake time if it happens to push a `BulkClass` pool over the 60% line —
@@ -269,7 +269,7 @@ Nothing is committed to `architecture_trait_catalog.toml` until it clears 1-4
4. Araminta adds the matching `visual_bundle`, including the D-235 fallback
parent for every new specific tag (a new template ships with a working
generic-parent fallback from day one, same as the shipped 28).
5. `make regen-db` bakes the change via `import_economics.py` →
5. `make regen-db` bakes the change via `reach ledger import` →
`traits.py:populate_trait_templates()`; `V-TT-01`/`V-TT-02` re-run
automatically (§6.5). `make check-systems-db` confirms the meta stamp.
6. Commit the TOML change and the regenerated `systems.db` together — the
server/data/systems-schema.sql
+4 -4
View File
@@ -298,7 +298,7 @@ CREATE TABLE IF NOT EXISTS commodities (
-- D-237 authored specialization layer: economic-specialization vocabulary.
-- Each value maps to a commodity + projected (BulkClass, ProductionUbiquity) for D-233.
-- Scale is encoded in the value (breadbasket vs estate_farming); no separate scale column.
-- Compiled from wiki/economics/specialization_vocabulary.toml by import_economics.py.
-- Compiled from wiki/economics/specialization_vocabulary.toml by `reach ledger import`.
CREATE TABLE IF NOT EXISTS specialization_vocabulary (
specialization_id TEXT PRIMARY KEY,
commodity_id TEXT NOT NULL REFERENCES commodities(commodity_id),
@@ -400,7 +400,7 @@ CREATE TABLE IF NOT EXISTS corp_lifecycle_events (
-- have to scan hundreds of JSON files. Polyline geometry stays in the files —
-- the DB only stores scalar/filterable fields + `point_count` as a rough length
-- proxy. The geometry tables are populated by the server-side generation
-- cascade (Phase 4, D-223) and start empty; tooling/economy-db/import_economics.py
-- cascade (Phase 4, D-223) and start empty; tooling/domains/ledger/ (reach ledger import)
-- extracts this entire block (between BEGIN/END ATLAS INDEX markers) from this
-- file at runtime and empties the geometry tables on regen, so the DDL lives in
-- exactly one place.
@@ -505,7 +505,7 @@ CREATE INDEX IF NOT EXISTS idx_atlas_mountain_ranges_body ON atlas_mountain_rang
-- Heightmap BLOB storage REMOVED (D-202 amended, #963): canonical elevation is
-- now a per-body 16-bit grayscale heightmap.png file (sea_level in a tEXt
-- chunk), not a systems.db BLOB. The atlas_body_heightmaps table is dropped via
-- MIGRATION_SQL in import_economics.py.
-- MIGRATION_SQL in tooling/domains/ledger/economy_import/migration.py.
-- City name reservations — replaces authored city positions in markers.json (D-207, #902)
-- Position is generated by the city placement algorithm; name is authored or LLM-generated.
@@ -675,7 +675,7 @@ CREATE INDEX IF NOT EXISTS idx_gate_links_to ON gate_links(to_system_id);
-- One row per generator, updated on each successful non-dry-run.
-- schema_version: monotonic semver string (e.g. "1.0.0") — bump on backwards-incompatible changes.
-- Orderable, enabling savegame migration lineage (Phase 5+).
-- Defined as SCHEMA_VERSION constant in tooling/schema_version.py.
-- Defined as SCHEMA_VERSION constant in tooling/domains/ledger/schema_version.py.
-- schema_sha: SHA-1 hex of systems-schema.sql content at generation time (tamper detection).
-- generator_sha: SHA-1 hex of the generator source file(s) content
-- generated_at: ISO-8601 UTC timestamp of the run
server/data/systems.db
BIN
View File
Binary file not shown.
server/src/atlas/trait_catalog_reader.rs
+1 -1
View File
@@ -8,7 +8,7 @@
//! stays DB-free downstream (T-987 keeps `GenerateSkeleton`/`FillChunk` pure).
//!
//! The catalog itself (`trait_templates`) is baked by
//! `tooling/economy-db/economy_import/traits.py` from
//! `tooling/domains/ledger/economy_import/traits.py` from
//! `wiki/economics/architecture_trait_catalog.toml` (#993, #1005) — see
//! `.claude/rules/asset-pipeline.md`. This reader only *consumes* the baked
//! table; it never writes to `systems.db`.
tooling/DOMAINS.md
+17 -11
View File
@@ -89,7 +89,7 @@ invented.
| `atlas` | **the whole spatial ladder** (D-191). Flat verbs for authoring and inspection; nested groups per rung for generation | flat: `atlas` (Rust binary), `atlas-check`, `atlas-names`, `atlas-commit-and-sync`, `atlas-systems-done`, `atlas-update-field`, `atlas-verify`, `atlas-flatness` · `atlas map`: the 5 star-map files · `atlas planet`: ✅ ported (T-1288) from `planet-gen/` (30) — ten verbs, each restating its module's options for real `--help`; `test_planet_router.py` fails if a router option drifts from the module parser |
| ~~`starmap`~~ | **folded into `atlas map`** — the top rung of the same ladder | — |
| ~~`planet`~~ | **folded into `atlas planet`** — the third rung of the same ladder | — |
| `ledger` | the economics pipeline, named for the UI component that will aggregate it | `economy-db/` (17 files), `schema_version.py` |
| `ledger` | the economics pipeline, named for the UI component that will aggregate it | ✅ ported (T-1289). `economy-db/` → `domains/ledger/` (`economy_import/` kept by name; the entrypoint became `service.py`), `schema_version.py` with it. `reach ledger import`; `make regen-db` survives as a one-line delegate, `make economy-db` retired. `generated_brands.toml` byte-identical across the move |
| `wiki` | wiki sync and content maintenance | `wiki/`, `db/wiki_sync.py`, `db/populate_gttr_hook.py`, `db/backfill_cultural_corridor.py`, `assign-astro-ids.py`, `fill-missing-globes.py`, `migrate-s-to-gj.py`, `patch-core-sector.py`, `process-wiki-system-changes` |
| `assets` | connectors to the tower-of-joy generators | `db/audio_*.py`, `db/audio-*`, `db/image_connector.py`, `db/trellis_connector.py`, `db/common.py`, `trellis-batch.sh`, `synth_ui_sounds.py` |
| `character` | bodies, garments, GLB handling | `garment-fit/make_logo.py`, `garment-qa/analyze_captures.py`, `convert_outfit.py`, `glb_strip_utility_nodes.py`, `inspect_glb.py`, `check_hair_symmetry.py`, `check_icosphere.py`, `render_quaternius_test.py`, `setup_clothing_metadata.py` — **note this is far smaller than `garment-fit/`'s file count suggests; 22 of its 23 files are Blender payloads and belong to the carve-out** |
@@ -110,11 +110,13 @@ They move beside the domain that reads them (`check` and `db` respectively) as
plain modules, not verbs. Making them commands would put something in
`reach --help` that answers no question a person has.
**`schema_version.py` belongs to `db`, and `check` imports it.** It is consumed
by the economics importer *and* by the stamp gate. Shared, but not equally
owned: the importer defines the version, the gate reads it. It goes where it is
defined, and the cross-domain import is legitimate — that is what a service
layer is for.
**`schema_version.py` belongs to `ledger` (named `db` when this map was
drawn), and `check` reads it.** It is consumed by the economics importer *and*
by the stamp gate. Shared, but not equally owned: the importer defines the
version, the gate reads it. It goes where it is defined. *As ported (T-1289)
the gate never imports the module at all — it reads the version back out of
the `meta` table and checks it is semver — so the cross-domain import this
ruling allowed for turned out not to be needed.*
**`test_*.py` files do NOT become a domain.** They are gate tests run by
`make test-tooling`, not commands anyone types. `reach test …` would imply a
@@ -138,14 +140,18 @@ data surface), `planet-gen/atlas_*.py` (`atlas_cohesion_audit`,
`atlas_common`, `atlas_quality_analysis` — quality analysis of generated
terrain), and `economy-db/atlas.py` (the atlas index tables in `systems.db`).
These are three concerns sharing a noun, not one domain in three places. Each
stays with its owner — `atlas`, `planet` and `db` respectively — and the port
should resist the pull to collect them, which would produce a domain whose only
common thread is a word.
stays with its owner — `atlas`, `atlas planet` and `ledger` respectively — and
the port should resist the pull to collect them, which would produce a domain
whose only common thread is a word. *Held through both ports (T-1288,
T-1289): `domains/atlas/`, `domains/atlas/planet/atlas_*.py` and
`domains/ledger/economy_import/atlas.py` are still three separate files.*
**`economy-db/errors.py` predates `core/errors.py` and is not the same thing.**
Domain-local error types are fine; what must not happen is a silent merge, or a
second `ReachError` with different semantics. Reconcile explicitly when `db` is
ported.
second `ReachError` with different semantics. *Reconciled in T-1289:
`ImportAborted` stays, as internal control flow only — a step raises it to
request a rollback. It never reaches a caller: `ledger/service.py` rolls back
and converts it to a `ReachError` carrying the remedy.*
**`character` rather than `garment`.** D-263's sketch says `garment`, but the
files cover bodies, hair, GLB utilities and Quaternius imports as well as
tooling/domains/ledger/__init__.py
+12
View File
@@ -0,0 +1,12 @@
"""`ledger` — the economics pipeline (D-263).
Named for the implant app the player will use to browse markets, wealth and
transactions: the economic counterpart to what the Atlas is for topography.
Not `db`, which named a storage layer nobody looks at, and not `economics`,
which names a subject rather than the thing on screen.
Formerly `tooling/economy-db/` plus `tooling/schema_version.py` (T-1289). The
importer here is the SOLE generator of `server/data/systems.db`; its stamped
source set lives in `tooling/generator_sources.py`, which is why moving these
files changed the stamp and the move landed together with a regen.
"""
tooling/economy-db/economy_import/__init__.py → tooling/domains/ledger/economy_import/__init__.py
+8 -15
View File
@@ -1,9 +1,8 @@
"""
economy_import — module package behind tooling/economy-db/import_economics.py.
economy_import — the import steps behind `reach ledger import`.
Split out of the former single-file importer (T-1067). The entrypoint
``import_economics.py`` remains the CLI (Makefile regen-db, make test-tooling)
and orchestrates the single-transaction import; the steps live here:
Split out of the former single-file importer (T-1067). `ledger/service.py`
orchestrates the single-transaction import; the steps live here:
paths.py source-file path constants (re-exports stamped paths
from tooling/generator_sources.py)
@@ -24,15 +23,9 @@ and orchestrates the single-transaction import; the steps live here:
Every module here is part of the import_economics meta-stamp source set —
tooling/generator_sources.py globs this directory, so adding a module
automatically extends staleness detection.
The sys.path bootstrap that used to sit here is gone (T-1289). It existed only
because `tooling/economy-db/` was hyphenated and so not importable; inside the
package, `tooling.generator_sources` and the ledger's `schema_version` resolve
by ordinary import.
"""
import sys
from pathlib import Path
# Bootstrap: make tooling/ importable (generator_sources, schema_version)
# before any submodule needs them. tooling/economy-db is not a package
# (hyphenated dir), so the entrypoint puts it on sys.path and this package
# adds tooling/ itself.
_TOOLING_DIR = Path(__file__).resolve().parents[2]
if str(_TOOLING_DIR) not in sys.path:
sys.path.insert(0, str(_TOOLING_DIR))
tooling/economy-db/economy_import/atlas.py → tooling/domains/ledger/economy_import/atlas.py
+5 -6
View File
@@ -19,6 +19,7 @@ import sqlite3
import tomllib
from .paths import SCHEMA_SQL, SETTLEMENT_NAME_LOCKED_TOML, WIKI_STAR_SYSTEMS
from tooling.core import console
# Atlas geometry index tables (D-191). These hold computed positions — city
# centres, road/river/rail polylines, ocean/mountain extents. Under D-223 the
@@ -139,7 +140,7 @@ def populate_atlas_city_names(conn: sqlite3.Connection, dry_run: bool) -> int:
if skipped_bodies:
unique = sorted(set(skipped_bodies))
print(f" warning: {len(unique)} body dirs not in DB — skipped: {unique[:5]}")
console.event(f"{len(unique)} body dirs not in DB — skipped: {unique[:5]}", level="warn")
if not dry_run:
# D-242: corporations.headquarters_city_id FK-references
@@ -236,7 +237,7 @@ def populate_atlas_feature_names(conn: sqlite3.Connection, dry_run: bool) -> int
if skipped_bodies:
unique = sorted(set(skipped_bodies))
print(f" warning: {len(unique)} body dirs not in DB — skipped: {unique[:5]}")
console.event(f"{len(unique)} body dirs not in DB — skipped: {unique[:5]}", level="warn")
if not dry_run:
conn.execute("DELETE FROM atlas_feature_names")
@@ -467,10 +468,8 @@ def populate_settlement_population_class(conn: sqlite3.Connection, dry_run: bool
name_locked_applied += match_count
if name_locked_unmatched:
print(
f" warning: {len(name_locked_unmatched)} NameLocked hero stanza(s) "
f"matched no atlas_city_names row: {name_locked_unmatched}"
)
console.event(f"{len(name_locked_unmatched)} NameLocked hero stanza(s) "
f"matched no atlas_city_names row: {name_locked_unmatched}", level="warn")
return {
"bodies_spread": len(city_rows_by_body),
tooling/economy-db/economy_import/bodies.py → tooling/domains/ledger/economy_import/bodies.py
+2 -1
View File
@@ -5,6 +5,7 @@ import sqlite3
from pathlib import Path
from .paths import WIKI_STAR_SYSTEMS
from tooling.core import console
def populate_body_radius_km(conn: sqlite3.Connection, dry_run: bool) -> int:
@@ -290,7 +291,7 @@ def populate_biosphere_class(conn: sqlite3.Connection, dry_run: bool) -> int:
if body_id and bc:
bc = bc.strip().strip("'\"")
if bc not in _VALID:
print(f" WARNING: {fpath}: biosphere_class {bc!r} not in {_VALID}; ignored")
console.event(f"{fpath}: biosphere_class {bc!r} not in {_VALID}; ignored", level="warn")
continue
override[body_id] = bc
tooling/economy-db/economy_import/brands.py → tooling/domains/ledger/economy_import/brands.py
+14 -22
View File
@@ -1,18 +1,11 @@
"""Brand layer (D-189, #827): generate_brands shell-out, TOML import, validation."""
import sqlite3
import sys
import tomllib
from pathlib import Path
from .errors import ImportAborted
from .paths import BRANDS_TOML, GENERATED_BRANDS_TOML, REPO_ROOT
# economy-db/ is hyphenated, so it is not importable as a package and cannot
# reach `tooling.core` by normal import. The bootstrap goes away with T-1272
# (the hyphen sweep); until then it is explicit rather than implied.
if str(REPO_ROOT) not in sys.path:
sys.path.insert(0, str(REPO_ROOT))
from .paths import BRANDS_TOML, GENERATED_BRANDS_TOML
from tooling.core import console
VALID_BRAND_CATEGORIES: set[str] = {
"terroir", "heritage_craft", "tech_premium", "cultural",
@@ -41,22 +34,21 @@ def regenerate_brands() -> None:
wrapper was one of three identical copies of build-if-missing-then-exec, so
it was retired into `core.process.cargo_binary` (T-1286) and this calls that
helper instead. Same binary, same seed, same output.
A build or run failure propagates as the ReachError `cargo_binary` raised.
It used to be printed and swapped for ImportAborted, which dropped its
remedy (T-1289). Nothing needs rolling back: this runs before the import
transaction opens.
"""
from tooling.core.errors import ReachError
from tooling.core.process import cargo_binary
print(" [pre/10] Running generate_brands (Rust) to refresh generated_brands.toml...")
try:
stdout = cargo_binary("generate_brands")
except ReachError as exc:
print(str(exc), file=sys.stderr)
raise ImportAborted() from exc
console.event("Running generate_brands (Rust) to refresh generated_brands.toml...", phase="pre")
stdout = cargo_binary("generate_brands")
# Print the Rust binary's own summary lines (brands generated, coverage).
# Indent so they fold under the pre-step heading.
# Relay the Rust binary's own summary lines (brands generated, coverage).
for line in stdout.splitlines():
if line.strip():
print(f" {line}")
console.event(line.strip(), phase="pre")
def _load_brand_file(path: Path) -> tuple[list, list]:
@@ -74,18 +66,18 @@ def import_brands(
"""Import brand_products and brand_inputs from brands.toml and generated_brands.toml.
Hand-authored brands (brands.toml) are imported first; generated brands
(generated_brands.toml, produced by `tooling/generate-brands`) are merged in.
(generated_brands.toml, produced by `reach generate brands`) are merged in.
Returns (n_products, n_inputs).
"""
if not BRANDS_TOML.exists():
print(" warning: brands.toml not found — brand layer skipped")
console.event("brands.toml not found — brand layer skipped", level="warn")
return 0, 0
products_authored, inputs_authored = _load_brand_file(BRANDS_TOML)
products_generated, inputs_generated = _load_brand_file(GENERATED_BRANDS_TOML)
if products_generated:
print(f" merging {len(products_generated)} generated brand_products from generated_brands.toml")
console.event(f"merging {len(products_generated)} generated brand_products from generated_brands.toml")
products = products_authored + products_generated
inputs = inputs_authored + inputs_generated
tooling/economy-db/economy_import/corporations.py → tooling/domains/ledger/economy_import/corporations.py
+14 -17
View File
@@ -9,6 +9,7 @@ from pathlib import Path
from .atlas import most_populated_body_in_system
from .errors import ImportAborted
from .paths import CORP_HQ_PLACEMENT_TOML, CORPORATIONS_DIR
from tooling.core import console
# ---------------------------------------------------------------------------
# Corporation wiki parsing
@@ -117,8 +118,8 @@ def sync_corporations(
system_id = corp.get("system_id")
hq_system = system_id if system_id and system_id in valid_systems else None
if system_id and system_id not in valid_systems:
print(f" warning: {corp_id} HQ system '{system_id}' not in DB, "
f"headquarters_system set to NULL")
console.event(f"{corp_id} HQ system '{system_id}' not in DB, "
f"headquarters_system set to NULL", level="warn")
to_insert.append((
corp_id,
proper_name,
@@ -274,11 +275,9 @@ def populate_corp_specialization(
r[0] for r in conn.execute("SELECT DISTINCT body_id FROM atlas_city_names").fetchall()
}
if not city_bearing_bodies:
print(
" warning: atlas_city_names is empty — the city-presence "
console.event("atlas_city_names is empty — the city-presence "
"tiebreak tier is inert this run (placements fall through to "
"body-type rank; they converge on the next run)"
)
"body-type rank; they converge on the next run)", level="warn")
errors: list[str] = []
unauthored: list[str] = []
@@ -333,16 +332,14 @@ def populate_corp_specialization(
spec_rows.append((corp_id, spec, placement, authored_body))
if errors:
print(f" CORP SPECIALIZATION ERRORS ({len(errors)}):")
console.event(f"CORP SPECIALIZATION ERRORS ({len(errors)}):", level="error")
for e in errors:
print(f" - {e}")
console.event(f"- {e}", level="error")
raise ImportAborted()
if unauthored:
print(
f" warning: {len(unauthored)} corp(s) have no authored "
f"corp_specialization (stay NULL; no HQ placement): {unauthored[:5]}"
)
console.event(f"{len(unauthored)} corp(s) have no authored "
f"corp_specialization (stay NULL; no HQ placement): {unauthored[:5]}", level="warn")
# Pre-compute the resolution so dry-run reports the same numbers a real
# run writes (write is gated on dry_run; the derivation is not).
@@ -440,7 +437,7 @@ def import_corp_presence(
if skipped:
for s in skipped:
print(f" warning: skipped corp_presence for {s}")
console.event(f"skipped corp_presence for {s}", level="warn")
if not dry_run:
conn.execute("DELETE FROM corp_presence")
@@ -547,11 +544,11 @@ def populate_standalone_hq_settlements(conn: sqlite3.Connection, dry_run: bool)
n_linked += 1
if unmatched_bodies:
print(f" warning: {len(unmatched_bodies)} corp(s) skipped (bad headquarters_body): "
f"{unmatched_bodies[:5]}")
console.event(f"{len(unmatched_bodies)} corp(s) skipped (bad headquarters_body): "
f"{unmatched_bodies[:5]}", level="warn")
if unmatched_corps:
print(f" warning: {len(unmatched_corps)} CityTenant corp(s) unmatched: "
f"{unmatched_corps[:5]}")
console.event(f"{len(unmatched_corps)} CityTenant corp(s) unmatched: "
f"{unmatched_corps[:5]}", level="warn")
return {
"standalone_inserted": n_inserted,
tooling/economy-db/economy_import/db.py → tooling/domains/ledger/economy_import/db.py
View File
tooling/economy-db/economy_import/economy.py → tooling/domains/ledger/economy_import/economy.py
+4 -3
View File
@@ -6,6 +6,7 @@ import sqlite3
import tomllib
from .paths import CHAINS_TOML, COMMODITIES_TOML, CURRENCY_ZONES_TOML, STAR_MAP
from tooling.core import console
# ---------------------------------------------------------------------------
# Gate links
@@ -33,7 +34,7 @@ def import_gate_links(conn: sqlite3.Connection, dry_run: bool) -> int:
if skipped:
unique_skipped = sorted(set(skipped))
print(f" warning: {len(unique_skipped)} system(s) in star-map.json not in DB: {unique_skipped[:5]}...")
console.event(f"{len(unique_skipped)} system(s) in star-map.json not in DB: {unique_skipped[:5]}...", level="warn")
if not dry_run:
conn.executemany(
@@ -169,8 +170,8 @@ def set_currency_zones(conn: sqlite3.Connection, dry_run: bool) -> dict[str, int
(sid,),
)
else:
print(" warning: wiki/economics/currency_zones.toml not found — "
"all systems default to TRACTUS_PRIMARY / Sol to MIXED")
console.event("wiki/economics/currency_zones.toml not found — "
"all systems default to TRACTUS_PRIMARY / Sol to MIXED", level="warn")
counts: dict[str, int | str] = {}
for row in conn.execute("SELECT currency_zone, COUNT(*) FROM star_systems GROUP BY currency_zone"):
tooling/economy-db/economy_import/errors.py → tooling/domains/ledger/economy_import/errors.py
View File
tooling/economy-db/economy_import/migration.py → tooling/domains/ledger/economy_import/migration.py
View File
tooling/economy-db/economy_import/paths.py → tooling/domains/ledger/economy_import/paths.py
+1 -1
View File
@@ -8,7 +8,7 @@ importer can never disagree about where a stamped source lives.
from pathlib import Path
from generator_sources import (
from tooling.generator_sources import (
ARCHITECTURE_TRAIT_BIAS_TOML,
ARCHITECTURE_TRAIT_CATALOG_TOML,
ARCHITECTURE_ZONE_BIAS_TOML,
tooling/economy-db/economy_import/specialization.py → tooling/domains/ledger/economy_import/specialization.py
+15 -16
View File
@@ -6,6 +6,7 @@ from collections import Counter
from .errors import ImportAborted
from .paths import SPECIALIZATION_VOCAB_TOML, SYSTEM_SPECIALIZATION_TOML
from tooling.core import console
# Authoritative D-233 projected enums. The full CI guardrail suite (V-SES-*)
# lands in #1015; this module performs the FK + enum sanity the import itself
@@ -194,9 +195,9 @@ def import_system_specialization(conn: sqlite3.Connection, dry_run: bool,
)
if errors:
print(f" SPECIALIZATION ERRORS ({len(errors)}):")
console.event(f"SPECIALIZATION ERRORS ({len(errors)}):", level="error")
for e in errors:
print(f" - {e}")
console.event(f"- {e}", level="error")
raise ImportAborted()
coverage: dict = {
@@ -403,26 +404,24 @@ def _specialization_checks(
if e and e in vocab:
bulk_dist[vocab[e].get("bulk_class_projected")] += 1
pu_dist[vocab[e].get("production_ubiquity_projected")] += 1
print(" Specialization coverage (D-237):")
print(f" economic: {n_econ} authored ({n_inhabited} inhabited; rest on fallback once #1014 lands)")
print(f" cultural: {n_cult} authored ({n_named} named; rest on corridor default)")
print(f" faction: {n_fac} authored ({n_named} named; rest on derivation)")
print(" BulkClass: " + " | ".join(f"{k}={v}" for k, v in sorted(bulk_dist.items())))
print(" ProductionUbiquity: " + " | ".join(f"{k}={v}" for k, v in sorted(pu_dist.items())))
console.event("Specialization coverage (D-237):")
console.event(f"economic: {n_econ} authored ({n_inhabited} inhabited; rest on fallback once #1014 lands)")
console.event(f"cultural: {n_cult} authored ({n_named} named; rest on corridor default)")
console.event(f"faction: {n_fac} authored ({n_named} named; rest on derivation)")
console.event("BulkClass: " + " | ".join(f"{k}={v}" for k, v in sorted(bulk_dist.items())))
console.event("ProductionUbiquity: " + " | ".join(f"{k}={v}" for k, v in sorted(pu_dist.items())))
if warnings:
print(f" Specialization warnings ({len(warnings)}):")
console.event(f"Specialization warnings ({len(warnings)}):")
for w in warnings:
print(f" - {w}")
console.event(f"- {w}", level="warn")
if gate_failures:
if strict:
print(f" SPECIALIZATION COMPLETENESS FAILURES ({len(gate_failures)}) [strict]:")
console.event(f"SPECIALIZATION COMPLETENESS FAILURES ({len(gate_failures)}) [strict]:", level="error")
for g in gate_failures:
print(f" - {g}")
console.event(f"- {g}", level="error")
raise ImportAborted()
else:
print(
f" Specialization completeness: {len(gate_failures)} gate item(s) "
console.event(f"Specialization completeness: {len(gate_failures)} gate item(s) "
f"pending (#1014 fallback / #1016 content pass) — warnings only until "
f"--strict-specialization"
)
f"--strict-specialization")
tooling/economy-db/economy_import/stamp.py → tooling/domains/ledger/economy_import/stamp.py
+3 -3
View File
@@ -1,15 +1,15 @@
"""Meta-table generator stamp (#855, #856).
Records the source-SHA of the generator that produced the DB so the pre-push
hook (tooling/check-systems-db-stamp) can detect stale snapshots. The source
hook (`reach check systems-db-stamp`) can detect stale snapshots. The source
set comes from tooling/generator_sources.py — the single shared registry.
"""
import sqlite3
from pathlib import Path
from generator_sources import file_sha1
from schema_version import SCHEMA_VERSION
from tooling.generator_sources import file_sha1
from tooling.domains.ledger.schema_version import SCHEMA_VERSION
from .paths import SCHEMA_SQL
tooling/economy-db/economy_import/traits.py → tooling/domains/ledger/economy_import/traits.py
+11 -10
View File
@@ -12,6 +12,7 @@ from .paths import (
COLOR_REGISTER_BANDS_TOML,
OBJECT_TAG_VOCABULARY_TOML,
)
from tooling.core import console
_TRAIT_CORRIDOR_POOLS: set[str] = {"baseline", "heritage", "cross_corridor"}
_TRAIT_BIAS_KINDS: set[str] = {"pin", "boost", "suppress"}
@@ -205,9 +206,9 @@ def populate_trait_templates(conn: sqlite3.Connection, dry_run: bool) -> int:
json.dumps(t["visual_bundle"]) if t.get("visual_bundle") else None,
))
if errors:
print(f" TRAIT TEMPLATE ERRORS ({len(errors)}):")
console.event(f"TRAIT TEMPLATE ERRORS ({len(errors)}):", level="error")
for e in errors:
print(f" - {e}")
console.event(f"- {e}", level="error")
raise ImportAborted()
# CI guardrails (D-232, Nigel): >=5 templates eligible per BulkClass (a
# template with no bulk_class_gate is eligible for all); no single template
@@ -232,9 +233,9 @@ def populate_trait_templates(conn: sqlite3.Connection, dry_run: bool) -> int:
f"{w * 100 // total}% of the {bc} pool weight (>60%)"
)
if gerrors:
print(f" TRAIT TEMPLATE GUARDRAIL FAILURES ({len(gerrors)}):")
console.event(f"TRAIT TEMPLATE GUARDRAIL FAILURES ({len(gerrors)}):", level="error")
for e in gerrors:
print(f" - {e}")
console.event(f"- {e}", level="error")
raise ImportAborted()
# Validation/guardrails run on dry-run too; only mutate when committing.
if not dry_run:
@@ -340,9 +341,9 @@ def populate_architecture_zone_bias(conn: sqlite3.Connection, dry_run: bool) ->
rows.append((template_tag, zone_type_id, bias_json))
if errors:
print(f" ARCHITECTURE ZONE BIAS ERRORS ({len(errors)}):")
console.event(f"ARCHITECTURE ZONE BIAS ERRORS ({len(errors)}):", level="error")
for e in errors:
print(f" - {e}")
console.event(f"- {e}", level="error")
raise ImportAborted()
if not dry_run:
@@ -432,9 +433,9 @@ def populate_color_register_bands(conn: sqlite3.Connection, dry_run: bool) -> in
errors.append(f"V-TT-07: color_register '{reg}' referenced by trait_templates but has no band")
if errors:
print(f" COLOR REGISTER BAND ERRORS ({len(errors)}):")
console.event(f"COLOR REGISTER BAND ERRORS ({len(errors)}):", level="error")
for e in errors:
print(f" - {e}")
console.event(f"- {e}", level="error")
raise ImportAborted()
if not dry_run:
@@ -507,9 +508,9 @@ def populate_atlas_body_trait_bias(conn: sqlite3.Connection, dry_run: bool) -> i
f"K of {_MAX_PINS_PER_BODY} (ComplexityTier::Full); pins count toward K (D-232)"
)
if errors:
print(f" TRAIT BIAS ERRORS ({len(errors)}):")
console.event(f"TRAIT BIAS ERRORS ({len(errors)}):", level="error")
for e in errors:
print(f" - {e}")
console.event(f"- {e}", level="error")
raise ImportAborted()
if not dry_run:
conn.execute("DELETE FROM atlas_body_trait_bias")
tooling/economy-db/economy_import/validators.py → tooling/domains/ledger/economy_import/validators.py
View File
tooling/domains/ledger/router.py
+47
View File
@@ -0,0 +1,47 @@
"""Transport for the `ledger` domain — args in, delegate, format out."""
from __future__ import annotations
from pathlib import Path
import typer
from tooling.core import cli
from tooling.core.command import command
app = cli.domain("ledger", "The economics pipeline — the import that builds systems.db.")
@app.callback()
def _domain() -> None:
"""Keeps `ledger` a group (Typer collapses a single-command app)."""
@app.command("import")
@command
def import_(
db: Path = typer.Option(None, "--db", help="systems.db to import into (default: server/data/systems.db)."),
dry_run: bool = typer.Option(False, "--dry-run", help="Validate every step, write nothing."),
strict_specialization: bool = typer.Option(
False,
"--strict-specialization",
help="Treat the D-237 completeness gates (V-SES-02, V-FAC-01) as hard errors.",
),
) -> None:
"""Rebuild systems.db from the economics sources, and stamp it.
The sole generator of server/data/systems.db — `make regen-db` runs this.
It regenerates generated_brands.toml first (skipped on --dry-run), imports
in one transaction, and stamps the meta table so the push gate can tell a
stale snapshot from a fresh one.
Exit 2 means imported AND stamped but the D-175 coverage gate is not met:
a content gap, not a failure, and tolerated by regen-db and the tooling
gate. Exit 1 means nothing was written.
"""
# Imported here, not at module level: the service pulls in the whole
# importer package and the stamp registry, and `reach ledger --help`
# should not pay for that (T-1260).
from tooling.domains.ledger import service
service.run(db or service.DB_PATH, dry_run=dry_run, strict_specialization=strict_specialization)
tooling/schema_version.py → tooling/domains/ledger/schema_version.py
+3 -2
View File
@@ -6,8 +6,9 @@ Bump SCHEMA_VERSION manually on any backwards-incompatible schema change
Additive changes (new nullable columns, new tables, new indexes) do not
require a bump.
Imported by:
tooling/economy-db/import_economics.py
Lives in `ledger` because the importer is where the version is DEFINED;
`reach check systems-db-stamp` only reads it back out of the meta table
(tooling/DOMAINS.md). Imported by economy_import/stamp.py.
"""
SCHEMA_VERSION = "1.0.0"
tooling/domains/ledger/service.py
Executable
+369
View File
@@ -0,0 +1,369 @@
"""Logic for the `ledger` domain — the economics import into systems.db. Transport-agnostic (D-263).
Reads TOML/JSON source files and populates the economics tables:
- gate_links from docs/design/star-map.json (335 edges, bidirectional)
- commodities from wiki/economics/commodities.toml (36 types)
- production_chains + chain_inputs from wiki/economics/production_chains.toml
- currency_zone on star_systems (default TRACTUS_PRIMARY)
- gate_energy_connected on star_systems (D-186: false for MARK_PRIMARY zones)
- corporations from wiki/corporations/*.md (sync + insert new records)
- corp_specialization/hq_placement/headquarters_body/headquarters_city_id
on corporations (D-242: HQ-placement key + baked
CityTenant link, from corp_hq_placement.toml)
- corp_presence from wiki/corporations/*.md (headquarters location data)
- atlas_city_names Standalone-HQ settlement rows (D-242, T-1074)
Validation (hard errors, non-zero exit on any failure):
- Wiki corporation names must match DB proper_name records (D-182 sync constraint)
- Chain completeness: every intermediate commodity has at least one production chain
- Commodity coverage: 3+ corporations per major commodity type (D-175)
- System coverage: 1+ corporation per inhabited system with population > 100K (D-175)
This is the sole generator of `server/data/systems.db` and was
`tooling/economy-db/import_economics.py` until T-1289. The steps live in the
`economy_import` package; the stamped source set is defined in
`tooling/generator_sources.py`.
**Exit codes are a contract, kept exactly.** 0 = imported and every coverage
threshold met. 2 = imported AND stamped, but the D-175 coverage gate is not met
— `make regen-db` and the tooling gate both tolerate it, because the data is
usable and the gap is content, not code. 1 = nothing written.
**Two error types, deliberately.** `economy_import.errors.ImportAborted` is
internal control flow — a step raises it to request rollback after printing
what went wrong. It never escapes this module: `run()` rolls back and converts
it to a `ReachError` carrying the remedy. That is the explicit reconciliation
DOMAINS.md asked for, rather than a second error type with different semantics
reaching the caller.
"""
from __future__ import annotations
import sqlite3
from pathlib import Path
from tooling.core import console
from tooling.core.errors import ReachError
from tooling.domains.ledger.economy_import import (
atlas,
bodies,
brands,
corporations,
db,
economy,
migration,
specialization,
stamp,
traits,
validators,
)
from tooling.domains.ledger.economy_import.errors import ImportAborted
from tooling.domains.ledger.economy_import.paths import DB_PATH
from tooling.generator_sources import IMPORT_ECONOMICS_SOURCES
__all__ = ["DB_PATH", "run"]
# Named once, so the progress fraction and the count cannot disagree.
STEPS = 24
ABORTED_FIX = (
"fix the errors listed above in their source (wiki/economics/*.toml, "
"wiki/corporations/*.md), then re-run: reach ledger import --dry-run"
)
class _Steps:
"""Numbered progress through the import, one event per step."""
def __init__(self) -> None:
self.n = 0
def __call__(self, message: str) -> None:
self.n += 1
console.event(message, phase=f"{self.n}/{STEPS}", progress=self.n / STEPS)
def run(db_path: Path, *, dry_run: bool = False, strict_specialization: bool = False) -> None:
"""Import every economics source into `db_path`, in one transaction.
Raises ReachError with exit code 1 when nothing was written, and with exit
code 2 when the import committed but the coverage gate is not met.
"""
if not db_path.exists():
raise ReachError(
f"{db_path} not found",
fix="pass --db with an existing systems.db, or restore it: git restore server/data/systems.db",
)
console.event(f"Economics import into {db_path}" + (" — DRY RUN" if dry_run else ""))
# Load wiki corps before opening the DB — allows early exit on parse failures.
console.event("Loading wiki corporations...")
wiki_corps = corporations.load_wiki_corps()
console.event(f"{len(wiki_corps)} corporation files parsed")
# Regenerate generated_brands.toml via the Rust binary before the import
# reads it — single pipeline, single stamp (review T2/H3). Skipped on a
# dry run to avoid a disk side effect during validation.
if not dry_run:
brands.regenerate_brands()
conn = db.connect(db_path)
try:
counts = _import(conn, wiki_corps, dry_run, strict_specialization)
except ImportAborted:
conn.rollback()
conn.close()
raise ReachError(
"economics import aborted — rolled back, nothing written", fix=ABORTED_FIX
) from None
except BaseException:
# Anything else (KeyboardInterrupt, MemoryError, a DB error, a bug)
# rolls back too, so the DB is never left half-imported. Re-raised so
# the traceback survives.
conn.rollback()
conn.close()
raise
# Stamp generator metadata (#855, #856) so the pre-push hook can detect a
# stale snapshot. Written BEFORE the coverage gate: the stamp records which
# code produced the DB, not whether the data is complete (#860).
if not dry_run:
try:
stamp.write_stamp(conn, "import_economics", *IMPORT_ECONOMICS_SOURCES)
conn.commit()
console.event("Stamped: import_economics (covers brand pipeline Rust sources)")
except Exception as exc: # noqa: BLE001
console.event(f"failed to write generator stamp: {exc}", level="warn")
coverage_errors = _coverage(conn, wiki_corps)
conn.close()
summary = (
f"{counts['links']} gate_links, {counts['commodities']} commodities, "
f"{counts['chains']} chains, {counts['inputs']} inputs, "
f"{counts['presence']} corp_presence, {counts['brands']} brand_products, "
f"{counts['brand_inputs']} brand_inputs, {counts['fiscal']} system_fiscal"
)
if coverage_errors:
for e in coverage_errors:
console.event(e, level="error")
raise ReachError(
f"imported ({summary}) but the D-175 Phase 2 coverage gate is not met — "
f"{len(coverage_errors)} gap(s) above. "
+ ("Nothing was written (dry run)." if dry_run else "Data and stamp are committed."),
fix="add corporations to wiki/corporations/ until the thresholds are met, then re-run",
exit_code=2,
)
written = "validated, nothing written (dry run)" if dry_run else "imported and stamped"
console.verdict(f"ledger import: {written} — {summary}")
def _import(
conn: sqlite3.Connection, wiki_corps, dry_run: bool, strict_specialization: bool
) -> dict[str, int]:
"""The clear-then-reimport cycle, as one explicit transaction.
Any crash, validation error or interrupt between the first DELETE and the
commit rolls everything back (the caller owns that): the DB never ends up
half-cleared. On success it commits exactly once, right after structural
validation passes. On a dry run the transaction is left open so the
coverage check can still SELECT the imported rows; closing discards it.
"""
step = _Steps()
conn.execute("BEGIN")
# Schema migration — idempotent, inside the tx so a crash leaves no
# half-applied ALTER TABLE.
step("Schema migration...")
migration.apply_schema_migrations(conn)
# Atlas index tables: canonical DDL + empty geometry (D-223, #951).
atlas.ensure_atlas_index_schema(conn, dry_run)
# Clear economics tables in FK-safe order (children before parents).
if not dry_run:
db.clear_economics_tables(conn)
step("Importing gate links...")
n_links = economy.import_gate_links(conn, dry_run)
console.event(f"{n_links} rows (bidirectional)")
step("Importing commodities...")
n_commodities = economy.import_commodities(conn, dry_run)
console.event(f"{n_commodities} commodities")
step("Importing production chains...")
n_chains, n_inputs = economy.import_chains(conn, dry_run)
console.event(f"{n_chains} chains, {n_inputs} inputs")
# D-237 authored layer — after commodities (FK), before currency zones.
# UPSERTs onto pre-existing system_economy / system_factions rows;
# unauthored systems stay NULL.
step("Importing system specialization (D-237)...")
spec = specialization.import_system_specialization(conn, dry_run, strict=strict_specialization)
console.event(
f"vocab {spec['vocab']} | economic {spec['economic']} | "
f"cultural {spec['cultural']} | faction {spec['faction']}"
)
if spec["missing_economy_row"]:
console.event(
f"{len(spec['missing_economy_row'])} authored system(s) lack a "
f"system_economy row (values dropped): {spec['missing_economy_row']}",
level="warn",
)
if spec["missing_faction_row"]:
console.event(
f"{len(spec['missing_faction_row'])} authored system(s) lack a "
f"system_factions row (faction dropped): {spec['missing_faction_row']}",
level="warn",
)
step("Setting currency zones...")
for zone, count in sorted(economy.set_currency_zones(conn, dry_run).items()):
console.event(f"{zone}: {count}")
# D-186 — must run after currency zones.
step("Setting gate energy connectivity...")
for label, count in sorted(economy.set_gate_energy(conn, dry_run).items()):
console.event(f"{label}: {count}")
# D-182: a name divergence is a hard error.
step("Syncing corporations...")
corp_errors = corporations.sync_corporations(conn, wiki_corps, dry_run)
if corp_errors:
console.event(f"name divergence detected (D-182) — {len(corp_errors)} error(s):", level="error")
for e in corp_errors:
console.event(e, level="error")
console.event("update the wiki title or the DB proper_name to match, then re-run", level="error")
raise ImportAborted()
n_db_corps = conn.execute("SELECT COUNT(*) FROM corporations").fetchone()[0]
console.event(f"{n_db_corps} corporations in DB ({len(wiki_corps)} from wiki)")
# D-242 Phase A — reset + derive every run (PR #177 H1/T2). After corp
# sync (rows must exist), before corp_presence (reads headquarters_body
# back), and before the city-presence tiebreak further down, which reads
# atlas_city_names BEFORE this run's rebuild — the previous run's settled
# state, by design.
step("Importing corp specialization + HQ placement (D-242)...")
corp_spec = corporations.populate_corp_specialization(conn, wiki_corps, dry_run)
console.event(
f"{corp_spec['specialized']}/{corp_spec['total']} corps specialized, "
f"{corp_spec['hq_resolved']} headquarters_body resolved (recomputed every run; "
f"{corp_spec['hq_overridden']} authored overrides)"
)
step("Importing corp presence...")
commodity_ids = {r[0] for r in conn.execute("SELECT commodity_id FROM commodities").fetchall()}
n_presence = corporations.import_corp_presence(conn, wiki_corps, commodity_ids, dry_run)
console.event(f"{n_presence} corp_presence rows")
# D-189, #827.
step("Importing brand products and inputs...")
n_brands, n_brand_inputs = brands.import_brands(conn, dry_run)
console.event(f"{n_brands} brand_products, {n_brand_inputs} brand_inputs")
# D-189 §6.
step("Populating system_fiscal...")
n_fiscal = economy.import_system_fiscal(conn, dry_run)
console.event(f"{n_fiscal} system_fiscal rows")
# D-204, #910.
step("Populating body_radius_km fallback...")
console.event(f"{bodies.populate_body_radius_km(conn, dry_run)} bodies updated")
# D-207, #908.
step("Populating atlas_city_names from wiki content...")
console.event(f"{atlas.populate_atlas_city_names(conn, dry_run)} city name rows")
# D-223, T-1169 — mirrors the city pool's shape over a different
# names-only markers.json key set; independent of the settlement pool.
step("Populating atlas_feature_names from wiki content...")
console.event(f"{atlas.populate_atlas_feature_names(conn, dry_run)} feature name rows")
# D-242 Phase B (T-1074). SUPERSEDES the retired corp-HQ cross-reference
# (D-207, #909), which inserted one city row per corp HQ with no
# UNIQUE(body_id, name) — ten co-named Groombridge rows on GJ380c. Must
# follow the city pool and the corp-specialization step.
step("Emitting Standalone-HQ settlements + CityTenant links (D-242)...")
hq = corporations.populate_standalone_hq_settlements(conn, dry_run)
console.event(
f"{hq['standalone_inserted']} Standalone-HQ settlements, "
f"{hq['tenant_linked']} CityTenant links ({hq['tenant_unmatched']} unmatched)"
)
# D-242, T-1075 — over the CORRECTED pool, so Standalone-HQ settlements
# are part of the rank-size spread rather than bolted on after.
step("Baking settlement population + settlement_class (D-242)...")
pop = atlas.populate_settlement_population_class(conn, dry_run)
console.event(
f"{pop['cities_populated']} cities populated across {pop['bodies_spread']} bodies, "
f"{pop['name_locked_applied']} NameLocked pins applied"
)
# D-232, #993 — catalog first, then sparse per-body hero bias (FK).
step("Baking trait_templates catalog (D-232)...")
console.event(f"{traits.populate_trait_templates(conn, dry_run)} trait templates")
step("Baking atlas_body_trait_bias hero pins (D-232)...")
console.event(f"{traits.populate_atlas_body_trait_bias(conn, dry_run)} body trait-bias rows")
# T-1024, D-239 §2.
step("Populating axial_tilt_deg from body-def frontmatter...")
console.event(f"{bodies.populate_axial_tilt_deg(conn, dry_run)} bodies updated with axial_tilt_deg")
# D-247, T-1085 — frontmatter override + two-gate default.
step("Populating biosphere_class (D-247)...")
console.event(f"{bodies.populate_biosphere_class(conn, dry_run)} bodies updated with biosphere_class")
# T-988, D-235 — both follow trait_templates (V-TT-06 / V-TT-07 validate
# against it).
step("Baking architecture_zone_bias table (D-235)...")
console.event(f"{traits.populate_architecture_zone_bias(conn, dry_run)} zone-bias rows")
step("Baking color_register_bands table (D-235)...")
console.event(f"{traits.populate_color_register_bands(conn, dry_run)} color register bands")
# Structural integrity (FK, chain refs, chain completeness). These mean
# the imported data is broken — do NOT commit.
step("Validating structural integrity...")
struct_errors = validators.validate(conn) + brands.validate_brands(conn)
if struct_errors:
console.event(f"STRUCTURAL ERRORS ({len(struct_errors)}) — rolling back:", level="error")
for e in struct_errors:
console.event(e, level="error")
raise ImportAborted()
console.event("FK integrity, chain completeness, and brand layer (V-B01–V-B06) OK")
# Commit before the coverage check: coverage is a Phase 2 gate (D-175),
# and the data should be queryable so the gaps can be reported clearly.
if not dry_run:
conn.commit()
console.event("Data committed.")
else:
console.event("Dry run — no changes written.")
if step.n != STEPS:
# A step added without bumping STEPS would make every progress
# fraction wrong without failing anything. Cheap to refuse here.
raise AssertionError(f"STEPS is {STEPS} but the import ran {step.n} steps")
return {
"links": n_links,
"commodities": n_commodities,
"chains": n_chains,
"inputs": n_inputs,
"presence": n_presence,
"brands": n_brands,
"brand_inputs": n_brand_inputs,
"fiscal": n_fiscal,
}
def _coverage(conn: sqlite3.Connection, wiki_corps) -> list[str]:
"""The D-175 Phase 2 gate. Runs after the commit, so the data is usable."""
console.event("Validating coverage (D-175 Phase 2 gate)...")
commodity_ids = {r[0] for r in conn.execute("SELECT commodity_id FROM commodities").fetchall()}
errors = validators.validate_commodity_coverage(conn, wiki_corps, commodity_ids)
errors += validators.validate_system_coverage(conn, wiki_corps)
if not errors:
console.event("All coverage thresholds met — Phase 2 gate PASSED.")
return errors
tooling/economy-db/import_economics.py
-377
View File
@@ -1,377 +0,0 @@
#!/usr/bin/env python3
"""
Import economics data into systems.db.
Reads TOML/JSON source files and populates the economics tables:
- gate_links from docs/design/star-map.json (335 edges, bidirectional)
- commodities from wiki/economics/commodities.toml (36 types)
- production_chains + chain_inputs from wiki/economics/production_chains.toml
- currency_zone on star_systems (default TRACTUS_PRIMARY)
- gate_energy_connected on star_systems (D-186: false for MARK_PRIMARY zones)
- corporations from wiki/corporations/*.md (sync + insert new records)
- corp_specialization/hq_placement/headquarters_body/headquarters_city_id
on corporations (D-242: HQ-placement key + baked
CityTenant link, from corp_hq_placement.toml)
- corp_presence from wiki/corporations/*.md (headquarters location data)
- atlas_city_names Standalone-HQ settlement rows (D-242, T-1074)
Validation (hard errors, non-zero exit on any failure):
- Wiki corporation names must match DB proper_name records (D-182 sync constraint)
- Chain completeness: every intermediate commodity has at least one production chain
- Commodity coverage: 3+ corporations per major commodity type (D-175)
- System coverage: 1+ corporation per inhabited system with population > 100K (D-175)
This file is the CLI entrypoint and single-transaction orchestrator; the import
steps live in the economy_import package (T-1067). The stamped source set is
defined in tooling/generator_sources.py.
Usage:
python3 tooling/economy-db/import_economics.py
python3 tooling/economy-db/import_economics.py --dry-run
python3 tooling/economy-db/import_economics.py --db path/to/systems.db
"""
import argparse
import sys
from pathlib import Path
# tooling/economy-db is not a package (hyphenated dir) — put it on sys.path so
# the economy_import package resolves; its __init__ adds tooling/ for
# generator_sources and schema_version.
sys.path.insert(0, str(Path(__file__).resolve().parent))
from economy_import import ( # noqa: E402
atlas,
bodies,
brands,
corporations,
db,
economy,
migration,
specialization,
stamp,
traits,
validators,
)
from economy_import.errors import ImportAborted # noqa: E402
from economy_import.paths import DB_PATH # noqa: E402
from generator_sources import IMPORT_ECONOMICS_SOURCES # noqa: E402
def main() -> None:
parser = argparse.ArgumentParser(description="Import economics data into systems.db")
parser.add_argument("--db", default=str(DB_PATH), help="Path to systems.db")
parser.add_argument("--dry-run", action="store_true", help="Validate without writing")
parser.add_argument(
"--strict-specialization", action="store_true",
help="Treat D-237 completeness gates (V-SES-02, V-FAC-01) as hard errors. "
"Off by default until the #1014 fallback and #1016 content pass land.",
)
args = parser.parse_args()
db_path = Path(args.db)
if not db_path.exists():
print(f"error: {db_path} not found", file=sys.stderr)
sys.exit(1)
print("\n Economics Import Pipeline")
print(f" DB: {db_path}")
if args.dry_run:
print(" Mode: DRY RUN")
print()
# Load wiki corps before opening DB — allows early exit on parse failures
print(" Loading wiki corporations...")
wiki_corps = corporations.load_wiki_corps()
print(f" {len(wiki_corps)} corporation files parsed")
# Regenerate generated_brands.toml via the Rust binary before the Python
# import reads it. Single pipeline, single stamp — resolves review T2/H3
# ("on-behalf stamping" coupling) by folding brand generation into
# import_economics' flow rather than having the caller (Makefile / user)
# remember to run it first. Skipped on --dry-run to avoid a disk
# side-effect during validation.
if not args.dry_run:
try:
brands.regenerate_brands()
except ImportAborted:
sys.exit(1)
conn = db.connect(db_path)
# The clear-then-reimport cycle below runs as a single explicit
# transaction. Any crash, validation error, or KeyboardInterrupt
# between the first DELETE and the final commit rolls everything
# back — the DB never ends up half-cleared with stale rows in some
# tables and empty rows in others. On success we commit exactly
# once, immediately after structural validation passes.
conn.execute("BEGIN")
try:
# 1. Migrate schema (idempotent, inside the tx so a crash here
# leaves no half-applied ALTER TABLE.)
print(" [1/10] Schema migration...")
migration.apply_schema_migrations(conn)
# Atlas index tables: apply canonical DDL + empty geometry (D-223, #951)
atlas.ensure_atlas_index_schema(conn, args.dry_run)
print(" tables and columns ready")
# Clear economics tables in FK-safe order (children before parents)
if not args.dry_run:
db.clear_economics_tables(conn)
# 2. Gate links
print(" [2/10] Importing gate links...")
n_links = economy.import_gate_links(conn, args.dry_run)
print(f" {n_links} rows (bidirectional)")
# 3. Commodities
print(" [3/10] Importing commodities...")
n_commodities = economy.import_commodities(conn, args.dry_run)
print(f" {n_commodities} commodities")
# 4. Production chains
print(" [4/10] Importing production chains...")
n_chains, n_inputs = economy.import_chains(conn, args.dry_run)
print(f" {n_chains} chains, {n_inputs} inputs")
# 4b. System specialization (D-237 authored layer) — after commodities
# (FK) and before currency zones. UPSERTs onto pre-existing
# system_economy / system_factions rows; unauthored systems stay NULL.
print(" [4b/10] Importing system specialization (D-237)...")
spec = specialization.import_system_specialization(
conn, args.dry_run, strict=args.strict_specialization
)
print(
f" vocab {spec['vocab']} | economic {spec['economic']} | "
f"cultural {spec['cultural']} | faction {spec['faction']}"
)
if spec["missing_economy_row"]:
print(
f" WARNING: {len(spec['missing_economy_row'])} authored "
f"system(s) lack a system_economy row (values dropped): "
f"{spec['missing_economy_row']}"
)
if spec["missing_faction_row"]:
print(
f" WARNING: {len(spec['missing_faction_row'])} authored "
f"system(s) lack a system_factions row (faction dropped): "
f"{spec['missing_faction_row']}"
)
# 5. Currency zones
print(" [5/10] Setting currency zones...")
zones = economy.set_currency_zones(conn, args.dry_run)
for zone, count in sorted(zones.items()):
print(f" {zone}: {count}")
# 6. Gate energy connectivity (D-186) — must run after currency zones
print(" [6/10] Setting gate energy connectivity...")
energy = economy.set_gate_energy(conn, args.dry_run)
for label, count in sorted(energy.items()):
print(f" {label}: {count}")
# 7. Sync corporations from wiki (D-182: hard error on name divergence)
print(" [7/10] Syncing corporations...")
corp_errors = corporations.sync_corporations(conn, wiki_corps, args.dry_run)
if corp_errors:
print(f" ERRORS ({len(corp_errors)}) — name divergence detected (D-182):")
for e in corp_errors:
print(f" - {e}")
print(" Fix: update wiki title or DB proper_name to match, then re-run.")
raise ImportAborted()
n_db_corps = conn.execute("SELECT COUNT(*) FROM corporations").fetchone()[0]
print(f" {n_db_corps} corporations in DB ({len(wiki_corps)} from wiki)")
# 7b. Corp specialization + HQ placement (D-242, T-1074) — Phase A.
# Reset + derive on every run (PR #177 H1/T2): the importer-owned
# corporations columns are NULLed and re-derived from source, never
# kept from a prior run. Must run after corp sync (rows must exist),
# before corp_presence (which reads headquarters_body back off
# corporations), and before step 12 (the H2 city-presence tiebreak
# reads atlas_city_names BEFORE this run's clear/rebuild — the
# previous run's settled state, by design).
print(" [7b/10] Importing corp specialization + HQ placement (D-242)...")
corp_spec = corporations.populate_corp_specialization(conn, wiki_corps, args.dry_run)
print(
f" {corp_spec['specialized']}/{corp_spec['total']} corps specialized, "
f"{corp_spec['hq_resolved']} headquarters_body resolved (recomputed every run; "
f"{corp_spec['hq_overridden']} authored overrides)"
)
# 8. Corp presence from wiki headquarters data
print(" [8/10] Importing corp presence...")
commodity_ids = {
r[0] for r in conn.execute("SELECT commodity_id FROM commodities").fetchall()
}
n_presence = corporations.import_corp_presence(
conn, wiki_corps, commodity_ids, args.dry_run
)
print(f" {n_presence} corp_presence rows")
# 9. Brand products and inputs (D-189, #827)
print(" [9/10] Importing brand products and inputs...")
n_brands, n_brand_inputs = brands.import_brands(conn, args.dry_run)
print(f" {n_brands} brand_products, {n_brand_inputs} brand_inputs")
# 10. System fiscal parameters (D-189 section 6)
print(" [10/13] Populating system_fiscal...")
n_fiscal = economy.import_system_fiscal(conn, args.dry_run)
print(f" {n_fiscal} system_fiscal rows")
# 11. body_radius_km fallback from planet_class (D-204, #910)
print(" [11/13] Populating body_radius_km fallback...")
n_radius = bodies.populate_body_radius_km(conn, args.dry_run)
print(f" {n_radius} bodies updated")
# 12. atlas_city_names from wiki markers.json (D-207, #908)
print(" [12/13] Populating atlas_city_names from wiki content...")
n_cities = atlas.populate_atlas_city_names(conn, args.dry_run)
print(f" {n_cities} city name rows")
# 12b. atlas_feature_names (rivers/mountains) from wiki markers.json
# (D-223, T-1169) — mirrors step 12's pool-load shape exactly, over a
# different names-only markers.json key set. Independent of the
# settlement pool, so order relative to step 13 doesn't matter; placed
# here to stay adjacent to its sibling pool-load step.
print(" [12b/13] Populating atlas_feature_names from wiki content...")
n_features = atlas.populate_atlas_feature_names(conn, args.dry_run)
print(f" {n_features} feature name rows")
# 13. Standalone-HQ settlements + CityTenant city links (D-242, T-1074) — Phase B.
# SUPERSEDES the retired corp-HQ cross-reference (D-207, #909) — that
# step inserted one atlas_city_names row per corp HQ with no
# UNIQUE(body_id, name), producing duplicate co-named "cities" (10
# Groombridge rows on GJ380c). Must run after populate_atlas_city_names
# (the city pool Standalone HQs join and CityTenant HQs tenant) and
# after step 7b (corp_specialization/hq_placement/headquarters_body).
print(" [13/13] Emitting Standalone-HQ settlements + CityTenant links (D-242)...")
hq_settlements = corporations.populate_standalone_hq_settlements(conn, args.dry_run)
print(
f" {hq_settlements['standalone_inserted']} Standalone-HQ settlements, "
f"{hq_settlements['tenant_linked']} CityTenant links "
f"({hq_settlements['tenant_unmatched']} unmatched)"
)
# 13b. Per-settlement population + settlement_class bake (D-242, T-1075).
# Runs over the CORRECTED pool — after both T-1074 steps, so
# Standalone-HQ settlements are included in the rank-size spread, not
# bolted on after.
print(" [13b/13] Baking settlement population + settlement_class (D-242)...")
pop_bake = atlas.populate_settlement_population_class(conn, args.dry_run)
print(
f" {pop_bake['cities_populated']} cities populated across "
f"{pop_bake['bodies_spread']} bodies, {pop_bake['name_locked_applied']} "
f"NameLocked pins applied"
)
# 14. Architecture-flavor trait templates (D-232, #993). Catalog first,
# then sparse per-body hero bias (FK -> trait_templates + bodies).
print(" [14/15] Baking trait_templates catalog (D-232)...")
n_templates = traits.populate_trait_templates(conn, args.dry_run)
print(f" {n_templates} trait templates")
print(" [15/15] Baking atlas_body_trait_bias hero pins (D-232)...")
n_bias = traits.populate_atlas_body_trait_bias(conn, args.dry_run)
print(f" {n_bias} body trait-bias rows")
# 16. axial_tilt_deg from body-def frontmatter (T-1024, D-239 §2)
print(" [16/17] Populating axial_tilt_deg from body-def frontmatter...")
n_tilt = bodies.populate_axial_tilt_deg(conn, args.dry_run)
print(f" {n_tilt} bodies updated with axial_tilt_deg")
# 17. biosphere_class from body frontmatter override + two-gate default (D-247, T-1085)
print(" [17/19] Populating biosphere_class (D-247)...")
n_bio = bodies.populate_biosphere_class(conn, args.dry_run)
print(f" {n_bio} bodies updated with biosphere_class")
# 18. Architecture zone-type bias table (T-988, D-235 step 2) — must
# follow trait_templates (V-TT-06 validates against its visual_bundle).
print(" [18/19] Baking architecture_zone_bias table (D-235)...")
n_zone_bias = traits.populate_architecture_zone_bias(conn, args.dry_run)
print(f" {n_zone_bias} zone-bias rows")
# 19. Color register bands (T-988, D-235) — numeric HSV sampling bands
# per trait-template color_register; also follows trait_templates (V-TT-07).
print(" [19/19] Baking color_register_bands table (D-235)...")
n_color_bands = traits.populate_color_register_bands(conn, args.dry_run)
print(f" {n_color_bands} color register bands")
# Validate structural integrity (FK, chain refs, chain completeness).
# These errors indicate broken imported data — do NOT commit.
print("\n Validating structural integrity...")
struct_errors = validators.validate(conn)
struct_errors.extend(brands.validate_brands(conn))
if struct_errors:
print(f" STRUCTURAL ERRORS ({len(struct_errors)}) — rolling back:")
for e in struct_errors:
print(f" - {e}")
raise ImportAborted()
print(" FK integrity, chain completeness, and brand layer (V-B01–V-B06) OK")
# Commit all imported data (corps, presence, etc.) before coverage check.
# Coverage validation is a Phase 2 gate (D-175) — data should be persisted
# so tools can query it and report gaps clearly.
if not args.dry_run:
conn.commit()
print(" Data committed.")
else:
# Dry-run: leave the transaction open so the coverage check below
# can still SELECT against the in-memory imported data. The
# transaction is discarded when conn.close() runs on exit.
print(" Dry run — no changes written.")
except ImportAborted:
conn.rollback()
conn.close()
sys.exit(1)
except BaseException:
# Any other exception (KeyboardInterrupt, MemoryError, DB error,
# programmer error) triggers a rollback so the DB is never left in
# a half-imported state. Re-raise so the user sees the traceback.
conn.rollback()
conn.close()
raise
# Stamp generator metadata (#855, #856): record source SHAs so the
# pre-push hook can detect stale DB snapshots. Written BEFORE the
# coverage gate — the stamp records generator execution (code version),
# not data completeness. Coverage gaps (#860) are pre-existing data
# issues and must not prevent the stamp from landing.
if not args.dry_run:
try:
stamp.write_stamp(conn, "import_economics", *IMPORT_ECONOMICS_SOURCES)
conn.commit()
print(" Stamped: import_economics (covers brand pipeline Rust sources)")
except Exception as exc: # noqa: BLE001
print(f" WARNING: failed to write generator stamp: {exc}", file=sys.stderr)
# Validate coverage (hard errors per D-175, but after commit so data is usable).
print("\n Validating coverage (D-175 Phase 2 gate)...")
coverage_errors: list[str] = []
commodity_ids_for_coverage = {
r[0] for r in conn.execute("SELECT commodity_id FROM commodities").fetchall()
}
coverage_errors.extend(
validators.validate_commodity_coverage(conn, wiki_corps, commodity_ids_for_coverage)
)
coverage_errors.extend(validators.validate_system_coverage(conn, wiki_corps))
if coverage_errors:
print(f" COVERAGE ERRORS ({len(coverage_errors)}) — Phase 2 gate not met:")
for e in coverage_errors:
print(f" - {e}")
print("\n Data committed but Phase 2 gate is NOT met. "
"Add corporations to meet coverage thresholds and re-run.")
conn.close()
sys.exit(2) # exit 2 = coverage warning (data+stamp committed); exit 1 = real error
else:
print(" All coverage thresholds met — Phase 2 gate PASSED.")
conn.close()
print(f"\n Done: {n_links} gate_links, {n_commodities} commodities, "
f"{n_chains} chains, {n_inputs} inputs, {n_presence} corp_presence, "
f"{n_brands} brand_products, {n_brand_inputs} brand_inputs, "
f"{n_fiscal} system_fiscal\n")
if __name__ == "__main__":
main()
tooling/generator_sources.py
+15 -14
View File
@@ -5,20 +5,21 @@ generator_sources — single source of truth for systems.db generator source set
Each generator that writes a stamp row to systems.db's ``meta`` table records the
SHA-1 of the concatenated bytes of its source files (sorted by path). This module
is the ONE place that defines which files belong to each generator's stamped set
(T-1067). It replaces the formerly triplicated lists in:
- ``IMPORT_ECONOMICS_SOURCES`` in tooling/economy-db/import_economics.py
- ``GENERATOR_SOURCES`` in tooling/check-systems-db-stamp
- the hardcoded watch list in .claude/skills/pr-process/SKILL.md
(now derived via ``python3 tooling/generator_sources.py --list``)
(T-1067). It replaced three formerly separate lists (in the importer, in the
old check-systems-db-stamp script, and in the pr-process skill).
Consumers:
- tooling/economy-db (the importer) — imports ``IMPORT_ECONOMICS_SOURCES`` and
``file_sha1`` to write the stamp after a successful run.
- tooling/check-systems-db-stamp — imports ``GENERATOR_SOURCES`` and
- the ledger importer (tooling/domains/ledger/) — imports
``IMPORT_ECONOMICS_SOURCES`` and ``file_sha1`` to write the stamp after a
successful run.
- ``reach check systems-db-stamp`` — imports ``GENERATOR_SOURCES`` and
``file_sha1`` to verify the stamp before a push.
- ``reach pr watchlist-diff`` — reads ``GENERATOR_SOURCES`` for its watch set.
- /pr-process (skill) — shells out to the ``--list`` CLI for its watch list.
Stays a module rather than a verb (tooling/DOMAINS.md): it is a list of paths
read by gates, and nothing types it.
This file is itself a member of the stamped source set: a registry change (adding
or removing a source) must flip the stamp and force ``make regen-db``, otherwise
edits to the list would be invisible to staleness detection.
@@ -44,10 +45,10 @@ REPO_ROOT = Path(__file__).resolve().parent.parent
# Entrypoint + split module package (T-1067).
IMPORT_ECONOMICS_ENTRYPOINT: Path = (
REPO_ROOT / "tooling" / "economy-db" / "import_economics.py"
REPO_ROOT / "tooling" / "domains" / "ledger" / "service.py"
)
ECONOMY_IMPORT_PACKAGE_DIR: Path = (
REPO_ROOT / "tooling" / "economy-db" / "economy_import"
REPO_ROOT / "tooling" / "domains" / "ledger" / "economy_import"
)
# Rust sources for the generate_brands subroutine. import_economics runs the
@@ -163,7 +164,7 @@ def _corporation_pages() -> tuple[Path, ...]:
# Canonical source set for import_economics' meta stamp. Covers the Python
# entrypoint and its module package, the Rust binary it invokes (generate_brands
# main.rs + names.rs + surname corpus + wrapper script), the shared schema
# main.rs + names.rs + surname corpus + core/process.py), the shared schema
# version constant, the authored data TOMLs, the corp wiki pages whose
# frontmatter the importer bakes (T1, PR #177), and this registry itself.
IMPORT_ECONOMICS_SOURCES: tuple[Path, ...] = (
@@ -175,7 +176,7 @@ IMPORT_ECONOMICS_SOURCES: tuple[Path, ...] = (
GENERATE_BRANDS_NAMES_RS,
GENERATE_BRANDS_SURNAMES_RS,
REPO_ROOT / "tooling" / "core" / "process.py", # runs the brand binary (T-1286)
REPO_ROOT / "tooling" / "schema_version.py",
REPO_ROOT / "tooling" / "domains" / "ledger" / "schema_version.py",
SPECIALIZATION_VOCAB_TOML,
SYSTEM_SPECIALIZATION_TOML,
CORP_HQ_PLACEMENT_TOML,
@@ -198,7 +199,7 @@ IMPORT_ECONOMICS_SOURCES: tuple[Path, ...] = (
# The atlas geometry generator (generate_atlas) was retired in #951 (D-223).
# import_economics is now the sole regen-db generator that writes systems.db; it
# owns the atlas index (names-only pool + empty geometry tables). The surviving
# planet-gen importers (import_heightmaps, import_province_boundaries) are
# `reach atlas planet` importers (import-heightmaps, import-provinces) are
# one-time build imports baked into the committed DB, not part of regen-db, so
# they are intentionally not stamped here.
GENERATOR_SOURCES: dict[str, tuple[Path, ...]] = {
tooling/main.py
+4
View File
@@ -59,6 +59,10 @@ DOMAINS: dict[str, tuple[str, str]] = {
"tooling.domains.atlas.router:app",
"The spatial ladder — authoring, inspection and the DB",
),
"ledger": (
"tooling.domains.ledger.router:app",
"The economics pipeline — the import that builds systems.db",
),
"godot": (
"tooling.domains.godot.router:app",
"Does the client parse, and does it parse cold",
tooling/economy-db/test_atlas_idempotency.py → tooling/test_ledger_atlas_idempotency.py
+4 -5
View File
@@ -17,14 +17,14 @@ would drift from the real FK web this populator depends on). Wiki content
(`wiki/star-systems/*/bodies/*/markers.json`) is read directly from the repo
— read-only input, safe to reuse as-is; only the DB connection is scratch.
Deliberately calls the two populator functions directly rather than the full
`import_economics.py` CLI: the CLI's first step shells out to the Rust
`reach ledger import`: its first step shells out to the Rust
`generate_brands` binary and overwrites `generated_brands.toml` on disk
(`brands.regenerate_brands`), a real side effect on shared repo content that
has nothing to do with atlas name-pool idempotency and would make this test
depend on a Rust build.
Stdlib only (unittest) — run directly or via `make test-tooling`:
python3 tooling/economy-db/test_atlas_idempotency.py
.venv/bin/python tooling/test_ledger_atlas_idempotency.py
"""
import shutil
@@ -34,11 +34,10 @@ import tempfile
import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from economy_import import atlas # noqa: E402
from economy_import.paths import DB_PATH # noqa: E402
from tooling.domains.ledger.economy_import import atlas # noqa: E402
from tooling.domains.ledger.economy_import.paths import DB_PATH # noqa: E402
class AtlasNamePoolIdempotencyTests(unittest.TestCase):
tooling/economy-db/test_traits.py → tooling/test_ledger_traits.py
+7 -8
View File
@@ -9,7 +9,7 @@ shape/coverage checks (T-988) — the `make test-tooling` dry-run only
exercises the happy path against the committed, already-valid content.
Stdlib only (unittest) — run directly or via `make test-tooling`:
python3 tooling/economy-db/test_traits.py
.venv/bin/python tooling/test_ledger_traits.py
"""
import json
@@ -19,11 +19,10 @@ import tempfile
import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from economy_import import traits # noqa: E402
from economy_import.errors import ImportAborted # noqa: E402
from tooling.domains.ledger.economy_import import traits # noqa: E402
from tooling.domains.ledger.economy_import.errors import ImportAborted # noqa: E402
def write_registry(tmp: Path, body: str) -> Path:
@@ -247,7 +246,7 @@ class CatalogCrossCheckTests(unittest.TestCase):
catalog.write_text(catalog_toml, encoding="utf-8")
traits.ARCHITECTURE_TRAIT_CATALOG_TOML = catalog
out = io.StringIO()
with contextlib.redirect_stdout(out):
with contextlib.redirect_stderr(out): # console events go to stderr (T-1289)
with self.assertRaises(ImportAborted):
traits.populate_trait_templates(self.conn, dry_run=True)
return out.getvalue()
@@ -365,7 +364,7 @@ class ZoneBiasValidationTests(unittest.TestCase):
import io
out = io.StringIO()
with contextlib.redirect_stdout(out):
with contextlib.redirect_stderr(out): # console events go to stderr (T-1289)
with self.assertRaises(ImportAborted):
self.populate(
'[bias.test_template.industrial_freight]\n'
@@ -397,7 +396,7 @@ class ZoneBiasValidationTests(unittest.TestCase):
for bad in ("wall = 15000\n", 'wall = ["brick_wall"]\n'):
out = io.StringIO()
with contextlib.redirect_stdout(out):
with contextlib.redirect_stderr(out): # console events go to stderr (T-1289)
with self.assertRaises(ImportAborted):
self.populate(f"[bias.test_template.commercial_market]\n{bad}")
self.assertIn("V-TT-06", out.getvalue())
@@ -461,7 +460,7 @@ class ColorRegisterBandTests(unittest.TestCase):
import io
out = io.StringIO()
with contextlib.redirect_stdout(out):
with contextlib.redirect_stderr(out): # console events go to stderr (T-1289)
with self.assertRaises(ImportAborted):
# A band for a DIFFERENT register — neutral_grey (referenced by
# test_template above) has no band at all.