diff --git a/.claude/rules/asset-pipeline.md b/.claude/rules/asset-pipeline.md index 7909e6c32..645539824 100644 --- a/.claude/rules/asset-pipeline.md +++ b/.claude/rules/asset-pipeline.md @@ -21,7 +21,7 @@ migration — those changes will be silently overwritten by the next `make regen ## What produces systems.db **One generator** writes to `systems.db`: `import_economics` -(`python3 tooling/economy-db/import_economics.py`, run via `make regen-db`). +(`reach ledger import`, run via `make regen-db`). The former atlas geometry generator (`generate_atlas`) was retired in #951 (D-223); `import_economics` now also owns the atlas index — it loads the names-only `markers.json` city pool into `atlas_city_names` and empties the @@ -36,9 +36,9 @@ itself didn't change. The full set of source files contributing to the meta stamp SHA (the Python importer entrypoint + its `economy_import` module package, the Rust brand -binary sources, `tooling/schema_version.py`, the authored economics TOMLs, and +binary sources, `tooling/domains/ledger/schema_version.py`, the authored economics TOMLs, and the registry itself) is defined once in `tooling/generator_sources.py` (T-1067) -— imported by both the importer's stamp writer and `check-systems-db-stamp`, +— imported by both the importer's stamp writer and `reach check systems-db-stamp`, and listed via `python3 tooling/generator_sources.py --list`. The surviving `reach atlas planet` importers (`import-heightmaps`, @@ -62,7 +62,7 @@ CREATE TABLE meta ( ``` `schema_version` is a **monotonic semver string** (e.g. `"1.0.0"`), not a hash. -It is defined as the `SCHEMA_VERSION` constant in `tooling/schema_version.py` +It is defined as the `SCHEMA_VERSION` constant in `tooling/domains/ledger/schema_version.py` and must be bumped manually whenever the schema changes in a backwards-incompatible way. Unlike a SHA-1 hash, semver strings are orderable — this enables savegame migration lineage in Phase 5+: a save file can record which schema version it derives from and @@ -89,7 +89,7 @@ fine: the freshness guarantee comes from the stamp, not from bytewise DB equalit 1. Edit the source files (TOML, JSON, markers.json). 2. Run `make regen-db`. -3. Run `make check-systems-db` to confirm the stamp is fresh. +3. Run `reach check systems-db-stamp` to confirm the stamp is fresh. 4. Stage and commit: ```bash git add server/data/systems.db @@ -99,7 +99,7 @@ fine: the freshness guarantee comes from the stamp, not from bytewise DB equalit ### Schema changes (new tables or columns) 1. Add the DDL to `server/data/systems-schema.sql`. -2. Add migration SQL to `MIGRATION_SQL` in `import_economics.py` if the change +2. Add migration SQL to `MIGRATION_SQL` in `economy_import/migration.py` if the change affects existing DBs (idempotent `CREATE TABLE IF NOT EXISTS` or `ALTER TABLE`). 3. Run `make regen-db`. 4. Stage `server/data/systems-schema.sql` and `server/data/systems.db` together. @@ -121,8 +121,8 @@ Fix: run `make regen-db`, stage `server/data/systems.db`, amend or add a commit. Or use `/pr-process` — it detects stale generator sources and reruns `make regen-db` automatically before pushing. -The check script is `tooling/check-systems-db-stamp`. Run it interactively with -`make check-systems-db` or `python3 tooling/check-systems-db-stamp --verbose`. The +The check is `reach check systems-db-stamp` (`tooling/domains/check/`), run by +the push gate and by hand the same way. The `GENERATOR_SOURCES` dict in `tooling/generator_sources.py` is the single registry (T-1067) — the check script and the importer's stamp writer both import it, and the `/pr-process` skill derives its source-file watch list from @@ -156,13 +156,13 @@ The rule above says "never run UPDATE or INSERT directly on systems.db outside of a migration." Here's what a legitimate migration looks like, and what isn't one: -**Sanctioned path: the `MIGRATION_SQL` block in `import_economics.py`.** That +**Sanctioned path: the `MIGRATION_SQL` block in `tooling/domains/ledger/economy_import/migration.py`.** That string is executed at the top of every import run (inside the same transaction that clears + reimports data) and contains idempotent `CREATE TABLE IF NOT EXISTS` / `CREATE INDEX IF NOT EXISTS` statements, plus `ALTER TABLE` additions handled via the `COLUMN_MIGRATIONS` list. When you need a new table, column, or index on systems.db, add it there. It'll run on the next `make regen-db` -and the meta stamp will flip because `import_economics.py` changed. +and the meta stamp will flip because `migration.py` is a stamped source. **Also legitimate:** edits to `server/data/systems-schema.sql` (the canonical DDL used by fresh builds) paired with matching entries in `MIGRATION_SQL` for @@ -195,6 +195,6 @@ loader can determine which migrations to apply by comparing that version to the current one. `meta.schema_sha` retains the old SHA-1 for tamper detection. **When to bump `SCHEMA_VERSION`:** edit the `SCHEMA_VERSION = "1.0.0"` constant in -`tooling/schema_version.py` whenever a schema change is backwards-incompatible +`tooling/domains/ledger/schema_version.py` whenever a schema change is backwards-incompatible (column removed, type changed, FK constraint added, table dropped). Additive changes (new nullable columns, new tables, new indexes) do not require a bump. diff --git a/.claude/skills/atlas/SKILL.md b/.claude/skills/atlas/SKILL.md index 97e8dcfd8..71594b96a 100644 --- a/.claude/skills/atlas/SKILL.md +++ b/.claude/skills/atlas/SKILL.md @@ -217,7 +217,7 @@ See `.claude/rules/asset-pipeline.md` (the "Golden Rule": edit sources, not `systems.db`) and D-223. The bodies/stations catalog this skill authors is **atlas-CLI-owned** and survives `make regen-db`: `import_economics` only *enriches* existing body rows (radius, axial tilt, biosphere class — see -`tooling/economy-db/economy_import/bodies.py`), it never deletes or +`tooling/domains/ledger/economy_import/bodies.py`), it never deletes or regenerates the catalog. What `import_economics` *does* clear on every regen is the separate `atlas_*` geometry index tables (`atlas_cities`, `atlas_roads`, etc., D-223) — those hold cascade-computed geometry, not the diff --git a/.gitignore b/.gitignore index 3b1b91500..f36ecbeaa 100644 --- a/.gitignore +++ b/.gitignore @@ -100,7 +100,7 @@ Thumbs.db *.swp *.swo -# Generated economics pipeline artifacts (re-created by make economy-db) +# Generated economics pipeline artifacts (re-created by make regen-db) wiki/economics/corporations/generated_brands.toml wiki/economics/corporations/generated_corporations.toml diff --git a/.pql/changelog/ticket_history/2026-09.sql b/.pql/changelog/ticket_history/2026-09.sql index 174a766ff..c7e1edf96 100644 --- a/.pql/changelog/ticket_history/2026-09.sql +++ b/.pql/changelog/ticket_history/2026-09.sql @@ -345,3 +345,61 @@ nothing prints but `console`, no `subprocess` outside `core/process`. DONE 2026-09-23. reach atlas planet: ten verbs over the moved package; tooling/test_planet_router.py (in make test-tooling) fails if a router option drifts from its module''s argparse. Finishing the 2026-09-02 half-move found: (1) lazy in-function imports and all of sol_data/ still used bare sibling names that only resolved via sys.path hacks — generate/batch/sol-import would have failed on first globe render under reach; qualified, hacks removed. (2) 247 print() + a per-8KB-block stdout progress writer routed through console (report verbs -> out, progress -> event, throttled download progress). (3) Every error exit now raises ReachError with fix=. (4) TWO CHECKS THAT COULD NOT FAIL: batch --verify-determinism and import-provinces both exited 0 on detected failures; both now raise. (5) sol-import --body is action=append but the router took one value — now repeatable. (6) test_conformance only walked one level, so the nested group''s ten verbs were never checked; now recursive, proven against a mutant. Stray PNGs from the 2026-09-03 runaway test run (8 modified reliefmaps, 6 new heightmaps) parked in .cache/t1288-stray-pngs/, reliefmaps restored from HEAD. NOTE: the reliefmaps differed from the committed bake, i.e. the current renderer no longer reproduces committed reliefmap bytes — not investigated here.', NULL, '2026-09-23 14:07:42', '2026-09-23 14:07:42.675', '2026-09-23 14:07:42.675', NULL, 'b6c6d1e211eada353630e3946da96310', 2) ON CONFLICT(hash) DO NOTHING; INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G65NSJ4RG6SJ12TQE78S6TKR', 'status', 'backlog', 'done', NULL, '2026-09-23 14:07:43', '2026-09-23 14:07:43.036', '2026-09-23 14:07:43.036', NULL, 'e3329ba1f64d0a63c635d4a9049d56b6', 2) ON CONFLICT(hash) DO NOTHING; +INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G65T2N1WAT491WY8SV736J3G', 'description', '`tooling/economy-db/` (17 files) + `schema_version.py` become `reach ledger`. + +The name is deliberate and is the user''s call (D-263 amendment): reach verbs +mirror the game''s implant-app presentation patterns. `ledger` is the player-facing +component that will aggregate economics — browse markets, wealth, transactions — +the way `atlas` aggregates topography. The tooling verb takes the same name so the +CLI and the UI do not diverge into two vocabularies for one subject. + +THIS IS THE HIGHEST-RISK PORT IN THE EPIC and the reason E5 (T-1252) exists as a +separate epic: `import_economics` is the SOLE generator of `server/data/systems.db`, +and its meta stamp is computed over a source-file registry. Two things bite: + +1. `tooling/generator_sources.py` lists every stamped path. Moving a file without + updating the registry either breaks the import (missing path) or silently + changes the SHA. T-1286 already had to swap the retired `generate-brands` + wrapper for `core/process.py` there and regen. Expect the same. +2. `economy-db` is hyphenated, so it is NOT importable as a package. `brands.py` + currently bootstraps `sys.path` with REPO_ROOT to reach `tooling.core` — see + the comment there. That bootstrap should DISAPPEAR in this port, not be + copied; it exists only because the directory could not be imported. Overlaps + T-1272 (hyphen sweep) — do them together or in that order. + +Every port of this domain ends with `make regen-db` + `reach check systems-db-stamp` ++ a commit of `server/data/systems.db`. `generated_brands.toml` must come back +byte-identical; if it does not, the port changed generator behaviour and that is +a bug, not a regen. + +Standard port acceptance as on T-1281/T-1286.', '`tooling/economy-db/` (17 files) + `schema_version.py` become `reach ledger`. + +The name is deliberate and is the user''s call (D-263 amendment): reach verbs +mirror the game''s implant-app presentation patterns. `ledger` is the player-facing +component that will aggregate economics — browse markets, wealth, transactions — +the way `atlas` aggregates topography. The tooling verb takes the same name so the +CLI and the UI do not diverge into two vocabularies for one subject. + +THIS IS THE HIGHEST-RISK PORT IN THE EPIC and the reason E5 (T-1252) exists as a +separate epic: `import_economics` is the SOLE generator of `server/data/systems.db`, +and its meta stamp is computed over a source-file registry. Two things bite: + +1. `tooling/generator_sources.py` lists every stamped path. Moving a file without + updating the registry either breaks the import (missing path) or silently + changes the SHA. T-1286 already had to swap the retired `generate-brands` + wrapper for `core/process.py` there and regen. Expect the same. +2. `economy-db` is hyphenated, so it is NOT importable as a package. `brands.py` + currently bootstraps `sys.path` with REPO_ROOT to reach `tooling.core` — see + the comment there. That bootstrap should DISAPPEAR in this port, not be + copied; it exists only because the directory could not be imported. Overlaps + T-1272 (hyphen sweep) — do them together or in that order. + +Every port of this domain ends with `make regen-db` + `reach check systems-db-stamp` ++ a commit of `server/data/systems.db`. `generated_brands.toml` must come back +byte-identical; if it does not, the port changed generator behaviour and that is +a bug, not a regen. + +Standard port acceptance as on T-1281/T-1286. + +DONE 2026-09-23. tooling/economy-db/ + schema_version.py -> tooling/domains/ledger/ (economy_import/ kept by name, since server/src Rust comments cite economy_import/*.py; import_economics.py became service.py). reach ledger import [--db] [--dry-run] [--strict-specialization]. Evidence: generated_brands.toml sha256 identical before/after (e748531…); stamp gate reported STALE after the move and OK after regen; dry-run transcript carries every count and warning of the baseline; exit 2 (coverage gap, data committed) proven to relay through @command with a probe. Make: regen-db kept as a one-line delegate per D-263''s muscle-memory clause (~50 files name it, incl. generated TOML headers and gate remedies); economy-db retired (it ran generate brands, which the import already does first). Reconciled economy_import.errors (DOMAINS.md ask): ImportAborted stays internal rollback control flow, converted to ReachError at the service boundary. Also: regenerate_brands used to swallow cargo_binary''s ReachError into ImportAborted, dropping its remedy — now propagates (it runs before the transaction, nothing to roll back). Both sys.path bootstraps removed. Step labels were inconsistent ([1/10]..[10/13]..[17/19]); one 24-step counter now drives phase + progress. Stale doc pointers fixed while there: MIGRATION_SQL had lived in economy_import/migration.py since T-1067 but rules/DEVOPS still said import_economics.py; asset-pipeline rule and DEVOPS still named the check-systems-db script/target T-1281 retired.', NULL, '2026-09-23 14:18:49', '2026-09-23 14:18:49.681', '2026-09-23 14:18:49.681', NULL, '13300d7ea5bd79ae4857cd59399531a6', 2) ON CONFLICT(hash) DO NOTHING; +INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G65T2N1WAT491WY8SV736J3G', 'status', 'backlog', 'done', NULL, '2026-09-23 14:18:50', '2026-09-23 14:18:50.093', '2026-09-23 14:18:50.093', NULL, '63d24743b229b1544d62104c4e68ca66', 2) ON CONFLICT(hash) DO NOTHING; diff --git a/.pql/changelog/tickets/2026-09.sql b/.pql/changelog/tickets/2026-09.sql index b7285299e..66115b3b6 100644 --- a/.pql/changelog/tickets/2026-09.sql +++ b/.pql/changelog/tickets/2026-09.sql @@ -501,3 +501,63 @@ nothing prints but `console`, no `subprocess` outside `core/process`. DONE 2026-09-23. reach atlas planet: ten verbs over the moved package; tooling/test_planet_router.py (in make test-tooling) fails if a router option drifts from its module''s argparse. Finishing the 2026-09-02 half-move found: (1) lazy in-function imports and all of sol_data/ still used bare sibling names that only resolved via sys.path hacks — generate/batch/sol-import would have failed on first globe render under reach; qualified, hacks removed. (2) 247 print() + a per-8KB-block stdout progress writer routed through console (report verbs -> out, progress -> event, throttled download progress). (3) Every error exit now raises ReachError with fix=. (4) TWO CHECKS THAT COULD NOT FAIL: batch --verify-determinism and import-provinces both exited 0 on detected failures; both now raise. (5) sol-import --body is action=append but the router took one value — now repeatable. (6) test_conformance only walked one level, so the nested group''s ten verbs were never checked; now recursive, proven against a mutant. Stray PNGs from the 2026-09-03 runaway test run (8 modified reliefmaps, 6 new heightmaps) parked in .cache/t1288-stray-pngs/, reliefmaps restored from HEAD. NOTE: the reliefmaps differed from the committed bake, i.e. the current renderer no longer reproduces committed reliefmap bytes — not investigated here.', 'done', 'high', NULL, 'tooling', 'D-263', '2026-09-02 15:57:45.382', '2026-09-23 14:07:43.036', NULL, '2a78f8f8646d915600ed55c5dc5d0993', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at; INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06GCX60QRMD7KJ1TVWTR7FWK28', 'bug', '06FBPPMZNNEV052DBYYY3A897C', 'Planet renderer no longer reproduces committed reliefmap.png bytes (8 bodies differ on re-render)', 'Found during T-1288. A 2026-09-03 accidental generate run re-rendered bodies in GJ-1002, GJ-1116B, GJ-139, GJ-147, GJ-216A, GJ-234A, GJ-273, GJ-282B: every reliefmap.png came back a few hundred bytes different from HEAD, while GJ1002b''s heightmap.png came back byte-identical (so the simulation is deterministic — make test-tooling confirms — and the drift is in the reliefmap RENDER path or its PNG encoding). Copies of the re-rendered files are in .cache/t1288-stray-pngs/. Question to answer first: is this a visible change (renderer code evolved since the April bake) or encoder-level noise (zlib/Pillow version, metadata)? Pixel-diff one pair before anything else. If visible: decide whether committed reliefmaps should be re-baked. If noise: make the encode deterministic so a re-render is a no-op in git.', 'backlog', 'low', NULL, 'tooling', NULL, '2026-09-23 14:07:49.189', '2026-09-23 14:07:49.189', NULL, '215478973901c77cdd0e6b5c446ff19e', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at; +INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G65T2N1WAT491WY8SV736J3G', 'task', '06G1S3D0M1TQW0GMFBBPQZG3ZM', 'Port ledger — the economics pipeline under its UI name', '`tooling/economy-db/` (17 files) + `schema_version.py` become `reach ledger`. + +The name is deliberate and is the user''s call (D-263 amendment): reach verbs +mirror the game''s implant-app presentation patterns. `ledger` is the player-facing +component that will aggregate economics — browse markets, wealth, transactions — +the way `atlas` aggregates topography. The tooling verb takes the same name so the +CLI and the UI do not diverge into two vocabularies for one subject. + +THIS IS THE HIGHEST-RISK PORT IN THE EPIC and the reason E5 (T-1252) exists as a +separate epic: `import_economics` is the SOLE generator of `server/data/systems.db`, +and its meta stamp is computed over a source-file registry. Two things bite: + +1. `tooling/generator_sources.py` lists every stamped path. Moving a file without + updating the registry either breaks the import (missing path) or silently + changes the SHA. T-1286 already had to swap the retired `generate-brands` + wrapper for `core/process.py` there and regen. Expect the same. +2. `economy-db` is hyphenated, so it is NOT importable as a package. `brands.py` + currently bootstraps `sys.path` with REPO_ROOT to reach `tooling.core` — see + the comment there. That bootstrap should DISAPPEAR in this port, not be + copied; it exists only because the directory could not be imported. Overlaps + T-1272 (hyphen sweep) — do them together or in that order. + +Every port of this domain ends with `make regen-db` + `reach check systems-db-stamp` ++ a commit of `server/data/systems.db`. `generated_brands.toml` must come back +byte-identical; if it does not, the port changed generator behaviour and that is +a bug, not a regen. + +Standard port acceptance as on T-1281/T-1286. + +DONE 2026-09-23. tooling/economy-db/ + schema_version.py -> tooling/domains/ledger/ (economy_import/ kept by name, since server/src Rust comments cite economy_import/*.py; import_economics.py became service.py). reach ledger import [--db] [--dry-run] [--strict-specialization]. Evidence: generated_brands.toml sha256 identical before/after (e748531…); stamp gate reported STALE after the move and OK after regen; dry-run transcript carries every count and warning of the baseline; exit 2 (coverage gap, data committed) proven to relay through @command with a probe. Make: regen-db kept as a one-line delegate per D-263''s muscle-memory clause (~50 files name it, incl. generated TOML headers and gate remedies); economy-db retired (it ran generate brands, which the import already does first). Reconciled economy_import.errors (DOMAINS.md ask): ImportAborted stays internal rollback control flow, converted to ReachError at the service boundary. Also: regenerate_brands used to swallow cargo_binary''s ReachError into ImportAborted, dropping its remedy — now propagates (it runs before the transaction, nothing to roll back). Both sys.path bootstraps removed. Step labels were inconsistent ([1/10]..[10/13]..[17/19]); one 24-step counter now drives phase + progress. Stale doc pointers fixed while there: MIGRATION_SQL had lived in economy_import/migration.py since T-1067 but rules/DEVOPS still said import_economics.py; asset-pipeline rule and DEVOPS still named the check-systems-db script/target T-1281 retired.', 'backlog', 'high', NULL, 'tooling', 'D-263', '2026-09-02 16:16:28.431', '2026-09-23 14:18:49.681', NULL, '4690f4d637a68bb49840c7a2de30801a', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at; +INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G65T2N1WAT491WY8SV736J3G', 'task', '06G1S3D0M1TQW0GMFBBPQZG3ZM', 'Port ledger — the economics pipeline under its UI name', '`tooling/economy-db/` (17 files) + `schema_version.py` become `reach ledger`. + +The name is deliberate and is the user''s call (D-263 amendment): reach verbs +mirror the game''s implant-app presentation patterns. `ledger` is the player-facing +component that will aggregate economics — browse markets, wealth, transactions — +the way `atlas` aggregates topography. The tooling verb takes the same name so the +CLI and the UI do not diverge into two vocabularies for one subject. + +THIS IS THE HIGHEST-RISK PORT IN THE EPIC and the reason E5 (T-1252) exists as a +separate epic: `import_economics` is the SOLE generator of `server/data/systems.db`, +and its meta stamp is computed over a source-file registry. Two things bite: + +1. `tooling/generator_sources.py` lists every stamped path. Moving a file without + updating the registry either breaks the import (missing path) or silently + changes the SHA. T-1286 already had to swap the retired `generate-brands` + wrapper for `core/process.py` there and regen. Expect the same. +2. `economy-db` is hyphenated, so it is NOT importable as a package. `brands.py` + currently bootstraps `sys.path` with REPO_ROOT to reach `tooling.core` — see + the comment there. That bootstrap should DISAPPEAR in this port, not be + copied; it exists only because the directory could not be imported. Overlaps + T-1272 (hyphen sweep) — do them together or in that order. + +Every port of this domain ends with `make regen-db` + `reach check systems-db-stamp` ++ a commit of `server/data/systems.db`. `generated_brands.toml` must come back +byte-identical; if it does not, the port changed generator behaviour and that is +a bug, not a regen. + +Standard port acceptance as on T-1281/T-1286. + +DONE 2026-09-23. tooling/economy-db/ + schema_version.py -> tooling/domains/ledger/ (economy_import/ kept by name, since server/src Rust comments cite economy_import/*.py; import_economics.py became service.py). reach ledger import [--db] [--dry-run] [--strict-specialization]. Evidence: generated_brands.toml sha256 identical before/after (e748531…); stamp gate reported STALE after the move and OK after regen; dry-run transcript carries every count and warning of the baseline; exit 2 (coverage gap, data committed) proven to relay through @command with a probe. Make: regen-db kept as a one-line delegate per D-263''s muscle-memory clause (~50 files name it, incl. generated TOML headers and gate remedies); economy-db retired (it ran generate brands, which the import already does first). Reconciled economy_import.errors (DOMAINS.md ask): ImportAborted stays internal rollback control flow, converted to ReachError at the service boundary. Also: regenerate_brands used to swallow cargo_binary''s ReachError into ImportAborted, dropping its remedy — now propagates (it runs before the transaction, nothing to roll back). Both sys.path bootstraps removed. Step labels were inconsistent ([1/10]..[10/13]..[17/19]); one 24-step counter now drives phase + progress. Stale doc pointers fixed while there: MIGRATION_SQL had lived in economy_import/migration.py since T-1067 but rules/DEVOPS still said import_economics.py; asset-pipeline rule and DEVOPS still named the check-systems-db script/target T-1281 retired.', 'done', 'high', NULL, 'tooling', 'D-263', '2026-09-02 16:16:28.431', '2026-09-23 14:18:50.093', NULL, '579a9be46ac9f07e3b7a687fbe4772fe', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at; diff --git a/Makefile b/Makefile index 15d9fb46a..60e557f8c 100644 --- a/Makefile +++ b/Makefile @@ -4,7 +4,7 @@ GODOT := $(shell command -v godot4 2>/dev/null || command -v godot 2>/dev/null) install-reach reach-repoint \ decisions-sync decisions-active decisions-validate \ setup-hooks install-hooks \ - audit deny economy-db regen-db \ + audit deny regen-db \ pre-pr pre-pr-lint pre-pr-build pre-pr-test pre-pr-validate pre-pr-fixtures \ pre-pr-server pre-pr-client pre-pr-content \ fixtures-client fixtures-gauntlet golden-diff golden-update \ @@ -57,8 +57,7 @@ help: @echo " make deny Run cargo deny check (license/ban policy)" @echo " make star-map-data Regenerate client/data/star_map_data.json from systems.db + wiki" @echo " make check-star-map Assert star_map_data.json is up to date (part of pre-pr-client)" - @echo " make economy-db Import economics data into systems.db (TOML/JSON → SQLite)" - @echo " make regen-db Regenerate systems.db from all sources + stamp meta table (#855)" + @echo " make regen-db Regenerate systems.db + stamp meta table (= reach ledger import)" @echo " make install-hooks Install pre-push + pre-commit git hooks (once per clone)" @echo " make fixtures-client Generate GDScript->Rust cross-encoder fixtures (#475)" @echo " make golden-diff Show diff if golden file output has changed" @@ -280,7 +279,7 @@ VENV_PY := $(shell test -x .venv/bin/python && echo .venv/bin/python || echo pyt # 1. atlas planet determinism guard (#963): simulating the same body twice must # be bit-identical (guards the expensive 271-body heightmap bake). # Exit 2 = no testable body found — warn, don't block. -# 2. import_economics --dry-run against the committed DB: full parse + +# 2. reach ledger import --dry-run against the committed DB: full parse + # structural/coverage validation, no writes. Exit 2 = coverage-gate # warning (D-175) — tolerated, matching regen-db's treatment. test-tooling: @@ -333,20 +332,20 @@ test-tooling: @mkdir -p .cache @$(VENV_PY) tooling/test_check.py 2> .cache/test-tooling-check.log || \ { echo " FAIL: check gates — log follows:"; cat .cache/test-tooling-check.log; exit 1; } - @echo " [test-tooling] economy_import.traits validation units (T-995/PR #173 H2)..." + @echo " [test-tooling] ledger traits validation units (T-995/PR #173 H2)..." @mkdir -p .cache - @python3 tooling/economy-db/test_traits.py 2> .cache/test-tooling-traits.log || \ + @$(VENV_PY) tooling/test_ledger_traits.py 2> .cache/test-tooling-traits.log || \ { echo " FAIL: traits validation units — log follows:"; cat .cache/test-tooling-traits.log; exit 1; } @echo " [test-tooling] atlas_city_names/atlas_feature_names idempotency (T-964)..." - @python3 tooling/economy-db/test_atlas_idempotency.py 2> .cache/test-tooling-atlas-idempotency.log || \ + @$(VENV_PY) tooling/test_ledger_atlas_idempotency.py 2> .cache/test-tooling-atlas-idempotency.log || \ { echo " FAIL: atlas name-pool idempotency — log follows:"; cat .cache/test-tooling-atlas-idempotency.log; exit 1; } - @echo " [test-tooling] import_economics --dry-run (committed DB)..." - @rc=0; python3 tooling/economy-db/import_economics.py --dry-run \ + @echo " [test-tooling] reach ledger import --dry-run (committed DB)..." + @rc=0; reach ledger import --dry-run \ > .cache/test-tooling-dryrun.log 2>&1 || rc=$$?; \ if [ $$rc -eq 2 ]; then \ echo " WARNING: coverage gate warning (exit 2) — not blocking (matches regen-db)"; \ elif [ $$rc -ne 0 ]; then \ - echo " FAIL: import_economics --dry-run exited $$rc — log follows:"; \ + echo " FAIL: reach ledger import --dry-run exited $$rc — log follows:"; \ cat .cache/test-tooling-dryrun.log; exit $$rc; \ fi @echo " test-tooling: PASS" @@ -450,33 +449,20 @@ ci-client: lint-client build-client test-client # --- Database --- -economy-db: ## Import economics data (commodities, chains, gate links) into systems.db - @echo " Generating minor brands (D-189 #829)..." - @reach generate brands - @python3 tooling/economy-db/import_economics.py +# Survives for muscle memory (D-263: "delegates to reach in one line and says +# so") — `make regen-db` is named in ~50 files, including the headers of the +# generated wiki TOMLs and the remedies the push gate prints. The behaviour is +# `reach ledger import`; this only tolerates its exit 2, which means imported +# AND stamped with the D-175 coverage gate unmet — a content gap, not a failure. +regen-db: ## Regenerate systems.db and stamp it — delegates to `reach ledger import` + @ec=0; reach ledger import || ec=$$?; [ $$ec -eq 0 ] || [ $$ec -eq 2 ] || exit $$ec -regen-db: ## Regenerate systems.db from all sources and stamp meta table (#855, #856) - @# Run as a single shell so `set -e` covers all steps. Without this - @# each recipe line was a fresh shell and a failure in step 1 did not - @# halt step 2, which could produce stale data with a fresh stamp - @# (PR #136 review T4). import_economics' exit code 2 is a valid - @# coverage-gate-warning state (DB and stamp committed), not an error, - @# so it's explicitly tolerated. Any other non-zero exit halts the - @# pipeline immediately. The atlas city/road/river geometry generator - @# was retired in #951 (D-223); import_economics now owns the atlas - @# index — it loads the names-only pool into atlas_city_names and empties - @# the geometry tables (the server cascade fills them, Phase 4). - @set -e; \ - echo " [regen-db] Importing economics data (runs generate_brands internally)..."; \ - ec=0; python3 tooling/economy-db/import_economics.py || ec=$$?; \ - if [ $$ec -ne 0 ] && [ $$ec -ne 2 ]; then exit $$ec; fi; \ - echo ""; \ - echo " regen-db complete — systems.db is up to date and stamped."; \ - echo " Stage it with: git add server/data/systems.db" - -# The five gate targets that lived here are RETIRED, not wrapped (D-263, +# The gate targets that lived here are RETIRED, not wrapped (D-263, # T-1281). They are tooling, and tooling has one door: # +# make economy-db -> reach ledger import (T-1289; it also ran +# `reach generate brands` first, which the +# import already does as its own first step) # make check-systems-db -> reach check systems-db-stamp # make check-client-version -> reach check client-version # make check-canvas-version -> reach check canvas-version diff --git a/docs/DEVOPS.md b/docs/DEVOPS.md index 06a498cf9..50482acde 100644 --- a/docs/DEVOPS.md +++ b/docs/DEVOPS.md @@ -239,14 +239,14 @@ make clean # Remove build artifacts and .cache/ contents ```bash reach validate content # Validate content YAML against JSON schemas -make check-fact-ids # Check fact_id references against knowledge catalogs +reach check fact-ids # Check fact_id references against knowledge catalogs ``` ### Gauntlet Checklists ```bash -make checklist-validate # Validate checklist YAML against schema (standalone) -make checklist-generate # Validate + print per-room condition summary +reach validate checklist --check # Validate checklist YAML against schema (standalone) +reach validate checklist # Validate + print per-room condition summary ``` Checklists live at `content/gauntlet/rooms/{room_id}/checklist.yaml` (per-room) and `content/gauntlet/cross_room_checks.yaml` (cross-room). Each condition is evaluable from an `ObserverSnapshot`. @@ -268,7 +268,7 @@ the next regeneration. ### Generator -`import_economics` (`tooling/economy-db/import_economics.py`) is the sole +`import_economics` (`reach ledger import`, `tooling/domains/ledger/`) is the sole generator. As its first step it runs the Rust `generate_brands` binary (via `core.process.cargo_binary`) to refresh `generated_brands.toml`, then imports economics data and the atlas index (names-only city pool; geometry tables stay @@ -288,7 +288,7 @@ After every successful non-dry-run, the generator writes a row to the `meta` tab registry is the `GENERATOR_SOURCES` dict in `tooling/generator_sources.py`. ```bash -make check-systems-db # Verify the stamp is fresh (exit 1 = stale) +reach check systems-db-stamp # Verify the stamp is fresh (exit 1 = stale) ``` ### Making a DB change @@ -299,7 +299,7 @@ make check-systems-db # Verify the stamp is fresh (exit 1 = stale) 4. Commit with `chore(db): regen systems.db — ` For schema changes, also update `server/data/systems-schema.sql` and add migration DDL -to `MIGRATION_SQL` in `import_economics.py`. +to `MIGRATION_SQL` in `tooling/domains/ledger/economy_import/migration.py`. See `.claude/rules/asset-pipeline.md` for the full rule set. diff --git a/docs/gemma-architecture-flavor-methodology.md b/docs/gemma-architecture-flavor-methodology.md index 25dc5b4e0..51ba3ca36 100644 --- a/docs/gemma-architecture-flavor-methodology.md +++ b/docs/gemma-architecture-flavor-methodology.md @@ -183,7 +183,7 @@ Gemma **never** sets a numeric or eligibility field. This is not a scoping choice made for simplicity — it's forced by the shape of the guardrails: the catalog-wide CI checks (`V-TT-01`: ≥5 eligible templates per `BulkClass` post-gate; `V-TT-02`: no template >60% of its eligible pool's weight, -`tooling/economy-db/economy_import/traits.py`) are properties of the **whole +`tooling/domains/ledger/economy_import/traits.py`) are properties of the **whole catalog**, not of any one template in isolation. A per-template extraction pass, looking at one system's prose, has no way to know whether adding this candidate at `base_weight = 9000` would push some `BulkClass`'s weighted pool @@ -244,7 +244,7 @@ automatically at bake time — see §7). 5. **CI guardrails as a post-authoring gate, not a pass/fail on Gemma's output.** Once Miri has assigned `base_weight`/`weight_mods`/gates, `make regen-db` re-runs `populate_trait_templates()` - (`tooling/economy-db/economy_import/traits.py`), which re-validates + (`tooling/domains/ledger/economy_import/traits.py`), which re-validates `V-TT-01` and `V-TT-02` across the **whole** catalog, not just the new entry. A proposal can clear 1-4 and still get rebalanced or rejected at bake time if it happens to push a `BulkClass` pool over the 60% line — @@ -269,7 +269,7 @@ Nothing is committed to `architecture_trait_catalog.toml` until it clears 1-4 4. Araminta adds the matching `visual_bundle`, including the D-235 fallback parent for every new specific tag (a new template ships with a working generic-parent fallback from day one, same as the shipped 28). -5. `make regen-db` bakes the change via `import_economics.py` → +5. `make regen-db` bakes the change via `reach ledger import` → `traits.py:populate_trait_templates()`; `V-TT-01`/`V-TT-02` re-run automatically (§6.5). `make check-systems-db` confirms the meta stamp. 6. Commit the TOML change and the regenerated `systems.db` together — the diff --git a/server/data/systems-schema.sql b/server/data/systems-schema.sql index c4654ea65..4ba45072a 100644 --- a/server/data/systems-schema.sql +++ b/server/data/systems-schema.sql @@ -298,7 +298,7 @@ CREATE TABLE IF NOT EXISTS commodities ( -- D-237 authored specialization layer: economic-specialization vocabulary. -- Each value maps to a commodity + projected (BulkClass, ProductionUbiquity) for D-233. -- Scale is encoded in the value (breadbasket vs estate_farming); no separate scale column. --- Compiled from wiki/economics/specialization_vocabulary.toml by import_economics.py. +-- Compiled from wiki/economics/specialization_vocabulary.toml by `reach ledger import`. CREATE TABLE IF NOT EXISTS specialization_vocabulary ( specialization_id TEXT PRIMARY KEY, commodity_id TEXT NOT NULL REFERENCES commodities(commodity_id), @@ -400,7 +400,7 @@ CREATE TABLE IF NOT EXISTS corp_lifecycle_events ( -- have to scan hundreds of JSON files. Polyline geometry stays in the files — -- the DB only stores scalar/filterable fields + `point_count` as a rough length -- proxy. The geometry tables are populated by the server-side generation --- cascade (Phase 4, D-223) and start empty; tooling/economy-db/import_economics.py +-- cascade (Phase 4, D-223) and start empty; tooling/domains/ledger/ (reach ledger import) -- extracts this entire block (between BEGIN/END ATLAS INDEX markers) from this -- file at runtime and empties the geometry tables on regen, so the DDL lives in -- exactly one place. @@ -505,7 +505,7 @@ CREATE INDEX IF NOT EXISTS idx_atlas_mountain_ranges_body ON atlas_mountain_rang -- Heightmap BLOB storage REMOVED (D-202 amended, #963): canonical elevation is -- now a per-body 16-bit grayscale heightmap.png file (sea_level in a tEXt -- chunk), not a systems.db BLOB. The atlas_body_heightmaps table is dropped via --- MIGRATION_SQL in import_economics.py. +-- MIGRATION_SQL in tooling/domains/ledger/economy_import/migration.py. -- City name reservations — replaces authored city positions in markers.json (D-207, #902) -- Position is generated by the city placement algorithm; name is authored or LLM-generated. @@ -675,7 +675,7 @@ CREATE INDEX IF NOT EXISTS idx_gate_links_to ON gate_links(to_system_id); -- One row per generator, updated on each successful non-dry-run. -- schema_version: monotonic semver string (e.g. "1.0.0") — bump on backwards-incompatible changes. -- Orderable, enabling savegame migration lineage (Phase 5+). --- Defined as SCHEMA_VERSION constant in tooling/schema_version.py. +-- Defined as SCHEMA_VERSION constant in tooling/domains/ledger/schema_version.py. -- schema_sha: SHA-1 hex of systems-schema.sql content at generation time (tamper detection). -- generator_sha: SHA-1 hex of the generator source file(s) content -- generated_at: ISO-8601 UTC timestamp of the run diff --git a/server/data/systems.db b/server/data/systems.db index bf9ae2357..d027f030a 100644 Binary files a/server/data/systems.db and b/server/data/systems.db differ diff --git a/server/src/atlas/trait_catalog_reader.rs b/server/src/atlas/trait_catalog_reader.rs index 9d3796eff..d3d69440f 100644 --- a/server/src/atlas/trait_catalog_reader.rs +++ b/server/src/atlas/trait_catalog_reader.rs @@ -8,7 +8,7 @@ //! stays DB-free downstream (T-987 keeps `GenerateSkeleton`/`FillChunk` pure). //! //! The catalog itself (`trait_templates`) is baked by -//! `tooling/economy-db/economy_import/traits.py` from +//! `tooling/domains/ledger/economy_import/traits.py` from //! `wiki/economics/architecture_trait_catalog.toml` (#993, #1005) — see //! `.claude/rules/asset-pipeline.md`. This reader only *consumes* the baked //! table; it never writes to `systems.db`. diff --git a/tooling/DOMAINS.md b/tooling/DOMAINS.md index 76bc23dee..a277563db 100644 --- a/tooling/DOMAINS.md +++ b/tooling/DOMAINS.md @@ -89,7 +89,7 @@ invented. | `atlas` | **the whole spatial ladder** (D-191). Flat verbs for authoring and inspection; nested groups per rung for generation | flat: `atlas` (Rust binary), `atlas-check`, `atlas-names`, `atlas-commit-and-sync`, `atlas-systems-done`, `atlas-update-field`, `atlas-verify`, `atlas-flatness` · `atlas map`: the 5 star-map files · `atlas planet`: ✅ ported (T-1288) from `planet-gen/` (30) — ten verbs, each restating its module's options for real `--help`; `test_planet_router.py` fails if a router option drifts from the module parser | | ~~`starmap`~~ | **folded into `atlas map`** — the top rung of the same ladder | — | | ~~`planet`~~ | **folded into `atlas planet`** — the third rung of the same ladder | — | -| `ledger` | the economics pipeline, named for the UI component that will aggregate it | `economy-db/` (17 files), `schema_version.py` | +| `ledger` | the economics pipeline, named for the UI component that will aggregate it | ✅ ported (T-1289). `economy-db/` → `domains/ledger/` (`economy_import/` kept by name; the entrypoint became `service.py`), `schema_version.py` with it. `reach ledger import`; `make regen-db` survives as a one-line delegate, `make economy-db` retired. `generated_brands.toml` byte-identical across the move | | `wiki` | wiki sync and content maintenance | `wiki/`, `db/wiki_sync.py`, `db/populate_gttr_hook.py`, `db/backfill_cultural_corridor.py`, `assign-astro-ids.py`, `fill-missing-globes.py`, `migrate-s-to-gj.py`, `patch-core-sector.py`, `process-wiki-system-changes` | | `assets` | connectors to the tower-of-joy generators | `db/audio_*.py`, `db/audio-*`, `db/image_connector.py`, `db/trellis_connector.py`, `db/common.py`, `trellis-batch.sh`, `synth_ui_sounds.py` | | `character` | bodies, garments, GLB handling | `garment-fit/make_logo.py`, `garment-qa/analyze_captures.py`, `convert_outfit.py`, `glb_strip_utility_nodes.py`, `inspect_glb.py`, `check_hair_symmetry.py`, `check_icosphere.py`, `render_quaternius_test.py`, `setup_clothing_metadata.py` — **note this is far smaller than `garment-fit/`'s file count suggests; 22 of its 23 files are Blender payloads and belong to the carve-out** | @@ -110,11 +110,13 @@ They move beside the domain that reads them (`check` and `db` respectively) as plain modules, not verbs. Making them commands would put something in `reach --help` that answers no question a person has. -**`schema_version.py` belongs to `db`, and `check` imports it.** It is consumed -by the economics importer *and* by the stamp gate. Shared, but not equally -owned: the importer defines the version, the gate reads it. It goes where it is -defined, and the cross-domain import is legitimate — that is what a service -layer is for. +**`schema_version.py` belongs to `ledger` (named `db` when this map was +drawn), and `check` reads it.** It is consumed by the economics importer *and* +by the stamp gate. Shared, but not equally owned: the importer defines the +version, the gate reads it. It goes where it is defined. *As ported (T-1289) +the gate never imports the module at all — it reads the version back out of +the `meta` table and checks it is semver — so the cross-domain import this +ruling allowed for turned out not to be needed.* **`test_*.py` files do NOT become a domain.** They are gate tests run by `make test-tooling`, not commands anyone types. `reach test …` would imply a @@ -138,14 +140,18 @@ data surface), `planet-gen/atlas_*.py` (`atlas_cohesion_audit`, `atlas_common`, `atlas_quality_analysis` — quality analysis of generated terrain), and `economy-db/atlas.py` (the atlas index tables in `systems.db`). These are three concerns sharing a noun, not one domain in three places. Each -stays with its owner — `atlas`, `planet` and `db` respectively — and the port -should resist the pull to collect them, which would produce a domain whose only -common thread is a word. +stays with its owner — `atlas`, `atlas planet` and `ledger` respectively — and +the port should resist the pull to collect them, which would produce a domain +whose only common thread is a word. *Held through both ports (T-1288, +T-1289): `domains/atlas/`, `domains/atlas/planet/atlas_*.py` and +`domains/ledger/economy_import/atlas.py` are still three separate files.* **`economy-db/errors.py` predates `core/errors.py` and is not the same thing.** Domain-local error types are fine; what must not happen is a silent merge, or a -second `ReachError` with different semantics. Reconcile explicitly when `db` is -ported. +second `ReachError` with different semantics. *Reconciled in T-1289: +`ImportAborted` stays, as internal control flow only — a step raises it to +request a rollback. It never reaches a caller: `ledger/service.py` rolls back +and converts it to a `ReachError` carrying the remedy.* **`character` rather than `garment`.** D-263's sketch says `garment`, but the files cover bodies, hair, GLB utilities and Quaternius imports as well as diff --git a/tooling/domains/ledger/__init__.py b/tooling/domains/ledger/__init__.py new file mode 100644 index 000000000..050b4b529 --- /dev/null +++ b/tooling/domains/ledger/__init__.py @@ -0,0 +1,12 @@ +"""`ledger` — the economics pipeline (D-263). + +Named for the implant app the player will use to browse markets, wealth and +transactions: the economic counterpart to what the Atlas is for topography. +Not `db`, which named a storage layer nobody looks at, and not `economics`, +which names a subject rather than the thing on screen. + +Formerly `tooling/economy-db/` plus `tooling/schema_version.py` (T-1289). The +importer here is the SOLE generator of `server/data/systems.db`; its stamped +source set lives in `tooling/generator_sources.py`, which is why moving these +files changed the stamp and the move landed together with a regen. +""" diff --git a/tooling/economy-db/economy_import/__init__.py b/tooling/domains/ledger/economy_import/__init__.py similarity index 62% rename from tooling/economy-db/economy_import/__init__.py rename to tooling/domains/ledger/economy_import/__init__.py index b1ac0f54a..26ba73794 100644 --- a/tooling/economy-db/economy_import/__init__.py +++ b/tooling/domains/ledger/economy_import/__init__.py @@ -1,9 +1,8 @@ """ -economy_import — module package behind tooling/economy-db/import_economics.py. +economy_import — the import steps behind `reach ledger import`. -Split out of the former single-file importer (T-1067). The entrypoint -``import_economics.py`` remains the CLI (Makefile regen-db, make test-tooling) -and orchestrates the single-transaction import; the steps live here: +Split out of the former single-file importer (T-1067). `ledger/service.py` +orchestrates the single-transaction import; the steps live here: paths.py source-file path constants (re-exports stamped paths from tooling/generator_sources.py) @@ -24,15 +23,9 @@ and orchestrates the single-transaction import; the steps live here: Every module here is part of the import_economics meta-stamp source set — tooling/generator_sources.py globs this directory, so adding a module automatically extends staleness detection. + +The sys.path bootstrap that used to sit here is gone (T-1289). It existed only +because `tooling/economy-db/` was hyphenated and so not importable; inside the +package, `tooling.generator_sources` and the ledger's `schema_version` resolve +by ordinary import. """ - -import sys -from pathlib import Path - -# Bootstrap: make tooling/ importable (generator_sources, schema_version) -# before any submodule needs them. tooling/economy-db is not a package -# (hyphenated dir), so the entrypoint puts it on sys.path and this package -# adds tooling/ itself. -_TOOLING_DIR = Path(__file__).resolve().parents[2] -if str(_TOOLING_DIR) not in sys.path: - sys.path.insert(0, str(_TOOLING_DIR)) diff --git a/tooling/economy-db/economy_import/atlas.py b/tooling/domains/ledger/economy_import/atlas.py similarity index 98% rename from tooling/economy-db/economy_import/atlas.py rename to tooling/domains/ledger/economy_import/atlas.py index d6c101972..4c5ae397e 100644 --- a/tooling/economy-db/economy_import/atlas.py +++ b/tooling/domains/ledger/economy_import/atlas.py @@ -19,6 +19,7 @@ import sqlite3 import tomllib from .paths import SCHEMA_SQL, SETTLEMENT_NAME_LOCKED_TOML, WIKI_STAR_SYSTEMS +from tooling.core import console # Atlas geometry index tables (D-191). These hold computed positions — city # centres, road/river/rail polylines, ocean/mountain extents. Under D-223 the @@ -139,7 +140,7 @@ def populate_atlas_city_names(conn: sqlite3.Connection, dry_run: bool) -> int: if skipped_bodies: unique = sorted(set(skipped_bodies)) - print(f" warning: {len(unique)} body dirs not in DB — skipped: {unique[:5]}") + console.event(f"{len(unique)} body dirs not in DB — skipped: {unique[:5]}", level="warn") if not dry_run: # D-242: corporations.headquarters_city_id FK-references @@ -236,7 +237,7 @@ def populate_atlas_feature_names(conn: sqlite3.Connection, dry_run: bool) -> int if skipped_bodies: unique = sorted(set(skipped_bodies)) - print(f" warning: {len(unique)} body dirs not in DB — skipped: {unique[:5]}") + console.event(f"{len(unique)} body dirs not in DB — skipped: {unique[:5]}", level="warn") if not dry_run: conn.execute("DELETE FROM atlas_feature_names") @@ -467,10 +468,8 @@ def populate_settlement_population_class(conn: sqlite3.Connection, dry_run: bool name_locked_applied += match_count if name_locked_unmatched: - print( - f" warning: {len(name_locked_unmatched)} NameLocked hero stanza(s) " - f"matched no atlas_city_names row: {name_locked_unmatched}" - ) + console.event(f"{len(name_locked_unmatched)} NameLocked hero stanza(s) " + f"matched no atlas_city_names row: {name_locked_unmatched}", level="warn") return { "bodies_spread": len(city_rows_by_body), diff --git a/tooling/economy-db/economy_import/bodies.py b/tooling/domains/ledger/economy_import/bodies.py similarity index 99% rename from tooling/economy-db/economy_import/bodies.py rename to tooling/domains/ledger/economy_import/bodies.py index 08ae37a11..6b22aac57 100644 --- a/tooling/economy-db/economy_import/bodies.py +++ b/tooling/domains/ledger/economy_import/bodies.py @@ -5,6 +5,7 @@ import sqlite3 from pathlib import Path from .paths import WIKI_STAR_SYSTEMS +from tooling.core import console def populate_body_radius_km(conn: sqlite3.Connection, dry_run: bool) -> int: @@ -290,7 +291,7 @@ def populate_biosphere_class(conn: sqlite3.Connection, dry_run: bool) -> int: if body_id and bc: bc = bc.strip().strip("'\"") if bc not in _VALID: - print(f" WARNING: {fpath}: biosphere_class {bc!r} not in {_VALID}; ignored") + console.event(f"{fpath}: biosphere_class {bc!r} not in {_VALID}; ignored", level="warn") continue override[body_id] = bc diff --git a/tooling/economy-db/economy_import/brands.py b/tooling/domains/ledger/economy_import/brands.py similarity index 89% rename from tooling/economy-db/economy_import/brands.py rename to tooling/domains/ledger/economy_import/brands.py index e8d835234..08b530d07 100644 --- a/tooling/economy-db/economy_import/brands.py +++ b/tooling/domains/ledger/economy_import/brands.py @@ -1,18 +1,11 @@ """Brand layer (D-189, #827): generate_brands shell-out, TOML import, validation.""" import sqlite3 -import sys import tomllib from pathlib import Path -from .errors import ImportAborted -from .paths import BRANDS_TOML, GENERATED_BRANDS_TOML, REPO_ROOT - -# economy-db/ is hyphenated, so it is not importable as a package and cannot -# reach `tooling.core` by normal import. The bootstrap goes away with T-1272 -# (the hyphen sweep); until then it is explicit rather than implied. -if str(REPO_ROOT) not in sys.path: - sys.path.insert(0, str(REPO_ROOT)) +from .paths import BRANDS_TOML, GENERATED_BRANDS_TOML +from tooling.core import console VALID_BRAND_CATEGORIES: set[str] = { "terroir", "heritage_craft", "tech_premium", "cultural", @@ -41,22 +34,21 @@ def regenerate_brands() -> None: wrapper was one of three identical copies of build-if-missing-then-exec, so it was retired into `core.process.cargo_binary` (T-1286) and this calls that helper instead. Same binary, same seed, same output. + + A build or run failure propagates as the ReachError `cargo_binary` raised. + It used to be printed and swapped for ImportAborted, which dropped its + remedy (T-1289). Nothing needs rolling back: this runs before the import + transaction opens. """ - from tooling.core.errors import ReachError from tooling.core.process import cargo_binary - print(" [pre/10] Running generate_brands (Rust) to refresh generated_brands.toml...") - try: - stdout = cargo_binary("generate_brands") - except ReachError as exc: - print(str(exc), file=sys.stderr) - raise ImportAborted() from exc + console.event("Running generate_brands (Rust) to refresh generated_brands.toml...", phase="pre") + stdout = cargo_binary("generate_brands") - # Print the Rust binary's own summary lines (brands generated, coverage). - # Indent so they fold under the pre-step heading. + # Relay the Rust binary's own summary lines (brands generated, coverage). for line in stdout.splitlines(): if line.strip(): - print(f" {line}") + console.event(line.strip(), phase="pre") def _load_brand_file(path: Path) -> tuple[list, list]: @@ -74,18 +66,18 @@ def import_brands( """Import brand_products and brand_inputs from brands.toml and generated_brands.toml. Hand-authored brands (brands.toml) are imported first; generated brands - (generated_brands.toml, produced by `tooling/generate-brands`) are merged in. + (generated_brands.toml, produced by `reach generate brands`) are merged in. Returns (n_products, n_inputs). """ if not BRANDS_TOML.exists(): - print(" warning: brands.toml not found — brand layer skipped") + console.event("brands.toml not found — brand layer skipped", level="warn") return 0, 0 products_authored, inputs_authored = _load_brand_file(BRANDS_TOML) products_generated, inputs_generated = _load_brand_file(GENERATED_BRANDS_TOML) if products_generated: - print(f" merging {len(products_generated)} generated brand_products from generated_brands.toml") + console.event(f"merging {len(products_generated)} generated brand_products from generated_brands.toml") products = products_authored + products_generated inputs = inputs_authored + inputs_generated diff --git a/tooling/economy-db/economy_import/corporations.py b/tooling/domains/ledger/economy_import/corporations.py similarity index 96% rename from tooling/economy-db/economy_import/corporations.py rename to tooling/domains/ledger/economy_import/corporations.py index 1bf88c389..0a476a7e1 100644 --- a/tooling/economy-db/economy_import/corporations.py +++ b/tooling/domains/ledger/economy_import/corporations.py @@ -9,6 +9,7 @@ from pathlib import Path from .atlas import most_populated_body_in_system from .errors import ImportAborted from .paths import CORP_HQ_PLACEMENT_TOML, CORPORATIONS_DIR +from tooling.core import console # --------------------------------------------------------------------------- # Corporation wiki parsing @@ -117,8 +118,8 @@ def sync_corporations( system_id = corp.get("system_id") hq_system = system_id if system_id and system_id in valid_systems else None if system_id and system_id not in valid_systems: - print(f" warning: {corp_id} HQ system '{system_id}' not in DB, " - f"headquarters_system set to NULL") + console.event(f"{corp_id} HQ system '{system_id}' not in DB, " + f"headquarters_system set to NULL", level="warn") to_insert.append(( corp_id, proper_name, @@ -274,11 +275,9 @@ def populate_corp_specialization( r[0] for r in conn.execute("SELECT DISTINCT body_id FROM atlas_city_names").fetchall() } if not city_bearing_bodies: - print( - " warning: atlas_city_names is empty — the city-presence " + console.event("atlas_city_names is empty — the city-presence " "tiebreak tier is inert this run (placements fall through to " - "body-type rank; they converge on the next run)" - ) + "body-type rank; they converge on the next run)", level="warn") errors: list[str] = [] unauthored: list[str] = [] @@ -333,16 +332,14 @@ def populate_corp_specialization( spec_rows.append((corp_id, spec, placement, authored_body)) if errors: - print(f" CORP SPECIALIZATION ERRORS ({len(errors)}):") + console.event(f"CORP SPECIALIZATION ERRORS ({len(errors)}):", level="error") for e in errors: - print(f" - {e}") + console.event(f"- {e}", level="error") raise ImportAborted() if unauthored: - print( - f" warning: {len(unauthored)} corp(s) have no authored " - f"corp_specialization (stay NULL; no HQ placement): {unauthored[:5]}" - ) + console.event(f"{len(unauthored)} corp(s) have no authored " + f"corp_specialization (stay NULL; no HQ placement): {unauthored[:5]}", level="warn") # Pre-compute the resolution so dry-run reports the same numbers a real # run writes (write is gated on dry_run; the derivation is not). @@ -440,7 +437,7 @@ def import_corp_presence( if skipped: for s in skipped: - print(f" warning: skipped corp_presence for {s}") + console.event(f"skipped corp_presence for {s}", level="warn") if not dry_run: conn.execute("DELETE FROM corp_presence") @@ -547,11 +544,11 @@ def populate_standalone_hq_settlements(conn: sqlite3.Connection, dry_run: bool) n_linked += 1 if unmatched_bodies: - print(f" warning: {len(unmatched_bodies)} corp(s) skipped (bad headquarters_body): " - f"{unmatched_bodies[:5]}") + console.event(f"{len(unmatched_bodies)} corp(s) skipped (bad headquarters_body): " + f"{unmatched_bodies[:5]}", level="warn") if unmatched_corps: - print(f" warning: {len(unmatched_corps)} CityTenant corp(s) unmatched: " - f"{unmatched_corps[:5]}") + console.event(f"{len(unmatched_corps)} CityTenant corp(s) unmatched: " + f"{unmatched_corps[:5]}", level="warn") return { "standalone_inserted": n_inserted, diff --git a/tooling/economy-db/economy_import/db.py b/tooling/domains/ledger/economy_import/db.py similarity index 100% rename from tooling/economy-db/economy_import/db.py rename to tooling/domains/ledger/economy_import/db.py diff --git a/tooling/economy-db/economy_import/economy.py b/tooling/domains/ledger/economy_import/economy.py similarity index 97% rename from tooling/economy-db/economy_import/economy.py rename to tooling/domains/ledger/economy_import/economy.py index 6560276f5..229ad3a66 100644 --- a/tooling/economy-db/economy_import/economy.py +++ b/tooling/domains/ledger/economy_import/economy.py @@ -6,6 +6,7 @@ import sqlite3 import tomllib from .paths import CHAINS_TOML, COMMODITIES_TOML, CURRENCY_ZONES_TOML, STAR_MAP +from tooling.core import console # --------------------------------------------------------------------------- # Gate links @@ -33,7 +34,7 @@ def import_gate_links(conn: sqlite3.Connection, dry_run: bool) -> int: if skipped: unique_skipped = sorted(set(skipped)) - print(f" warning: {len(unique_skipped)} system(s) in star-map.json not in DB: {unique_skipped[:5]}...") + console.event(f"{len(unique_skipped)} system(s) in star-map.json not in DB: {unique_skipped[:5]}...", level="warn") if not dry_run: conn.executemany( @@ -169,8 +170,8 @@ def set_currency_zones(conn: sqlite3.Connection, dry_run: bool) -> dict[str, int (sid,), ) else: - print(" warning: wiki/economics/currency_zones.toml not found — " - "all systems default to TRACTUS_PRIMARY / Sol to MIXED") + console.event("wiki/economics/currency_zones.toml not found — " + "all systems default to TRACTUS_PRIMARY / Sol to MIXED", level="warn") counts: dict[str, int | str] = {} for row in conn.execute("SELECT currency_zone, COUNT(*) FROM star_systems GROUP BY currency_zone"): diff --git a/tooling/economy-db/economy_import/errors.py b/tooling/domains/ledger/economy_import/errors.py similarity index 100% rename from tooling/economy-db/economy_import/errors.py rename to tooling/domains/ledger/economy_import/errors.py diff --git a/tooling/economy-db/economy_import/migration.py b/tooling/domains/ledger/economy_import/migration.py similarity index 100% rename from tooling/economy-db/economy_import/migration.py rename to tooling/domains/ledger/economy_import/migration.py diff --git a/tooling/economy-db/economy_import/paths.py b/tooling/domains/ledger/economy_import/paths.py similarity index 98% rename from tooling/economy-db/economy_import/paths.py rename to tooling/domains/ledger/economy_import/paths.py index 204202912..4931e46c0 100644 --- a/tooling/economy-db/economy_import/paths.py +++ b/tooling/domains/ledger/economy_import/paths.py @@ -8,7 +8,7 @@ importer can never disagree about where a stamped source lives. from pathlib import Path -from generator_sources import ( +from tooling.generator_sources import ( ARCHITECTURE_TRAIT_BIAS_TOML, ARCHITECTURE_TRAIT_CATALOG_TOML, ARCHITECTURE_ZONE_BIAS_TOML, diff --git a/tooling/economy-db/economy_import/specialization.py b/tooling/domains/ledger/economy_import/specialization.py similarity index 94% rename from tooling/economy-db/economy_import/specialization.py rename to tooling/domains/ledger/economy_import/specialization.py index cbf2bcaf3..bc2d06b32 100644 --- a/tooling/economy-db/economy_import/specialization.py +++ b/tooling/domains/ledger/economy_import/specialization.py @@ -6,6 +6,7 @@ from collections import Counter from .errors import ImportAborted from .paths import SPECIALIZATION_VOCAB_TOML, SYSTEM_SPECIALIZATION_TOML +from tooling.core import console # Authoritative D-233 projected enums. The full CI guardrail suite (V-SES-*) # lands in #1015; this module performs the FK + enum sanity the import itself @@ -194,9 +195,9 @@ def import_system_specialization(conn: sqlite3.Connection, dry_run: bool, ) if errors: - print(f" SPECIALIZATION ERRORS ({len(errors)}):") + console.event(f"SPECIALIZATION ERRORS ({len(errors)}):", level="error") for e in errors: - print(f" - {e}") + console.event(f"- {e}", level="error") raise ImportAborted() coverage: dict = { @@ -403,26 +404,24 @@ def _specialization_checks( if e and e in vocab: bulk_dist[vocab[e].get("bulk_class_projected")] += 1 pu_dist[vocab[e].get("production_ubiquity_projected")] += 1 - print(" Specialization coverage (D-237):") - print(f" economic: {n_econ} authored ({n_inhabited} inhabited; rest on fallback once #1014 lands)") - print(f" cultural: {n_cult} authored ({n_named} named; rest on corridor default)") - print(f" faction: {n_fac} authored ({n_named} named; rest on derivation)") - print(" BulkClass: " + " | ".join(f"{k}={v}" for k, v in sorted(bulk_dist.items()))) - print(" ProductionUbiquity: " + " | ".join(f"{k}={v}" for k, v in sorted(pu_dist.items()))) + console.event("Specialization coverage (D-237):") + console.event(f"economic: {n_econ} authored ({n_inhabited} inhabited; rest on fallback once #1014 lands)") + console.event(f"cultural: {n_cult} authored ({n_named} named; rest on corridor default)") + console.event(f"faction: {n_fac} authored ({n_named} named; rest on derivation)") + console.event("BulkClass: " + " | ".join(f"{k}={v}" for k, v in sorted(bulk_dist.items()))) + console.event("ProductionUbiquity: " + " | ".join(f"{k}={v}" for k, v in sorted(pu_dist.items()))) if warnings: - print(f" Specialization warnings ({len(warnings)}):") + console.event(f"Specialization warnings ({len(warnings)}):") for w in warnings: - print(f" - {w}") + console.event(f"- {w}", level="warn") if gate_failures: if strict: - print(f" SPECIALIZATION COMPLETENESS FAILURES ({len(gate_failures)}) [strict]:") + console.event(f"SPECIALIZATION COMPLETENESS FAILURES ({len(gate_failures)}) [strict]:", level="error") for g in gate_failures: - print(f" - {g}") + console.event(f"- {g}", level="error") raise ImportAborted() else: - print( - f" Specialization completeness: {len(gate_failures)} gate item(s) " + console.event(f"Specialization completeness: {len(gate_failures)} gate item(s) " f"pending (#1014 fallback / #1016 content pass) — warnings only until " - f"--strict-specialization" - ) + f"--strict-specialization") diff --git a/tooling/economy-db/economy_import/stamp.py b/tooling/domains/ledger/economy_import/stamp.py similarity index 89% rename from tooling/economy-db/economy_import/stamp.py rename to tooling/domains/ledger/economy_import/stamp.py index 5b2fc1b58..5f63507fd 100644 --- a/tooling/economy-db/economy_import/stamp.py +++ b/tooling/domains/ledger/economy_import/stamp.py @@ -1,15 +1,15 @@ """Meta-table generator stamp (#855, #856). Records the source-SHA of the generator that produced the DB so the pre-push -hook (tooling/check-systems-db-stamp) can detect stale snapshots. The source +hook (`reach check systems-db-stamp`) can detect stale snapshots. The source set comes from tooling/generator_sources.py — the single shared registry. """ import sqlite3 from pathlib import Path -from generator_sources import file_sha1 -from schema_version import SCHEMA_VERSION +from tooling.generator_sources import file_sha1 +from tooling.domains.ledger.schema_version import SCHEMA_VERSION from .paths import SCHEMA_SQL diff --git a/tooling/economy-db/economy_import/traits.py b/tooling/domains/ledger/economy_import/traits.py similarity index 97% rename from tooling/economy-db/economy_import/traits.py rename to tooling/domains/ledger/economy_import/traits.py index 36af4e143..cc37f72e6 100644 --- a/tooling/economy-db/economy_import/traits.py +++ b/tooling/domains/ledger/economy_import/traits.py @@ -12,6 +12,7 @@ from .paths import ( COLOR_REGISTER_BANDS_TOML, OBJECT_TAG_VOCABULARY_TOML, ) +from tooling.core import console _TRAIT_CORRIDOR_POOLS: set[str] = {"baseline", "heritage", "cross_corridor"} _TRAIT_BIAS_KINDS: set[str] = {"pin", "boost", "suppress"} @@ -205,9 +206,9 @@ def populate_trait_templates(conn: sqlite3.Connection, dry_run: bool) -> int: json.dumps(t["visual_bundle"]) if t.get("visual_bundle") else None, )) if errors: - print(f" TRAIT TEMPLATE ERRORS ({len(errors)}):") + console.event(f"TRAIT TEMPLATE ERRORS ({len(errors)}):", level="error") for e in errors: - print(f" - {e}") + console.event(f"- {e}", level="error") raise ImportAborted() # CI guardrails (D-232, Nigel): >=5 templates eligible per BulkClass (a # template with no bulk_class_gate is eligible for all); no single template @@ -232,9 +233,9 @@ def populate_trait_templates(conn: sqlite3.Connection, dry_run: bool) -> int: f"{w * 100 // total}% of the {bc} pool weight (>60%)" ) if gerrors: - print(f" TRAIT TEMPLATE GUARDRAIL FAILURES ({len(gerrors)}):") + console.event(f"TRAIT TEMPLATE GUARDRAIL FAILURES ({len(gerrors)}):", level="error") for e in gerrors: - print(f" - {e}") + console.event(f"- {e}", level="error") raise ImportAborted() # Validation/guardrails run on dry-run too; only mutate when committing. if not dry_run: @@ -340,9 +341,9 @@ def populate_architecture_zone_bias(conn: sqlite3.Connection, dry_run: bool) -> rows.append((template_tag, zone_type_id, bias_json)) if errors: - print(f" ARCHITECTURE ZONE BIAS ERRORS ({len(errors)}):") + console.event(f"ARCHITECTURE ZONE BIAS ERRORS ({len(errors)}):", level="error") for e in errors: - print(f" - {e}") + console.event(f"- {e}", level="error") raise ImportAborted() if not dry_run: @@ -432,9 +433,9 @@ def populate_color_register_bands(conn: sqlite3.Connection, dry_run: bool) -> in errors.append(f"V-TT-07: color_register '{reg}' referenced by trait_templates but has no band") if errors: - print(f" COLOR REGISTER BAND ERRORS ({len(errors)}):") + console.event(f"COLOR REGISTER BAND ERRORS ({len(errors)}):", level="error") for e in errors: - print(f" - {e}") + console.event(f"- {e}", level="error") raise ImportAborted() if not dry_run: @@ -507,9 +508,9 @@ def populate_atlas_body_trait_bias(conn: sqlite3.Connection, dry_run: bool) -> i f"K of {_MAX_PINS_PER_BODY} (ComplexityTier::Full); pins count toward K (D-232)" ) if errors: - print(f" TRAIT BIAS ERRORS ({len(errors)}):") + console.event(f"TRAIT BIAS ERRORS ({len(errors)}):", level="error") for e in errors: - print(f" - {e}") + console.event(f"- {e}", level="error") raise ImportAborted() if not dry_run: conn.execute("DELETE FROM atlas_body_trait_bias") diff --git a/tooling/economy-db/economy_import/validators.py b/tooling/domains/ledger/economy_import/validators.py similarity index 100% rename from tooling/economy-db/economy_import/validators.py rename to tooling/domains/ledger/economy_import/validators.py diff --git a/tooling/domains/ledger/router.py b/tooling/domains/ledger/router.py new file mode 100644 index 000000000..540e7f542 --- /dev/null +++ b/tooling/domains/ledger/router.py @@ -0,0 +1,47 @@ +"""Transport for the `ledger` domain — args in, delegate, format out.""" + +from __future__ import annotations + +from pathlib import Path + +import typer + +from tooling.core import cli +from tooling.core.command import command + +app = cli.domain("ledger", "The economics pipeline — the import that builds systems.db.") + + +@app.callback() +def _domain() -> None: + """Keeps `ledger` a group (Typer collapses a single-command app).""" + + +@app.command("import") +@command +def import_( + db: Path = typer.Option(None, "--db", help="systems.db to import into (default: server/data/systems.db)."), + dry_run: bool = typer.Option(False, "--dry-run", help="Validate every step, write nothing."), + strict_specialization: bool = typer.Option( + False, + "--strict-specialization", + help="Treat the D-237 completeness gates (V-SES-02, V-FAC-01) as hard errors.", + ), +) -> None: + """Rebuild systems.db from the economics sources, and stamp it. + + The sole generator of server/data/systems.db — `make regen-db` runs this. + It regenerates generated_brands.toml first (skipped on --dry-run), imports + in one transaction, and stamps the meta table so the push gate can tell a + stale snapshot from a fresh one. + + Exit 2 means imported AND stamped but the D-175 coverage gate is not met: + a content gap, not a failure, and tolerated by regen-db and the tooling + gate. Exit 1 means nothing was written. + """ + # Imported here, not at module level: the service pulls in the whole + # importer package and the stamp registry, and `reach ledger --help` + # should not pay for that (T-1260). + from tooling.domains.ledger import service + + service.run(db or service.DB_PATH, dry_run=dry_run, strict_specialization=strict_specialization) diff --git a/tooling/schema_version.py b/tooling/domains/ledger/schema_version.py similarity index 62% rename from tooling/schema_version.py rename to tooling/domains/ledger/schema_version.py index c090cf434..07aa624db 100644 --- a/tooling/schema_version.py +++ b/tooling/domains/ledger/schema_version.py @@ -6,8 +6,9 @@ Bump SCHEMA_VERSION manually on any backwards-incompatible schema change Additive changes (new nullable columns, new tables, new indexes) do not require a bump. -Imported by: - tooling/economy-db/import_economics.py +Lives in `ledger` because the importer is where the version is DEFINED; +`reach check systems-db-stamp` only reads it back out of the meta table +(tooling/DOMAINS.md). Imported by economy_import/stamp.py. """ SCHEMA_VERSION = "1.0.0" diff --git a/tooling/domains/ledger/service.py b/tooling/domains/ledger/service.py new file mode 100755 index 000000000..14fd4c1c5 --- /dev/null +++ b/tooling/domains/ledger/service.py @@ -0,0 +1,369 @@ +"""Logic for the `ledger` domain — the economics import into systems.db. Transport-agnostic (D-263). + +Reads TOML/JSON source files and populates the economics tables: + - gate_links from docs/design/star-map.json (335 edges, bidirectional) + - commodities from wiki/economics/commodities.toml (36 types) + - production_chains + chain_inputs from wiki/economics/production_chains.toml + - currency_zone on star_systems (default TRACTUS_PRIMARY) + - gate_energy_connected on star_systems (D-186: false for MARK_PRIMARY zones) + - corporations from wiki/corporations/*.md (sync + insert new records) + - corp_specialization/hq_placement/headquarters_body/headquarters_city_id + on corporations (D-242: HQ-placement key + baked + CityTenant link, from corp_hq_placement.toml) + - corp_presence from wiki/corporations/*.md (headquarters location data) + - atlas_city_names Standalone-HQ settlement rows (D-242, T-1074) + +Validation (hard errors, non-zero exit on any failure): + - Wiki corporation names must match DB proper_name records (D-182 sync constraint) + - Chain completeness: every intermediate commodity has at least one production chain + - Commodity coverage: 3+ corporations per major commodity type (D-175) + - System coverage: 1+ corporation per inhabited system with population > 100K (D-175) + +This is the sole generator of `server/data/systems.db` and was +`tooling/economy-db/import_economics.py` until T-1289. The steps live in the +`economy_import` package; the stamped source set is defined in +`tooling/generator_sources.py`. + +**Exit codes are a contract, kept exactly.** 0 = imported and every coverage +threshold met. 2 = imported AND stamped, but the D-175 coverage gate is not met +— `make regen-db` and the tooling gate both tolerate it, because the data is +usable and the gap is content, not code. 1 = nothing written. + +**Two error types, deliberately.** `economy_import.errors.ImportAborted` is +internal control flow — a step raises it to request rollback after printing +what went wrong. It never escapes this module: `run()` rolls back and converts +it to a `ReachError` carrying the remedy. That is the explicit reconciliation +DOMAINS.md asked for, rather than a second error type with different semantics +reaching the caller. +""" + +from __future__ import annotations + +import sqlite3 +from pathlib import Path + +from tooling.core import console +from tooling.core.errors import ReachError +from tooling.domains.ledger.economy_import import ( + atlas, + bodies, + brands, + corporations, + db, + economy, + migration, + specialization, + stamp, + traits, + validators, +) +from tooling.domains.ledger.economy_import.errors import ImportAborted +from tooling.domains.ledger.economy_import.paths import DB_PATH +from tooling.generator_sources import IMPORT_ECONOMICS_SOURCES + +__all__ = ["DB_PATH", "run"] + +# Named once, so the progress fraction and the count cannot disagree. +STEPS = 24 + +ABORTED_FIX = ( + "fix the errors listed above in their source (wiki/economics/*.toml, " + "wiki/corporations/*.md), then re-run: reach ledger import --dry-run" +) + + +class _Steps: + """Numbered progress through the import, one event per step.""" + + def __init__(self) -> None: + self.n = 0 + + def __call__(self, message: str) -> None: + self.n += 1 + console.event(message, phase=f"{self.n}/{STEPS}", progress=self.n / STEPS) + + +def run(db_path: Path, *, dry_run: bool = False, strict_specialization: bool = False) -> None: + """Import every economics source into `db_path`, in one transaction. + + Raises ReachError with exit code 1 when nothing was written, and with exit + code 2 when the import committed but the coverage gate is not met. + """ + if not db_path.exists(): + raise ReachError( + f"{db_path} not found", + fix="pass --db with an existing systems.db, or restore it: git restore server/data/systems.db", + ) + + console.event(f"Economics import into {db_path}" + (" — DRY RUN" if dry_run else "")) + + # Load wiki corps before opening the DB — allows early exit on parse failures. + console.event("Loading wiki corporations...") + wiki_corps = corporations.load_wiki_corps() + console.event(f"{len(wiki_corps)} corporation files parsed") + + # Regenerate generated_brands.toml via the Rust binary before the import + # reads it — single pipeline, single stamp (review T2/H3). Skipped on a + # dry run to avoid a disk side effect during validation. + if not dry_run: + brands.regenerate_brands() + + conn = db.connect(db_path) + try: + counts = _import(conn, wiki_corps, dry_run, strict_specialization) + except ImportAborted: + conn.rollback() + conn.close() + raise ReachError( + "economics import aborted — rolled back, nothing written", fix=ABORTED_FIX + ) from None + except BaseException: + # Anything else (KeyboardInterrupt, MemoryError, a DB error, a bug) + # rolls back too, so the DB is never left half-imported. Re-raised so + # the traceback survives. + conn.rollback() + conn.close() + raise + + # Stamp generator metadata (#855, #856) so the pre-push hook can detect a + # stale snapshot. Written BEFORE the coverage gate: the stamp records which + # code produced the DB, not whether the data is complete (#860). + if not dry_run: + try: + stamp.write_stamp(conn, "import_economics", *IMPORT_ECONOMICS_SOURCES) + conn.commit() + console.event("Stamped: import_economics (covers brand pipeline Rust sources)") + except Exception as exc: # noqa: BLE001 + console.event(f"failed to write generator stamp: {exc}", level="warn") + + coverage_errors = _coverage(conn, wiki_corps) + conn.close() + + summary = ( + f"{counts['links']} gate_links, {counts['commodities']} commodities, " + f"{counts['chains']} chains, {counts['inputs']} inputs, " + f"{counts['presence']} corp_presence, {counts['brands']} brand_products, " + f"{counts['brand_inputs']} brand_inputs, {counts['fiscal']} system_fiscal" + ) + if coverage_errors: + for e in coverage_errors: + console.event(e, level="error") + raise ReachError( + f"imported ({summary}) but the D-175 Phase 2 coverage gate is not met — " + f"{len(coverage_errors)} gap(s) above. " + + ("Nothing was written (dry run)." if dry_run else "Data and stamp are committed."), + fix="add corporations to wiki/corporations/ until the thresholds are met, then re-run", + exit_code=2, + ) + + written = "validated, nothing written (dry run)" if dry_run else "imported and stamped" + console.verdict(f"ledger import: {written} — {summary}") + + +def _import( + conn: sqlite3.Connection, wiki_corps, dry_run: bool, strict_specialization: bool +) -> dict[str, int]: + """The clear-then-reimport cycle, as one explicit transaction. + + Any crash, validation error or interrupt between the first DELETE and the + commit rolls everything back (the caller owns that): the DB never ends up + half-cleared. On success it commits exactly once, right after structural + validation passes. On a dry run the transaction is left open so the + coverage check can still SELECT the imported rows; closing discards it. + """ + step = _Steps() + conn.execute("BEGIN") + + # Schema migration — idempotent, inside the tx so a crash leaves no + # half-applied ALTER TABLE. + step("Schema migration...") + migration.apply_schema_migrations(conn) + # Atlas index tables: canonical DDL + empty geometry (D-223, #951). + atlas.ensure_atlas_index_schema(conn, dry_run) + + # Clear economics tables in FK-safe order (children before parents). + if not dry_run: + db.clear_economics_tables(conn) + + step("Importing gate links...") + n_links = economy.import_gate_links(conn, dry_run) + console.event(f"{n_links} rows (bidirectional)") + + step("Importing commodities...") + n_commodities = economy.import_commodities(conn, dry_run) + console.event(f"{n_commodities} commodities") + + step("Importing production chains...") + n_chains, n_inputs = economy.import_chains(conn, dry_run) + console.event(f"{n_chains} chains, {n_inputs} inputs") + + # D-237 authored layer — after commodities (FK), before currency zones. + # UPSERTs onto pre-existing system_economy / system_factions rows; + # unauthored systems stay NULL. + step("Importing system specialization (D-237)...") + spec = specialization.import_system_specialization(conn, dry_run, strict=strict_specialization) + console.event( + f"vocab {spec['vocab']} | economic {spec['economic']} | " + f"cultural {spec['cultural']} | faction {spec['faction']}" + ) + if spec["missing_economy_row"]: + console.event( + f"{len(spec['missing_economy_row'])} authored system(s) lack a " + f"system_economy row (values dropped): {spec['missing_economy_row']}", + level="warn", + ) + if spec["missing_faction_row"]: + console.event( + f"{len(spec['missing_faction_row'])} authored system(s) lack a " + f"system_factions row (faction dropped): {spec['missing_faction_row']}", + level="warn", + ) + + step("Setting currency zones...") + for zone, count in sorted(economy.set_currency_zones(conn, dry_run).items()): + console.event(f"{zone}: {count}") + + # D-186 — must run after currency zones. + step("Setting gate energy connectivity...") + for label, count in sorted(economy.set_gate_energy(conn, dry_run).items()): + console.event(f"{label}: {count}") + + # D-182: a name divergence is a hard error. + step("Syncing corporations...") + corp_errors = corporations.sync_corporations(conn, wiki_corps, dry_run) + if corp_errors: + console.event(f"name divergence detected (D-182) — {len(corp_errors)} error(s):", level="error") + for e in corp_errors: + console.event(e, level="error") + console.event("update the wiki title or the DB proper_name to match, then re-run", level="error") + raise ImportAborted() + n_db_corps = conn.execute("SELECT COUNT(*) FROM corporations").fetchone()[0] + console.event(f"{n_db_corps} corporations in DB ({len(wiki_corps)} from wiki)") + + # D-242 Phase A — reset + derive every run (PR #177 H1/T2). After corp + # sync (rows must exist), before corp_presence (reads headquarters_body + # back), and before the city-presence tiebreak further down, which reads + # atlas_city_names BEFORE this run's rebuild — the previous run's settled + # state, by design. + step("Importing corp specialization + HQ placement (D-242)...") + corp_spec = corporations.populate_corp_specialization(conn, wiki_corps, dry_run) + console.event( + f"{corp_spec['specialized']}/{corp_spec['total']} corps specialized, " + f"{corp_spec['hq_resolved']} headquarters_body resolved (recomputed every run; " + f"{corp_spec['hq_overridden']} authored overrides)" + ) + + step("Importing corp presence...") + commodity_ids = {r[0] for r in conn.execute("SELECT commodity_id FROM commodities").fetchall()} + n_presence = corporations.import_corp_presence(conn, wiki_corps, commodity_ids, dry_run) + console.event(f"{n_presence} corp_presence rows") + + # D-189, #827. + step("Importing brand products and inputs...") + n_brands, n_brand_inputs = brands.import_brands(conn, dry_run) + console.event(f"{n_brands} brand_products, {n_brand_inputs} brand_inputs") + + # D-189 §6. + step("Populating system_fiscal...") + n_fiscal = economy.import_system_fiscal(conn, dry_run) + console.event(f"{n_fiscal} system_fiscal rows") + + # D-204, #910. + step("Populating body_radius_km fallback...") + console.event(f"{bodies.populate_body_radius_km(conn, dry_run)} bodies updated") + + # D-207, #908. + step("Populating atlas_city_names from wiki content...") + console.event(f"{atlas.populate_atlas_city_names(conn, dry_run)} city name rows") + + # D-223, T-1169 — mirrors the city pool's shape over a different + # names-only markers.json key set; independent of the settlement pool. + step("Populating atlas_feature_names from wiki content...") + console.event(f"{atlas.populate_atlas_feature_names(conn, dry_run)} feature name rows") + + # D-242 Phase B (T-1074). SUPERSEDES the retired corp-HQ cross-reference + # (D-207, #909), which inserted one city row per corp HQ with no + # UNIQUE(body_id, name) — ten co-named Groombridge rows on GJ380c. Must + # follow the city pool and the corp-specialization step. + step("Emitting Standalone-HQ settlements + CityTenant links (D-242)...") + hq = corporations.populate_standalone_hq_settlements(conn, dry_run) + console.event( + f"{hq['standalone_inserted']} Standalone-HQ settlements, " + f"{hq['tenant_linked']} CityTenant links ({hq['tenant_unmatched']} unmatched)" + ) + + # D-242, T-1075 — over the CORRECTED pool, so Standalone-HQ settlements + # are part of the rank-size spread rather than bolted on after. + step("Baking settlement population + settlement_class (D-242)...") + pop = atlas.populate_settlement_population_class(conn, dry_run) + console.event( + f"{pop['cities_populated']} cities populated across {pop['bodies_spread']} bodies, " + f"{pop['name_locked_applied']} NameLocked pins applied" + ) + + # D-232, #993 — catalog first, then sparse per-body hero bias (FK). + step("Baking trait_templates catalog (D-232)...") + console.event(f"{traits.populate_trait_templates(conn, dry_run)} trait templates") + step("Baking atlas_body_trait_bias hero pins (D-232)...") + console.event(f"{traits.populate_atlas_body_trait_bias(conn, dry_run)} body trait-bias rows") + + # T-1024, D-239 §2. + step("Populating axial_tilt_deg from body-def frontmatter...") + console.event(f"{bodies.populate_axial_tilt_deg(conn, dry_run)} bodies updated with axial_tilt_deg") + + # D-247, T-1085 — frontmatter override + two-gate default. + step("Populating biosphere_class (D-247)...") + console.event(f"{bodies.populate_biosphere_class(conn, dry_run)} bodies updated with biosphere_class") + + # T-988, D-235 — both follow trait_templates (V-TT-06 / V-TT-07 validate + # against it). + step("Baking architecture_zone_bias table (D-235)...") + console.event(f"{traits.populate_architecture_zone_bias(conn, dry_run)} zone-bias rows") + step("Baking color_register_bands table (D-235)...") + console.event(f"{traits.populate_color_register_bands(conn, dry_run)} color register bands") + + # Structural integrity (FK, chain refs, chain completeness). These mean + # the imported data is broken — do NOT commit. + step("Validating structural integrity...") + struct_errors = validators.validate(conn) + brands.validate_brands(conn) + if struct_errors: + console.event(f"STRUCTURAL ERRORS ({len(struct_errors)}) — rolling back:", level="error") + for e in struct_errors: + console.event(e, level="error") + raise ImportAborted() + console.event("FK integrity, chain completeness, and brand layer (V-B01–V-B06) OK") + + # Commit before the coverage check: coverage is a Phase 2 gate (D-175), + # and the data should be queryable so the gaps can be reported clearly. + if not dry_run: + conn.commit() + console.event("Data committed.") + else: + console.event("Dry run — no changes written.") + + if step.n != STEPS: + # A step added without bumping STEPS would make every progress + # fraction wrong without failing anything. Cheap to refuse here. + raise AssertionError(f"STEPS is {STEPS} but the import ran {step.n} steps") + + return { + "links": n_links, + "commodities": n_commodities, + "chains": n_chains, + "inputs": n_inputs, + "presence": n_presence, + "brands": n_brands, + "brand_inputs": n_brand_inputs, + "fiscal": n_fiscal, + } + + +def _coverage(conn: sqlite3.Connection, wiki_corps) -> list[str]: + """The D-175 Phase 2 gate. Runs after the commit, so the data is usable.""" + console.event("Validating coverage (D-175 Phase 2 gate)...") + commodity_ids = {r[0] for r in conn.execute("SELECT commodity_id FROM commodities").fetchall()} + errors = validators.validate_commodity_coverage(conn, wiki_corps, commodity_ids) + errors += validators.validate_system_coverage(conn, wiki_corps) + if not errors: + console.event("All coverage thresholds met — Phase 2 gate PASSED.") + return errors diff --git a/tooling/economy-db/import_economics.py b/tooling/economy-db/import_economics.py deleted file mode 100755 index a6523ba6a..000000000 --- a/tooling/economy-db/import_economics.py +++ /dev/null @@ -1,377 +0,0 @@ -#!/usr/bin/env python3 -""" -Import economics data into systems.db. - -Reads TOML/JSON source files and populates the economics tables: - - gate_links from docs/design/star-map.json (335 edges, bidirectional) - - commodities from wiki/economics/commodities.toml (36 types) - - production_chains + chain_inputs from wiki/economics/production_chains.toml - - currency_zone on star_systems (default TRACTUS_PRIMARY) - - gate_energy_connected on star_systems (D-186: false for MARK_PRIMARY zones) - - corporations from wiki/corporations/*.md (sync + insert new records) - - corp_specialization/hq_placement/headquarters_body/headquarters_city_id - on corporations (D-242: HQ-placement key + baked - CityTenant link, from corp_hq_placement.toml) - - corp_presence from wiki/corporations/*.md (headquarters location data) - - atlas_city_names Standalone-HQ settlement rows (D-242, T-1074) - -Validation (hard errors, non-zero exit on any failure): - - Wiki corporation names must match DB proper_name records (D-182 sync constraint) - - Chain completeness: every intermediate commodity has at least one production chain - - Commodity coverage: 3+ corporations per major commodity type (D-175) - - System coverage: 1+ corporation per inhabited system with population > 100K (D-175) - -This file is the CLI entrypoint and single-transaction orchestrator; the import -steps live in the economy_import package (T-1067). The stamped source set is -defined in tooling/generator_sources.py. - -Usage: - python3 tooling/economy-db/import_economics.py - python3 tooling/economy-db/import_economics.py --dry-run - python3 tooling/economy-db/import_economics.py --db path/to/systems.db -""" - -import argparse -import sys -from pathlib import Path - -# tooling/economy-db is not a package (hyphenated dir) — put it on sys.path so -# the economy_import package resolves; its __init__ adds tooling/ for -# generator_sources and schema_version. -sys.path.insert(0, str(Path(__file__).resolve().parent)) - -from economy_import import ( # noqa: E402 - atlas, - bodies, - brands, - corporations, - db, - economy, - migration, - specialization, - stamp, - traits, - validators, -) -from economy_import.errors import ImportAborted # noqa: E402 -from economy_import.paths import DB_PATH # noqa: E402 -from generator_sources import IMPORT_ECONOMICS_SOURCES # noqa: E402 - - -def main() -> None: - parser = argparse.ArgumentParser(description="Import economics data into systems.db") - parser.add_argument("--db", default=str(DB_PATH), help="Path to systems.db") - parser.add_argument("--dry-run", action="store_true", help="Validate without writing") - parser.add_argument( - "--strict-specialization", action="store_true", - help="Treat D-237 completeness gates (V-SES-02, V-FAC-01) as hard errors. " - "Off by default until the #1014 fallback and #1016 content pass land.", - ) - args = parser.parse_args() - - db_path = Path(args.db) - if not db_path.exists(): - print(f"error: {db_path} not found", file=sys.stderr) - sys.exit(1) - - print("\n Economics Import Pipeline") - print(f" DB: {db_path}") - if args.dry_run: - print(" Mode: DRY RUN") - print() - - # Load wiki corps before opening DB — allows early exit on parse failures - print(" Loading wiki corporations...") - wiki_corps = corporations.load_wiki_corps() - print(f" {len(wiki_corps)} corporation files parsed") - - # Regenerate generated_brands.toml via the Rust binary before the Python - # import reads it. Single pipeline, single stamp — resolves review T2/H3 - # ("on-behalf stamping" coupling) by folding brand generation into - # import_economics' flow rather than having the caller (Makefile / user) - # remember to run it first. Skipped on --dry-run to avoid a disk - # side-effect during validation. - if not args.dry_run: - try: - brands.regenerate_brands() - except ImportAborted: - sys.exit(1) - - conn = db.connect(db_path) - - # The clear-then-reimport cycle below runs as a single explicit - # transaction. Any crash, validation error, or KeyboardInterrupt - # between the first DELETE and the final commit rolls everything - # back — the DB never ends up half-cleared with stale rows in some - # tables and empty rows in others. On success we commit exactly - # once, immediately after structural validation passes. - conn.execute("BEGIN") - try: - # 1. Migrate schema (idempotent, inside the tx so a crash here - # leaves no half-applied ALTER TABLE.) - print(" [1/10] Schema migration...") - migration.apply_schema_migrations(conn) - # Atlas index tables: apply canonical DDL + empty geometry (D-223, #951) - atlas.ensure_atlas_index_schema(conn, args.dry_run) - print(" tables and columns ready") - - # Clear economics tables in FK-safe order (children before parents) - if not args.dry_run: - db.clear_economics_tables(conn) - - # 2. Gate links - print(" [2/10] Importing gate links...") - n_links = economy.import_gate_links(conn, args.dry_run) - print(f" {n_links} rows (bidirectional)") - - # 3. Commodities - print(" [3/10] Importing commodities...") - n_commodities = economy.import_commodities(conn, args.dry_run) - print(f" {n_commodities} commodities") - - # 4. Production chains - print(" [4/10] Importing production chains...") - n_chains, n_inputs = economy.import_chains(conn, args.dry_run) - print(f" {n_chains} chains, {n_inputs} inputs") - - # 4b. System specialization (D-237 authored layer) — after commodities - # (FK) and before currency zones. UPSERTs onto pre-existing - # system_economy / system_factions rows; unauthored systems stay NULL. - print(" [4b/10] Importing system specialization (D-237)...") - spec = specialization.import_system_specialization( - conn, args.dry_run, strict=args.strict_specialization - ) - print( - f" vocab {spec['vocab']} | economic {spec['economic']} | " - f"cultural {spec['cultural']} | faction {spec['faction']}" - ) - if spec["missing_economy_row"]: - print( - f" WARNING: {len(spec['missing_economy_row'])} authored " - f"system(s) lack a system_economy row (values dropped): " - f"{spec['missing_economy_row']}" - ) - if spec["missing_faction_row"]: - print( - f" WARNING: {len(spec['missing_faction_row'])} authored " - f"system(s) lack a system_factions row (faction dropped): " - f"{spec['missing_faction_row']}" - ) - - # 5. Currency zones - print(" [5/10] Setting currency zones...") - zones = economy.set_currency_zones(conn, args.dry_run) - for zone, count in sorted(zones.items()): - print(f" {zone}: {count}") - - # 6. Gate energy connectivity (D-186) — must run after currency zones - print(" [6/10] Setting gate energy connectivity...") - energy = economy.set_gate_energy(conn, args.dry_run) - for label, count in sorted(energy.items()): - print(f" {label}: {count}") - - # 7. Sync corporations from wiki (D-182: hard error on name divergence) - print(" [7/10] Syncing corporations...") - corp_errors = corporations.sync_corporations(conn, wiki_corps, args.dry_run) - if corp_errors: - print(f" ERRORS ({len(corp_errors)}) — name divergence detected (D-182):") - for e in corp_errors: - print(f" - {e}") - print(" Fix: update wiki title or DB proper_name to match, then re-run.") - raise ImportAborted() - n_db_corps = conn.execute("SELECT COUNT(*) FROM corporations").fetchone()[0] - print(f" {n_db_corps} corporations in DB ({len(wiki_corps)} from wiki)") - - # 7b. Corp specialization + HQ placement (D-242, T-1074) — Phase A. - # Reset + derive on every run (PR #177 H1/T2): the importer-owned - # corporations columns are NULLed and re-derived from source, never - # kept from a prior run. Must run after corp sync (rows must exist), - # before corp_presence (which reads headquarters_body back off - # corporations), and before step 12 (the H2 city-presence tiebreak - # reads atlas_city_names BEFORE this run's clear/rebuild — the - # previous run's settled state, by design). - print(" [7b/10] Importing corp specialization + HQ placement (D-242)...") - corp_spec = corporations.populate_corp_specialization(conn, wiki_corps, args.dry_run) - print( - f" {corp_spec['specialized']}/{corp_spec['total']} corps specialized, " - f"{corp_spec['hq_resolved']} headquarters_body resolved (recomputed every run; " - f"{corp_spec['hq_overridden']} authored overrides)" - ) - - # 8. Corp presence from wiki headquarters data - print(" [8/10] Importing corp presence...") - commodity_ids = { - r[0] for r in conn.execute("SELECT commodity_id FROM commodities").fetchall() - } - n_presence = corporations.import_corp_presence( - conn, wiki_corps, commodity_ids, args.dry_run - ) - print(f" {n_presence} corp_presence rows") - - # 9. Brand products and inputs (D-189, #827) - print(" [9/10] Importing brand products and inputs...") - n_brands, n_brand_inputs = brands.import_brands(conn, args.dry_run) - print(f" {n_brands} brand_products, {n_brand_inputs} brand_inputs") - - # 10. System fiscal parameters (D-189 section 6) - print(" [10/13] Populating system_fiscal...") - n_fiscal = economy.import_system_fiscal(conn, args.dry_run) - print(f" {n_fiscal} system_fiscal rows") - - # 11. body_radius_km fallback from planet_class (D-204, #910) - print(" [11/13] Populating body_radius_km fallback...") - n_radius = bodies.populate_body_radius_km(conn, args.dry_run) - print(f" {n_radius} bodies updated") - - # 12. atlas_city_names from wiki markers.json (D-207, #908) - print(" [12/13] Populating atlas_city_names from wiki content...") - n_cities = atlas.populate_atlas_city_names(conn, args.dry_run) - print(f" {n_cities} city name rows") - - # 12b. atlas_feature_names (rivers/mountains) from wiki markers.json - # (D-223, T-1169) — mirrors step 12's pool-load shape exactly, over a - # different names-only markers.json key set. Independent of the - # settlement pool, so order relative to step 13 doesn't matter; placed - # here to stay adjacent to its sibling pool-load step. - print(" [12b/13] Populating atlas_feature_names from wiki content...") - n_features = atlas.populate_atlas_feature_names(conn, args.dry_run) - print(f" {n_features} feature name rows") - - # 13. Standalone-HQ settlements + CityTenant city links (D-242, T-1074) — Phase B. - # SUPERSEDES the retired corp-HQ cross-reference (D-207, #909) — that - # step inserted one atlas_city_names row per corp HQ with no - # UNIQUE(body_id, name), producing duplicate co-named "cities" (10 - # Groombridge rows on GJ380c). Must run after populate_atlas_city_names - # (the city pool Standalone HQs join and CityTenant HQs tenant) and - # after step 7b (corp_specialization/hq_placement/headquarters_body). - print(" [13/13] Emitting Standalone-HQ settlements + CityTenant links (D-242)...") - hq_settlements = corporations.populate_standalone_hq_settlements(conn, args.dry_run) - print( - f" {hq_settlements['standalone_inserted']} Standalone-HQ settlements, " - f"{hq_settlements['tenant_linked']} CityTenant links " - f"({hq_settlements['tenant_unmatched']} unmatched)" - ) - - # 13b. Per-settlement population + settlement_class bake (D-242, T-1075). - # Runs over the CORRECTED pool — after both T-1074 steps, so - # Standalone-HQ settlements are included in the rank-size spread, not - # bolted on after. - print(" [13b/13] Baking settlement population + settlement_class (D-242)...") - pop_bake = atlas.populate_settlement_population_class(conn, args.dry_run) - print( - f" {pop_bake['cities_populated']} cities populated across " - f"{pop_bake['bodies_spread']} bodies, {pop_bake['name_locked_applied']} " - f"NameLocked pins applied" - ) - - # 14. Architecture-flavor trait templates (D-232, #993). Catalog first, - # then sparse per-body hero bias (FK -> trait_templates + bodies). - print(" [14/15] Baking trait_templates catalog (D-232)...") - n_templates = traits.populate_trait_templates(conn, args.dry_run) - print(f" {n_templates} trait templates") - print(" [15/15] Baking atlas_body_trait_bias hero pins (D-232)...") - n_bias = traits.populate_atlas_body_trait_bias(conn, args.dry_run) - print(f" {n_bias} body trait-bias rows") - - # 16. axial_tilt_deg from body-def frontmatter (T-1024, D-239 §2) - print(" [16/17] Populating axial_tilt_deg from body-def frontmatter...") - n_tilt = bodies.populate_axial_tilt_deg(conn, args.dry_run) - print(f" {n_tilt} bodies updated with axial_tilt_deg") - - # 17. biosphere_class from body frontmatter override + two-gate default (D-247, T-1085) - print(" [17/19] Populating biosphere_class (D-247)...") - n_bio = bodies.populate_biosphere_class(conn, args.dry_run) - print(f" {n_bio} bodies updated with biosphere_class") - - # 18. Architecture zone-type bias table (T-988, D-235 step 2) — must - # follow trait_templates (V-TT-06 validates against its visual_bundle). - print(" [18/19] Baking architecture_zone_bias table (D-235)...") - n_zone_bias = traits.populate_architecture_zone_bias(conn, args.dry_run) - print(f" {n_zone_bias} zone-bias rows") - - # 19. Color register bands (T-988, D-235) — numeric HSV sampling bands - # per trait-template color_register; also follows trait_templates (V-TT-07). - print(" [19/19] Baking color_register_bands table (D-235)...") - n_color_bands = traits.populate_color_register_bands(conn, args.dry_run) - print(f" {n_color_bands} color register bands") - - # Validate structural integrity (FK, chain refs, chain completeness). - # These errors indicate broken imported data — do NOT commit. - print("\n Validating structural integrity...") - struct_errors = validators.validate(conn) - struct_errors.extend(brands.validate_brands(conn)) - if struct_errors: - print(f" STRUCTURAL ERRORS ({len(struct_errors)}) — rolling back:") - for e in struct_errors: - print(f" - {e}") - raise ImportAborted() - print(" FK integrity, chain completeness, and brand layer (V-B01–V-B06) OK") - - # Commit all imported data (corps, presence, etc.) before coverage check. - # Coverage validation is a Phase 2 gate (D-175) — data should be persisted - # so tools can query it and report gaps clearly. - if not args.dry_run: - conn.commit() - print(" Data committed.") - else: - # Dry-run: leave the transaction open so the coverage check below - # can still SELECT against the in-memory imported data. The - # transaction is discarded when conn.close() runs on exit. - print(" Dry run — no changes written.") - except ImportAborted: - conn.rollback() - conn.close() - sys.exit(1) - except BaseException: - # Any other exception (KeyboardInterrupt, MemoryError, DB error, - # programmer error) triggers a rollback so the DB is never left in - # a half-imported state. Re-raise so the user sees the traceback. - conn.rollback() - conn.close() - raise - - # Stamp generator metadata (#855, #856): record source SHAs so the - # pre-push hook can detect stale DB snapshots. Written BEFORE the - # coverage gate — the stamp records generator execution (code version), - # not data completeness. Coverage gaps (#860) are pre-existing data - # issues and must not prevent the stamp from landing. - if not args.dry_run: - try: - stamp.write_stamp(conn, "import_economics", *IMPORT_ECONOMICS_SOURCES) - conn.commit() - print(" Stamped: import_economics (covers brand pipeline Rust sources)") - except Exception as exc: # noqa: BLE001 - print(f" WARNING: failed to write generator stamp: {exc}", file=sys.stderr) - - # Validate coverage (hard errors per D-175, but after commit so data is usable). - print("\n Validating coverage (D-175 Phase 2 gate)...") - coverage_errors: list[str] = [] - commodity_ids_for_coverage = { - r[0] for r in conn.execute("SELECT commodity_id FROM commodities").fetchall() - } - coverage_errors.extend( - validators.validate_commodity_coverage(conn, wiki_corps, commodity_ids_for_coverage) - ) - coverage_errors.extend(validators.validate_system_coverage(conn, wiki_corps)) - - if coverage_errors: - print(f" COVERAGE ERRORS ({len(coverage_errors)}) — Phase 2 gate not met:") - for e in coverage_errors: - print(f" - {e}") - print("\n Data committed but Phase 2 gate is NOT met. " - "Add corporations to meet coverage thresholds and re-run.") - conn.close() - sys.exit(2) # exit 2 = coverage warning (data+stamp committed); exit 1 = real error - else: - print(" All coverage thresholds met — Phase 2 gate PASSED.") - - conn.close() - - print(f"\n Done: {n_links} gate_links, {n_commodities} commodities, " - f"{n_chains} chains, {n_inputs} inputs, {n_presence} corp_presence, " - f"{n_brands} brand_products, {n_brand_inputs} brand_inputs, " - f"{n_fiscal} system_fiscal\n") - - -if __name__ == "__main__": - main() diff --git a/tooling/generator_sources.py b/tooling/generator_sources.py index 345edf8b6..65f8efab1 100644 --- a/tooling/generator_sources.py +++ b/tooling/generator_sources.py @@ -5,20 +5,21 @@ generator_sources — single source of truth for systems.db generator source set Each generator that writes a stamp row to systems.db's ``meta`` table records the SHA-1 of the concatenated bytes of its source files (sorted by path). This module is the ONE place that defines which files belong to each generator's stamped set -(T-1067). It replaces the formerly triplicated lists in: - - - ``IMPORT_ECONOMICS_SOURCES`` in tooling/economy-db/import_economics.py - - ``GENERATOR_SOURCES`` in tooling/check-systems-db-stamp - - the hardcoded watch list in .claude/skills/pr-process/SKILL.md - (now derived via ``python3 tooling/generator_sources.py --list``) +(T-1067). It replaced three formerly separate lists (in the importer, in the +old check-systems-db-stamp script, and in the pr-process skill). Consumers: - - tooling/economy-db (the importer) — imports ``IMPORT_ECONOMICS_SOURCES`` and - ``file_sha1`` to write the stamp after a successful run. - - tooling/check-systems-db-stamp — imports ``GENERATOR_SOURCES`` and + - the ledger importer (tooling/domains/ledger/) — imports + ``IMPORT_ECONOMICS_SOURCES`` and ``file_sha1`` to write the stamp after a + successful run. + - ``reach check systems-db-stamp`` — imports ``GENERATOR_SOURCES`` and ``file_sha1`` to verify the stamp before a push. + - ``reach pr watchlist-diff`` — reads ``GENERATOR_SOURCES`` for its watch set. - /pr-process (skill) — shells out to the ``--list`` CLI for its watch list. +Stays a module rather than a verb (tooling/DOMAINS.md): it is a list of paths +read by gates, and nothing types it. + This file is itself a member of the stamped source set: a registry change (adding or removing a source) must flip the stamp and force ``make regen-db``, otherwise edits to the list would be invisible to staleness detection. @@ -44,10 +45,10 @@ REPO_ROOT = Path(__file__).resolve().parent.parent # Entrypoint + split module package (T-1067). IMPORT_ECONOMICS_ENTRYPOINT: Path = ( - REPO_ROOT / "tooling" / "economy-db" / "import_economics.py" + REPO_ROOT / "tooling" / "domains" / "ledger" / "service.py" ) ECONOMY_IMPORT_PACKAGE_DIR: Path = ( - REPO_ROOT / "tooling" / "economy-db" / "economy_import" + REPO_ROOT / "tooling" / "domains" / "ledger" / "economy_import" ) # Rust sources for the generate_brands subroutine. import_economics runs the @@ -163,7 +164,7 @@ def _corporation_pages() -> tuple[Path, ...]: # Canonical source set for import_economics' meta stamp. Covers the Python # entrypoint and its module package, the Rust binary it invokes (generate_brands -# main.rs + names.rs + surname corpus + wrapper script), the shared schema +# main.rs + names.rs + surname corpus + core/process.py), the shared schema # version constant, the authored data TOMLs, the corp wiki pages whose # frontmatter the importer bakes (T1, PR #177), and this registry itself. IMPORT_ECONOMICS_SOURCES: tuple[Path, ...] = ( @@ -175,7 +176,7 @@ IMPORT_ECONOMICS_SOURCES: tuple[Path, ...] = ( GENERATE_BRANDS_NAMES_RS, GENERATE_BRANDS_SURNAMES_RS, REPO_ROOT / "tooling" / "core" / "process.py", # runs the brand binary (T-1286) - REPO_ROOT / "tooling" / "schema_version.py", + REPO_ROOT / "tooling" / "domains" / "ledger" / "schema_version.py", SPECIALIZATION_VOCAB_TOML, SYSTEM_SPECIALIZATION_TOML, CORP_HQ_PLACEMENT_TOML, @@ -198,7 +199,7 @@ IMPORT_ECONOMICS_SOURCES: tuple[Path, ...] = ( # The atlas geometry generator (generate_atlas) was retired in #951 (D-223). # import_economics is now the sole regen-db generator that writes systems.db; it # owns the atlas index (names-only pool + empty geometry tables). The surviving -# planet-gen importers (import_heightmaps, import_province_boundaries) are +# `reach atlas planet` importers (import-heightmaps, import-provinces) are # one-time build imports baked into the committed DB, not part of regen-db, so # they are intentionally not stamped here. GENERATOR_SOURCES: dict[str, tuple[Path, ...]] = { diff --git a/tooling/main.py b/tooling/main.py index 248ba1e86..9d4b7594e 100644 --- a/tooling/main.py +++ b/tooling/main.py @@ -59,6 +59,10 @@ DOMAINS: dict[str, tuple[str, str]] = { "tooling.domains.atlas.router:app", "The spatial ladder — authoring, inspection and the DB", ), + "ledger": ( + "tooling.domains.ledger.router:app", + "The economics pipeline — the import that builds systems.db", + ), "godot": ( "tooling.domains.godot.router:app", "Does the client parse, and does it parse cold", diff --git a/tooling/economy-db/test_atlas_idempotency.py b/tooling/test_ledger_atlas_idempotency.py similarity index 95% rename from tooling/economy-db/test_atlas_idempotency.py rename to tooling/test_ledger_atlas_idempotency.py index 349075a6d..8f4e71178 100644 --- a/tooling/economy-db/test_atlas_idempotency.py +++ b/tooling/test_ledger_atlas_idempotency.py @@ -17,14 +17,14 @@ would drift from the real FK web this populator depends on). Wiki content (`wiki/star-systems/*/bodies/*/markers.json`) is read directly from the repo — read-only input, safe to reuse as-is; only the DB connection is scratch. Deliberately calls the two populator functions directly rather than the full -`import_economics.py` CLI: the CLI's first step shells out to the Rust +`reach ledger import`: its first step shells out to the Rust `generate_brands` binary and overwrites `generated_brands.toml` on disk (`brands.regenerate_brands`), a real side effect on shared repo content that has nothing to do with atlas name-pool idempotency and would make this test depend on a Rust build. Stdlib only (unittest) — run directly or via `make test-tooling`: - python3 tooling/economy-db/test_atlas_idempotency.py + .venv/bin/python tooling/test_ledger_atlas_idempotency.py """ import shutil @@ -34,11 +34,10 @@ import tempfile import unittest from pathlib import Path -sys.path.insert(0, str(Path(__file__).resolve().parent)) sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) -from economy_import import atlas # noqa: E402 -from economy_import.paths import DB_PATH # noqa: E402 +from tooling.domains.ledger.economy_import import atlas # noqa: E402 +from tooling.domains.ledger.economy_import.paths import DB_PATH # noqa: E402 class AtlasNamePoolIdempotencyTests(unittest.TestCase): diff --git a/tooling/economy-db/test_traits.py b/tooling/test_ledger_traits.py similarity index 96% rename from tooling/economy-db/test_traits.py rename to tooling/test_ledger_traits.py index ac9da5d13..9f6a2041e 100644 --- a/tooling/economy-db/test_traits.py +++ b/tooling/test_ledger_traits.py @@ -9,7 +9,7 @@ shape/coverage checks (T-988) — the `make test-tooling` dry-run only exercises the happy path against the committed, already-valid content. Stdlib only (unittest) — run directly or via `make test-tooling`: - python3 tooling/economy-db/test_traits.py + .venv/bin/python tooling/test_ledger_traits.py """ import json @@ -19,11 +19,10 @@ import tempfile import unittest from pathlib import Path -sys.path.insert(0, str(Path(__file__).resolve().parent)) sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) -from economy_import import traits # noqa: E402 -from economy_import.errors import ImportAborted # noqa: E402 +from tooling.domains.ledger.economy_import import traits # noqa: E402 +from tooling.domains.ledger.economy_import.errors import ImportAborted # noqa: E402 def write_registry(tmp: Path, body: str) -> Path: @@ -247,7 +246,7 @@ class CatalogCrossCheckTests(unittest.TestCase): catalog.write_text(catalog_toml, encoding="utf-8") traits.ARCHITECTURE_TRAIT_CATALOG_TOML = catalog out = io.StringIO() - with contextlib.redirect_stdout(out): + with contextlib.redirect_stderr(out): # console events go to stderr (T-1289) with self.assertRaises(ImportAborted): traits.populate_trait_templates(self.conn, dry_run=True) return out.getvalue() @@ -365,7 +364,7 @@ class ZoneBiasValidationTests(unittest.TestCase): import io out = io.StringIO() - with contextlib.redirect_stdout(out): + with contextlib.redirect_stderr(out): # console events go to stderr (T-1289) with self.assertRaises(ImportAborted): self.populate( '[bias.test_template.industrial_freight]\n' @@ -397,7 +396,7 @@ class ZoneBiasValidationTests(unittest.TestCase): for bad in ("wall = 15000\n", 'wall = ["brick_wall"]\n'): out = io.StringIO() - with contextlib.redirect_stdout(out): + with contextlib.redirect_stderr(out): # console events go to stderr (T-1289) with self.assertRaises(ImportAborted): self.populate(f"[bias.test_template.commercial_market]\n{bad}") self.assertIn("V-TT-06", out.getvalue()) @@ -461,7 +460,7 @@ class ColorRegisterBandTests(unittest.TestCase): import io out = io.StringIO() - with contextlib.redirect_stdout(out): + with contextlib.redirect_stderr(out): # console events go to stderr (T-1289) with self.assertRaises(ImportAborted): # A band for a DIFFERENT register — neutral_grey (referenced by # test_template above) has no band at all.