refactor(tooling): T-1289 — economy-db becomes reach ledger
The sole generator of systems.db moves to tooling/domains/ledger/ and is now `reach ledger import`. economy_import/ keeps its name (Rust comments in server/src cite it); the entrypoint becomes service.py; schema_version.py moves with the importer, which is where the version is defined. The stamp survived the move, which is the thing that had to hold: - generated_brands.toml is byte-identical (sha256 e748531…) before and after - `reach check systems-db-stamp` reported STALE after the move (the registry saw it) and OK after the regen - the dry-run carries every count and warning of the baseline transcript, and exit 2 — imported and stamped, coverage gate unmet — still reaches the caller through @command `make regen-db` survives as a one-line delegate, per D-263's muscle-memory clause: about fifty files name it, including the headers of generated wiki TOMLs and the remedies the push gate prints. `make economy-db` is retired; it ran `reach generate brands` before the import, which the import already does as its first step. economy_import.errors is reconciled as DOMAINS.md asked. ImportAborted stays as internal rollback control flow and never reaches a caller; the service converts it to a ReachError carrying the remedy. regenerate_brands caught cargo_binary's ReachError, printed it and raised ImportAborted, dropping the remedy. It runs before the import transaction opens, so there is nothing to roll back — it now propagates. Both sys.path bootstraps are gone; they existed only because the directory was hyphenated. The step labels ran [1/10]…[10/13]…[17/19]; one 24-step counter now drives the event phase and progress. Stale pointers fixed on the way: MIGRATION_SQL has lived in economy_import/migration.py since T-1067, but the asset-pipeline rule, DEVOPS and the schema comments still sent readers to import_economics.py; the rule and DEVOPS also still named the check scripts T-1281 retired. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
+17
-11
@@ -89,7 +89,7 @@ invented.
|
||||
| `atlas` | **the whole spatial ladder** (D-191). Flat verbs for authoring and inspection; nested groups per rung for generation | flat: `atlas` (Rust binary), `atlas-check`, `atlas-names`, `atlas-commit-and-sync`, `atlas-systems-done`, `atlas-update-field`, `atlas-verify`, `atlas-flatness` · `atlas map`: the 5 star-map files · `atlas planet`: ✅ ported (T-1288) from `planet-gen/` (30) — ten verbs, each restating its module's options for real `--help`; `test_planet_router.py` fails if a router option drifts from the module parser |
|
||||
| ~~`starmap`~~ | **folded into `atlas map`** — the top rung of the same ladder | — |
|
||||
| ~~`planet`~~ | **folded into `atlas planet`** — the third rung of the same ladder | — |
|
||||
| `ledger` | the economics pipeline, named for the UI component that will aggregate it | `economy-db/` (17 files), `schema_version.py` |
|
||||
| `ledger` | the economics pipeline, named for the UI component that will aggregate it | ✅ ported (T-1289). `economy-db/` → `domains/ledger/` (`economy_import/` kept by name; the entrypoint became `service.py`), `schema_version.py` with it. `reach ledger import`; `make regen-db` survives as a one-line delegate, `make economy-db` retired. `generated_brands.toml` byte-identical across the move |
|
||||
| `wiki` | wiki sync and content maintenance | `wiki/`, `db/wiki_sync.py`, `db/populate_gttr_hook.py`, `db/backfill_cultural_corridor.py`, `assign-astro-ids.py`, `fill-missing-globes.py`, `migrate-s-to-gj.py`, `patch-core-sector.py`, `process-wiki-system-changes` |
|
||||
| `assets` | connectors to the tower-of-joy generators | `db/audio_*.py`, `db/audio-*`, `db/image_connector.py`, `db/trellis_connector.py`, `db/common.py`, `trellis-batch.sh`, `synth_ui_sounds.py` |
|
||||
| `character` | bodies, garments, GLB handling | `garment-fit/make_logo.py`, `garment-qa/analyze_captures.py`, `convert_outfit.py`, `glb_strip_utility_nodes.py`, `inspect_glb.py`, `check_hair_symmetry.py`, `check_icosphere.py`, `render_quaternius_test.py`, `setup_clothing_metadata.py` — **note this is far smaller than `garment-fit/`'s file count suggests; 22 of its 23 files are Blender payloads and belong to the carve-out** |
|
||||
@@ -110,11 +110,13 @@ They move beside the domain that reads them (`check` and `db` respectively) as
|
||||
plain modules, not verbs. Making them commands would put something in
|
||||
`reach --help` that answers no question a person has.
|
||||
|
||||
**`schema_version.py` belongs to `db`, and `check` imports it.** It is consumed
|
||||
by the economics importer *and* by the stamp gate. Shared, but not equally
|
||||
owned: the importer defines the version, the gate reads it. It goes where it is
|
||||
defined, and the cross-domain import is legitimate — that is what a service
|
||||
layer is for.
|
||||
**`schema_version.py` belongs to `ledger` (named `db` when this map was
|
||||
drawn), and `check` reads it.** It is consumed by the economics importer *and*
|
||||
by the stamp gate. Shared, but not equally owned: the importer defines the
|
||||
version, the gate reads it. It goes where it is defined. *As ported (T-1289)
|
||||
the gate never imports the module at all — it reads the version back out of
|
||||
the `meta` table and checks it is semver — so the cross-domain import this
|
||||
ruling allowed for turned out not to be needed.*
|
||||
|
||||
**`test_*.py` files do NOT become a domain.** They are gate tests run by
|
||||
`make test-tooling`, not commands anyone types. `reach test …` would imply a
|
||||
@@ -138,14 +140,18 @@ data surface), `planet-gen/atlas_*.py` (`atlas_cohesion_audit`,
|
||||
`atlas_common`, `atlas_quality_analysis` — quality analysis of generated
|
||||
terrain), and `economy-db/atlas.py` (the atlas index tables in `systems.db`).
|
||||
These are three concerns sharing a noun, not one domain in three places. Each
|
||||
stays with its owner — `atlas`, `planet` and `db` respectively — and the port
|
||||
should resist the pull to collect them, which would produce a domain whose only
|
||||
common thread is a word.
|
||||
stays with its owner — `atlas`, `atlas planet` and `ledger` respectively — and
|
||||
the port should resist the pull to collect them, which would produce a domain
|
||||
whose only common thread is a word. *Held through both ports (T-1288,
|
||||
T-1289): `domains/atlas/`, `domains/atlas/planet/atlas_*.py` and
|
||||
`domains/ledger/economy_import/atlas.py` are still three separate files.*
|
||||
|
||||
**`economy-db/errors.py` predates `core/errors.py` and is not the same thing.**
|
||||
Domain-local error types are fine; what must not happen is a silent merge, or a
|
||||
second `ReachError` with different semantics. Reconcile explicitly when `db` is
|
||||
ported.
|
||||
second `ReachError` with different semantics. *Reconciled in T-1289:
|
||||
`ImportAborted` stays, as internal control flow only — a step raises it to
|
||||
request a rollback. It never reaches a caller: `ledger/service.py` rolls back
|
||||
and converts it to a `ReachError` carrying the remedy.*
|
||||
|
||||
**`character` rather than `garment`.** D-263's sketch says `garment`, but the
|
||||
files cover bodies, hair, GLB utilities and Quaternius imports as well as
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
"""`ledger` — the economics pipeline (D-263).
|
||||
|
||||
Named for the implant app the player will use to browse markets, wealth and
|
||||
transactions: the economic counterpart to what the Atlas is for topography.
|
||||
Not `db`, which named a storage layer nobody looks at, and not `economics`,
|
||||
which names a subject rather than the thing on screen.
|
||||
|
||||
Formerly `tooling/economy-db/` plus `tooling/schema_version.py` (T-1289). The
|
||||
importer here is the SOLE generator of `server/data/systems.db`; its stamped
|
||||
source set lives in `tooling/generator_sources.py`, which is why moving these
|
||||
files changed the stamp and the move landed together with a regen.
|
||||
"""
|
||||
+8
-15
@@ -1,9 +1,8 @@
|
||||
"""
|
||||
economy_import — module package behind tooling/economy-db/import_economics.py.
|
||||
economy_import — the import steps behind `reach ledger import`.
|
||||
|
||||
Split out of the former single-file importer (T-1067). The entrypoint
|
||||
``import_economics.py`` remains the CLI (Makefile regen-db, make test-tooling)
|
||||
and orchestrates the single-transaction import; the steps live here:
|
||||
Split out of the former single-file importer (T-1067). `ledger/service.py`
|
||||
orchestrates the single-transaction import; the steps live here:
|
||||
|
||||
paths.py source-file path constants (re-exports stamped paths
|
||||
from tooling/generator_sources.py)
|
||||
@@ -24,15 +23,9 @@ and orchestrates the single-transaction import; the steps live here:
|
||||
Every module here is part of the import_economics meta-stamp source set —
|
||||
tooling/generator_sources.py globs this directory, so adding a module
|
||||
automatically extends staleness detection.
|
||||
|
||||
The sys.path bootstrap that used to sit here is gone (T-1289). It existed only
|
||||
because `tooling/economy-db/` was hyphenated and so not importable; inside the
|
||||
package, `tooling.generator_sources` and the ledger's `schema_version` resolve
|
||||
by ordinary import.
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Bootstrap: make tooling/ importable (generator_sources, schema_version)
|
||||
# before any submodule needs them. tooling/economy-db is not a package
|
||||
# (hyphenated dir), so the entrypoint puts it on sys.path and this package
|
||||
# adds tooling/ itself.
|
||||
_TOOLING_DIR = Path(__file__).resolve().parents[2]
|
||||
if str(_TOOLING_DIR) not in sys.path:
|
||||
sys.path.insert(0, str(_TOOLING_DIR))
|
||||
+5
-6
@@ -19,6 +19,7 @@ import sqlite3
|
||||
import tomllib
|
||||
|
||||
from .paths import SCHEMA_SQL, SETTLEMENT_NAME_LOCKED_TOML, WIKI_STAR_SYSTEMS
|
||||
from tooling.core import console
|
||||
|
||||
# Atlas geometry index tables (D-191). These hold computed positions — city
|
||||
# centres, road/river/rail polylines, ocean/mountain extents. Under D-223 the
|
||||
@@ -139,7 +140,7 @@ def populate_atlas_city_names(conn: sqlite3.Connection, dry_run: bool) -> int:
|
||||
|
||||
if skipped_bodies:
|
||||
unique = sorted(set(skipped_bodies))
|
||||
print(f" warning: {len(unique)} body dirs not in DB — skipped: {unique[:5]}")
|
||||
console.event(f"{len(unique)} body dirs not in DB — skipped: {unique[:5]}", level="warn")
|
||||
|
||||
if not dry_run:
|
||||
# D-242: corporations.headquarters_city_id FK-references
|
||||
@@ -236,7 +237,7 @@ def populate_atlas_feature_names(conn: sqlite3.Connection, dry_run: bool) -> int
|
||||
|
||||
if skipped_bodies:
|
||||
unique = sorted(set(skipped_bodies))
|
||||
print(f" warning: {len(unique)} body dirs not in DB — skipped: {unique[:5]}")
|
||||
console.event(f"{len(unique)} body dirs not in DB — skipped: {unique[:5]}", level="warn")
|
||||
|
||||
if not dry_run:
|
||||
conn.execute("DELETE FROM atlas_feature_names")
|
||||
@@ -467,10 +468,8 @@ def populate_settlement_population_class(conn: sqlite3.Connection, dry_run: bool
|
||||
name_locked_applied += match_count
|
||||
|
||||
if name_locked_unmatched:
|
||||
print(
|
||||
f" warning: {len(name_locked_unmatched)} NameLocked hero stanza(s) "
|
||||
f"matched no atlas_city_names row: {name_locked_unmatched}"
|
||||
)
|
||||
console.event(f"{len(name_locked_unmatched)} NameLocked hero stanza(s) "
|
||||
f"matched no atlas_city_names row: {name_locked_unmatched}", level="warn")
|
||||
|
||||
return {
|
||||
"bodies_spread": len(city_rows_by_body),
|
||||
+2
-1
@@ -5,6 +5,7 @@ import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
from .paths import WIKI_STAR_SYSTEMS
|
||||
from tooling.core import console
|
||||
|
||||
|
||||
def populate_body_radius_km(conn: sqlite3.Connection, dry_run: bool) -> int:
|
||||
@@ -290,7 +291,7 @@ def populate_biosphere_class(conn: sqlite3.Connection, dry_run: bool) -> int:
|
||||
if body_id and bc:
|
||||
bc = bc.strip().strip("'\"")
|
||||
if bc not in _VALID:
|
||||
print(f" WARNING: {fpath}: biosphere_class {bc!r} not in {_VALID}; ignored")
|
||||
console.event(f"{fpath}: biosphere_class {bc!r} not in {_VALID}; ignored", level="warn")
|
||||
continue
|
||||
override[body_id] = bc
|
||||
|
||||
+14
-22
@@ -1,18 +1,11 @@
|
||||
"""Brand layer (D-189, #827): generate_brands shell-out, TOML import, validation."""
|
||||
|
||||
import sqlite3
|
||||
import sys
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
|
||||
from .errors import ImportAborted
|
||||
from .paths import BRANDS_TOML, GENERATED_BRANDS_TOML, REPO_ROOT
|
||||
|
||||
# economy-db/ is hyphenated, so it is not importable as a package and cannot
|
||||
# reach `tooling.core` by normal import. The bootstrap goes away with T-1272
|
||||
# (the hyphen sweep); until then it is explicit rather than implied.
|
||||
if str(REPO_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(REPO_ROOT))
|
||||
from .paths import BRANDS_TOML, GENERATED_BRANDS_TOML
|
||||
from tooling.core import console
|
||||
|
||||
VALID_BRAND_CATEGORIES: set[str] = {
|
||||
"terroir", "heritage_craft", "tech_premium", "cultural",
|
||||
@@ -41,22 +34,21 @@ def regenerate_brands() -> None:
|
||||
wrapper was one of three identical copies of build-if-missing-then-exec, so
|
||||
it was retired into `core.process.cargo_binary` (T-1286) and this calls that
|
||||
helper instead. Same binary, same seed, same output.
|
||||
|
||||
A build or run failure propagates as the ReachError `cargo_binary` raised.
|
||||
It used to be printed and swapped for ImportAborted, which dropped its
|
||||
remedy (T-1289). Nothing needs rolling back: this runs before the import
|
||||
transaction opens.
|
||||
"""
|
||||
from tooling.core.errors import ReachError
|
||||
from tooling.core.process import cargo_binary
|
||||
|
||||
print(" [pre/10] Running generate_brands (Rust) to refresh generated_brands.toml...")
|
||||
try:
|
||||
stdout = cargo_binary("generate_brands")
|
||||
except ReachError as exc:
|
||||
print(str(exc), file=sys.stderr)
|
||||
raise ImportAborted() from exc
|
||||
console.event("Running generate_brands (Rust) to refresh generated_brands.toml...", phase="pre")
|
||||
stdout = cargo_binary("generate_brands")
|
||||
|
||||
# Print the Rust binary's own summary lines (brands generated, coverage).
|
||||
# Indent so they fold under the pre-step heading.
|
||||
# Relay the Rust binary's own summary lines (brands generated, coverage).
|
||||
for line in stdout.splitlines():
|
||||
if line.strip():
|
||||
print(f" {line}")
|
||||
console.event(line.strip(), phase="pre")
|
||||
|
||||
|
||||
def _load_brand_file(path: Path) -> tuple[list, list]:
|
||||
@@ -74,18 +66,18 @@ def import_brands(
|
||||
"""Import brand_products and brand_inputs from brands.toml and generated_brands.toml.
|
||||
|
||||
Hand-authored brands (brands.toml) are imported first; generated brands
|
||||
(generated_brands.toml, produced by `tooling/generate-brands`) are merged in.
|
||||
(generated_brands.toml, produced by `reach generate brands`) are merged in.
|
||||
Returns (n_products, n_inputs).
|
||||
"""
|
||||
if not BRANDS_TOML.exists():
|
||||
print(" warning: brands.toml not found — brand layer skipped")
|
||||
console.event("brands.toml not found — brand layer skipped", level="warn")
|
||||
return 0, 0
|
||||
|
||||
products_authored, inputs_authored = _load_brand_file(BRANDS_TOML)
|
||||
products_generated, inputs_generated = _load_brand_file(GENERATED_BRANDS_TOML)
|
||||
|
||||
if products_generated:
|
||||
print(f" merging {len(products_generated)} generated brand_products from generated_brands.toml")
|
||||
console.event(f"merging {len(products_generated)} generated brand_products from generated_brands.toml")
|
||||
|
||||
products = products_authored + products_generated
|
||||
inputs = inputs_authored + inputs_generated
|
||||
+14
-17
@@ -9,6 +9,7 @@ from pathlib import Path
|
||||
from .atlas import most_populated_body_in_system
|
||||
from .errors import ImportAborted
|
||||
from .paths import CORP_HQ_PLACEMENT_TOML, CORPORATIONS_DIR
|
||||
from tooling.core import console
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Corporation wiki parsing
|
||||
@@ -117,8 +118,8 @@ def sync_corporations(
|
||||
system_id = corp.get("system_id")
|
||||
hq_system = system_id if system_id and system_id in valid_systems else None
|
||||
if system_id and system_id not in valid_systems:
|
||||
print(f" warning: {corp_id} HQ system '{system_id}' not in DB, "
|
||||
f"headquarters_system set to NULL")
|
||||
console.event(f"{corp_id} HQ system '{system_id}' not in DB, "
|
||||
f"headquarters_system set to NULL", level="warn")
|
||||
to_insert.append((
|
||||
corp_id,
|
||||
proper_name,
|
||||
@@ -274,11 +275,9 @@ def populate_corp_specialization(
|
||||
r[0] for r in conn.execute("SELECT DISTINCT body_id FROM atlas_city_names").fetchall()
|
||||
}
|
||||
if not city_bearing_bodies:
|
||||
print(
|
||||
" warning: atlas_city_names is empty — the city-presence "
|
||||
console.event("atlas_city_names is empty — the city-presence "
|
||||
"tiebreak tier is inert this run (placements fall through to "
|
||||
"body-type rank; they converge on the next run)"
|
||||
)
|
||||
"body-type rank; they converge on the next run)", level="warn")
|
||||
|
||||
errors: list[str] = []
|
||||
unauthored: list[str] = []
|
||||
@@ -333,16 +332,14 @@ def populate_corp_specialization(
|
||||
spec_rows.append((corp_id, spec, placement, authored_body))
|
||||
|
||||
if errors:
|
||||
print(f" CORP SPECIALIZATION ERRORS ({len(errors)}):")
|
||||
console.event(f"CORP SPECIALIZATION ERRORS ({len(errors)}):", level="error")
|
||||
for e in errors:
|
||||
print(f" - {e}")
|
||||
console.event(f"- {e}", level="error")
|
||||
raise ImportAborted()
|
||||
|
||||
if unauthored:
|
||||
print(
|
||||
f" warning: {len(unauthored)} corp(s) have no authored "
|
||||
f"corp_specialization (stay NULL; no HQ placement): {unauthored[:5]}"
|
||||
)
|
||||
console.event(f"{len(unauthored)} corp(s) have no authored "
|
||||
f"corp_specialization (stay NULL; no HQ placement): {unauthored[:5]}", level="warn")
|
||||
|
||||
# Pre-compute the resolution so dry-run reports the same numbers a real
|
||||
# run writes (write is gated on dry_run; the derivation is not).
|
||||
@@ -440,7 +437,7 @@ def import_corp_presence(
|
||||
|
||||
if skipped:
|
||||
for s in skipped:
|
||||
print(f" warning: skipped corp_presence for {s}")
|
||||
console.event(f"skipped corp_presence for {s}", level="warn")
|
||||
|
||||
if not dry_run:
|
||||
conn.execute("DELETE FROM corp_presence")
|
||||
@@ -547,11 +544,11 @@ def populate_standalone_hq_settlements(conn: sqlite3.Connection, dry_run: bool)
|
||||
n_linked += 1
|
||||
|
||||
if unmatched_bodies:
|
||||
print(f" warning: {len(unmatched_bodies)} corp(s) skipped (bad headquarters_body): "
|
||||
f"{unmatched_bodies[:5]}")
|
||||
console.event(f"{len(unmatched_bodies)} corp(s) skipped (bad headquarters_body): "
|
||||
f"{unmatched_bodies[:5]}", level="warn")
|
||||
if unmatched_corps:
|
||||
print(f" warning: {len(unmatched_corps)} CityTenant corp(s) unmatched: "
|
||||
f"{unmatched_corps[:5]}")
|
||||
console.event(f"{len(unmatched_corps)} CityTenant corp(s) unmatched: "
|
||||
f"{unmatched_corps[:5]}", level="warn")
|
||||
|
||||
return {
|
||||
"standalone_inserted": n_inserted,
|
||||
+4
-3
@@ -6,6 +6,7 @@ import sqlite3
|
||||
import tomllib
|
||||
|
||||
from .paths import CHAINS_TOML, COMMODITIES_TOML, CURRENCY_ZONES_TOML, STAR_MAP
|
||||
from tooling.core import console
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Gate links
|
||||
@@ -33,7 +34,7 @@ def import_gate_links(conn: sqlite3.Connection, dry_run: bool) -> int:
|
||||
|
||||
if skipped:
|
||||
unique_skipped = sorted(set(skipped))
|
||||
print(f" warning: {len(unique_skipped)} system(s) in star-map.json not in DB: {unique_skipped[:5]}...")
|
||||
console.event(f"{len(unique_skipped)} system(s) in star-map.json not in DB: {unique_skipped[:5]}...", level="warn")
|
||||
|
||||
if not dry_run:
|
||||
conn.executemany(
|
||||
@@ -169,8 +170,8 @@ def set_currency_zones(conn: sqlite3.Connection, dry_run: bool) -> dict[str, int
|
||||
(sid,),
|
||||
)
|
||||
else:
|
||||
print(" warning: wiki/economics/currency_zones.toml not found — "
|
||||
"all systems default to TRACTUS_PRIMARY / Sol to MIXED")
|
||||
console.event("wiki/economics/currency_zones.toml not found — "
|
||||
"all systems default to TRACTUS_PRIMARY / Sol to MIXED", level="warn")
|
||||
|
||||
counts: dict[str, int | str] = {}
|
||||
for row in conn.execute("SELECT currency_zone, COUNT(*) FROM star_systems GROUP BY currency_zone"):
|
||||
+1
-1
@@ -8,7 +8,7 @@ importer can never disagree about where a stamped source lives.
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from generator_sources import (
|
||||
from tooling.generator_sources import (
|
||||
ARCHITECTURE_TRAIT_BIAS_TOML,
|
||||
ARCHITECTURE_TRAIT_CATALOG_TOML,
|
||||
ARCHITECTURE_ZONE_BIAS_TOML,
|
||||
+15
-16
@@ -6,6 +6,7 @@ from collections import Counter
|
||||
|
||||
from .errors import ImportAborted
|
||||
from .paths import SPECIALIZATION_VOCAB_TOML, SYSTEM_SPECIALIZATION_TOML
|
||||
from tooling.core import console
|
||||
|
||||
# Authoritative D-233 projected enums. The full CI guardrail suite (V-SES-*)
|
||||
# lands in #1015; this module performs the FK + enum sanity the import itself
|
||||
@@ -194,9 +195,9 @@ def import_system_specialization(conn: sqlite3.Connection, dry_run: bool,
|
||||
)
|
||||
|
||||
if errors:
|
||||
print(f" SPECIALIZATION ERRORS ({len(errors)}):")
|
||||
console.event(f"SPECIALIZATION ERRORS ({len(errors)}):", level="error")
|
||||
for e in errors:
|
||||
print(f" - {e}")
|
||||
console.event(f"- {e}", level="error")
|
||||
raise ImportAborted()
|
||||
|
||||
coverage: dict = {
|
||||
@@ -403,26 +404,24 @@ def _specialization_checks(
|
||||
if e and e in vocab:
|
||||
bulk_dist[vocab[e].get("bulk_class_projected")] += 1
|
||||
pu_dist[vocab[e].get("production_ubiquity_projected")] += 1
|
||||
print(" Specialization coverage (D-237):")
|
||||
print(f" economic: {n_econ} authored ({n_inhabited} inhabited; rest on fallback once #1014 lands)")
|
||||
print(f" cultural: {n_cult} authored ({n_named} named; rest on corridor default)")
|
||||
print(f" faction: {n_fac} authored ({n_named} named; rest on derivation)")
|
||||
print(" BulkClass: " + " | ".join(f"{k}={v}" for k, v in sorted(bulk_dist.items())))
|
||||
print(" ProductionUbiquity: " + " | ".join(f"{k}={v}" for k, v in sorted(pu_dist.items())))
|
||||
console.event("Specialization coverage (D-237):")
|
||||
console.event(f"economic: {n_econ} authored ({n_inhabited} inhabited; rest on fallback once #1014 lands)")
|
||||
console.event(f"cultural: {n_cult} authored ({n_named} named; rest on corridor default)")
|
||||
console.event(f"faction: {n_fac} authored ({n_named} named; rest on derivation)")
|
||||
console.event("BulkClass: " + " | ".join(f"{k}={v}" for k, v in sorted(bulk_dist.items())))
|
||||
console.event("ProductionUbiquity: " + " | ".join(f"{k}={v}" for k, v in sorted(pu_dist.items())))
|
||||
if warnings:
|
||||
print(f" Specialization warnings ({len(warnings)}):")
|
||||
console.event(f"Specialization warnings ({len(warnings)}):")
|
||||
for w in warnings:
|
||||
print(f" - {w}")
|
||||
console.event(f"- {w}", level="warn")
|
||||
|
||||
if gate_failures:
|
||||
if strict:
|
||||
print(f" SPECIALIZATION COMPLETENESS FAILURES ({len(gate_failures)}) [strict]:")
|
||||
console.event(f"SPECIALIZATION COMPLETENESS FAILURES ({len(gate_failures)}) [strict]:", level="error")
|
||||
for g in gate_failures:
|
||||
print(f" - {g}")
|
||||
console.event(f"- {g}", level="error")
|
||||
raise ImportAborted()
|
||||
else:
|
||||
print(
|
||||
f" Specialization completeness: {len(gate_failures)} gate item(s) "
|
||||
console.event(f"Specialization completeness: {len(gate_failures)} gate item(s) "
|
||||
f"pending (#1014 fallback / #1016 content pass) — warnings only until "
|
||||
f"--strict-specialization"
|
||||
)
|
||||
f"--strict-specialization")
|
||||
+3
-3
@@ -1,15 +1,15 @@
|
||||
"""Meta-table generator stamp (#855, #856).
|
||||
|
||||
Records the source-SHA of the generator that produced the DB so the pre-push
|
||||
hook (tooling/check-systems-db-stamp) can detect stale snapshots. The source
|
||||
hook (`reach check systems-db-stamp`) can detect stale snapshots. The source
|
||||
set comes from tooling/generator_sources.py — the single shared registry.
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
from generator_sources import file_sha1
|
||||
from schema_version import SCHEMA_VERSION
|
||||
from tooling.generator_sources import file_sha1
|
||||
from tooling.domains.ledger.schema_version import SCHEMA_VERSION
|
||||
|
||||
from .paths import SCHEMA_SQL
|
||||
|
||||
+11
-10
@@ -12,6 +12,7 @@ from .paths import (
|
||||
COLOR_REGISTER_BANDS_TOML,
|
||||
OBJECT_TAG_VOCABULARY_TOML,
|
||||
)
|
||||
from tooling.core import console
|
||||
|
||||
_TRAIT_CORRIDOR_POOLS: set[str] = {"baseline", "heritage", "cross_corridor"}
|
||||
_TRAIT_BIAS_KINDS: set[str] = {"pin", "boost", "suppress"}
|
||||
@@ -205,9 +206,9 @@ def populate_trait_templates(conn: sqlite3.Connection, dry_run: bool) -> int:
|
||||
json.dumps(t["visual_bundle"]) if t.get("visual_bundle") else None,
|
||||
))
|
||||
if errors:
|
||||
print(f" TRAIT TEMPLATE ERRORS ({len(errors)}):")
|
||||
console.event(f"TRAIT TEMPLATE ERRORS ({len(errors)}):", level="error")
|
||||
for e in errors:
|
||||
print(f" - {e}")
|
||||
console.event(f"- {e}", level="error")
|
||||
raise ImportAborted()
|
||||
# CI guardrails (D-232, Nigel): >=5 templates eligible per BulkClass (a
|
||||
# template with no bulk_class_gate is eligible for all); no single template
|
||||
@@ -232,9 +233,9 @@ def populate_trait_templates(conn: sqlite3.Connection, dry_run: bool) -> int:
|
||||
f"{w * 100 // total}% of the {bc} pool weight (>60%)"
|
||||
)
|
||||
if gerrors:
|
||||
print(f" TRAIT TEMPLATE GUARDRAIL FAILURES ({len(gerrors)}):")
|
||||
console.event(f"TRAIT TEMPLATE GUARDRAIL FAILURES ({len(gerrors)}):", level="error")
|
||||
for e in gerrors:
|
||||
print(f" - {e}")
|
||||
console.event(f"- {e}", level="error")
|
||||
raise ImportAborted()
|
||||
# Validation/guardrails run on dry-run too; only mutate when committing.
|
||||
if not dry_run:
|
||||
@@ -340,9 +341,9 @@ def populate_architecture_zone_bias(conn: sqlite3.Connection, dry_run: bool) ->
|
||||
rows.append((template_tag, zone_type_id, bias_json))
|
||||
|
||||
if errors:
|
||||
print(f" ARCHITECTURE ZONE BIAS ERRORS ({len(errors)}):")
|
||||
console.event(f"ARCHITECTURE ZONE BIAS ERRORS ({len(errors)}):", level="error")
|
||||
for e in errors:
|
||||
print(f" - {e}")
|
||||
console.event(f"- {e}", level="error")
|
||||
raise ImportAborted()
|
||||
|
||||
if not dry_run:
|
||||
@@ -432,9 +433,9 @@ def populate_color_register_bands(conn: sqlite3.Connection, dry_run: bool) -> in
|
||||
errors.append(f"V-TT-07: color_register '{reg}' referenced by trait_templates but has no band")
|
||||
|
||||
if errors:
|
||||
print(f" COLOR REGISTER BAND ERRORS ({len(errors)}):")
|
||||
console.event(f"COLOR REGISTER BAND ERRORS ({len(errors)}):", level="error")
|
||||
for e in errors:
|
||||
print(f" - {e}")
|
||||
console.event(f"- {e}", level="error")
|
||||
raise ImportAborted()
|
||||
|
||||
if not dry_run:
|
||||
@@ -507,9 +508,9 @@ def populate_atlas_body_trait_bias(conn: sqlite3.Connection, dry_run: bool) -> i
|
||||
f"K of {_MAX_PINS_PER_BODY} (ComplexityTier::Full); pins count toward K (D-232)"
|
||||
)
|
||||
if errors:
|
||||
print(f" TRAIT BIAS ERRORS ({len(errors)}):")
|
||||
console.event(f"TRAIT BIAS ERRORS ({len(errors)}):", level="error")
|
||||
for e in errors:
|
||||
print(f" - {e}")
|
||||
console.event(f"- {e}", level="error")
|
||||
raise ImportAborted()
|
||||
if not dry_run:
|
||||
conn.execute("DELETE FROM atlas_body_trait_bias")
|
||||
@@ -0,0 +1,47 @@
|
||||
"""Transport for the `ledger` domain — args in, delegate, format out."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import typer
|
||||
|
||||
from tooling.core import cli
|
||||
from tooling.core.command import command
|
||||
|
||||
app = cli.domain("ledger", "The economics pipeline — the import that builds systems.db.")
|
||||
|
||||
|
||||
@app.callback()
|
||||
def _domain() -> None:
|
||||
"""Keeps `ledger` a group (Typer collapses a single-command app)."""
|
||||
|
||||
|
||||
@app.command("import")
|
||||
@command
|
||||
def import_(
|
||||
db: Path = typer.Option(None, "--db", help="systems.db to import into (default: server/data/systems.db)."),
|
||||
dry_run: bool = typer.Option(False, "--dry-run", help="Validate every step, write nothing."),
|
||||
strict_specialization: bool = typer.Option(
|
||||
False,
|
||||
"--strict-specialization",
|
||||
help="Treat the D-237 completeness gates (V-SES-02, V-FAC-01) as hard errors.",
|
||||
),
|
||||
) -> None:
|
||||
"""Rebuild systems.db from the economics sources, and stamp it.
|
||||
|
||||
The sole generator of server/data/systems.db — `make regen-db` runs this.
|
||||
It regenerates generated_brands.toml first (skipped on --dry-run), imports
|
||||
in one transaction, and stamps the meta table so the push gate can tell a
|
||||
stale snapshot from a fresh one.
|
||||
|
||||
Exit 2 means imported AND stamped but the D-175 coverage gate is not met:
|
||||
a content gap, not a failure, and tolerated by regen-db and the tooling
|
||||
gate. Exit 1 means nothing was written.
|
||||
"""
|
||||
# Imported here, not at module level: the service pulls in the whole
|
||||
# importer package and the stamp registry, and `reach ledger --help`
|
||||
# should not pay for that (T-1260).
|
||||
from tooling.domains.ledger import service
|
||||
|
||||
service.run(db or service.DB_PATH, dry_run=dry_run, strict_specialization=strict_specialization)
|
||||
@@ -6,8 +6,9 @@ Bump SCHEMA_VERSION manually on any backwards-incompatible schema change
|
||||
Additive changes (new nullable columns, new tables, new indexes) do not
|
||||
require a bump.
|
||||
|
||||
Imported by:
|
||||
tooling/economy-db/import_economics.py
|
||||
Lives in `ledger` because the importer is where the version is DEFINED;
|
||||
`reach check systems-db-stamp` only reads it back out of the meta table
|
||||
(tooling/DOMAINS.md). Imported by economy_import/stamp.py.
|
||||
"""
|
||||
|
||||
SCHEMA_VERSION = "1.0.0"
|
||||
Executable
+369
@@ -0,0 +1,369 @@
|
||||
"""Logic for the `ledger` domain — the economics import into systems.db. Transport-agnostic (D-263).
|
||||
|
||||
Reads TOML/JSON source files and populates the economics tables:
|
||||
- gate_links from docs/design/star-map.json (335 edges, bidirectional)
|
||||
- commodities from wiki/economics/commodities.toml (36 types)
|
||||
- production_chains + chain_inputs from wiki/economics/production_chains.toml
|
||||
- currency_zone on star_systems (default TRACTUS_PRIMARY)
|
||||
- gate_energy_connected on star_systems (D-186: false for MARK_PRIMARY zones)
|
||||
- corporations from wiki/corporations/*.md (sync + insert new records)
|
||||
- corp_specialization/hq_placement/headquarters_body/headquarters_city_id
|
||||
on corporations (D-242: HQ-placement key + baked
|
||||
CityTenant link, from corp_hq_placement.toml)
|
||||
- corp_presence from wiki/corporations/*.md (headquarters location data)
|
||||
- atlas_city_names Standalone-HQ settlement rows (D-242, T-1074)
|
||||
|
||||
Validation (hard errors, non-zero exit on any failure):
|
||||
- Wiki corporation names must match DB proper_name records (D-182 sync constraint)
|
||||
- Chain completeness: every intermediate commodity has at least one production chain
|
||||
- Commodity coverage: 3+ corporations per major commodity type (D-175)
|
||||
- System coverage: 1+ corporation per inhabited system with population > 100K (D-175)
|
||||
|
||||
This is the sole generator of `server/data/systems.db` and was
|
||||
`tooling/economy-db/import_economics.py` until T-1289. The steps live in the
|
||||
`economy_import` package; the stamped source set is defined in
|
||||
`tooling/generator_sources.py`.
|
||||
|
||||
**Exit codes are a contract, kept exactly.** 0 = imported and every coverage
|
||||
threshold met. 2 = imported AND stamped, but the D-175 coverage gate is not met
|
||||
— `make regen-db` and the tooling gate both tolerate it, because the data is
|
||||
usable and the gap is content, not code. 1 = nothing written.
|
||||
|
||||
**Two error types, deliberately.** `economy_import.errors.ImportAborted` is
|
||||
internal control flow — a step raises it to request rollback after printing
|
||||
what went wrong. It never escapes this module: `run()` rolls back and converts
|
||||
it to a `ReachError` carrying the remedy. That is the explicit reconciliation
|
||||
DOMAINS.md asked for, rather than a second error type with different semantics
|
||||
reaching the caller.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
from tooling.core import console
|
||||
from tooling.core.errors import ReachError
|
||||
from tooling.domains.ledger.economy_import import (
|
||||
atlas,
|
||||
bodies,
|
||||
brands,
|
||||
corporations,
|
||||
db,
|
||||
economy,
|
||||
migration,
|
||||
specialization,
|
||||
stamp,
|
||||
traits,
|
||||
validators,
|
||||
)
|
||||
from tooling.domains.ledger.economy_import.errors import ImportAborted
|
||||
from tooling.domains.ledger.economy_import.paths import DB_PATH
|
||||
from tooling.generator_sources import IMPORT_ECONOMICS_SOURCES
|
||||
|
||||
__all__ = ["DB_PATH", "run"]
|
||||
|
||||
# Named once, so the progress fraction and the count cannot disagree.
|
||||
STEPS = 24
|
||||
|
||||
ABORTED_FIX = (
|
||||
"fix the errors listed above in their source (wiki/economics/*.toml, "
|
||||
"wiki/corporations/*.md), then re-run: reach ledger import --dry-run"
|
||||
)
|
||||
|
||||
|
||||
class _Steps:
|
||||
"""Numbered progress through the import, one event per step."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.n = 0
|
||||
|
||||
def __call__(self, message: str) -> None:
|
||||
self.n += 1
|
||||
console.event(message, phase=f"{self.n}/{STEPS}", progress=self.n / STEPS)
|
||||
|
||||
|
||||
def run(db_path: Path, *, dry_run: bool = False, strict_specialization: bool = False) -> None:
|
||||
"""Import every economics source into `db_path`, in one transaction.
|
||||
|
||||
Raises ReachError with exit code 1 when nothing was written, and with exit
|
||||
code 2 when the import committed but the coverage gate is not met.
|
||||
"""
|
||||
if not db_path.exists():
|
||||
raise ReachError(
|
||||
f"{db_path} not found",
|
||||
fix="pass --db with an existing systems.db, or restore it: git restore server/data/systems.db",
|
||||
)
|
||||
|
||||
console.event(f"Economics import into {db_path}" + (" — DRY RUN" if dry_run else ""))
|
||||
|
||||
# Load wiki corps before opening the DB — allows early exit on parse failures.
|
||||
console.event("Loading wiki corporations...")
|
||||
wiki_corps = corporations.load_wiki_corps()
|
||||
console.event(f"{len(wiki_corps)} corporation files parsed")
|
||||
|
||||
# Regenerate generated_brands.toml via the Rust binary before the import
|
||||
# reads it — single pipeline, single stamp (review T2/H3). Skipped on a
|
||||
# dry run to avoid a disk side effect during validation.
|
||||
if not dry_run:
|
||||
brands.regenerate_brands()
|
||||
|
||||
conn = db.connect(db_path)
|
||||
try:
|
||||
counts = _import(conn, wiki_corps, dry_run, strict_specialization)
|
||||
except ImportAborted:
|
||||
conn.rollback()
|
||||
conn.close()
|
||||
raise ReachError(
|
||||
"economics import aborted — rolled back, nothing written", fix=ABORTED_FIX
|
||||
) from None
|
||||
except BaseException:
|
||||
# Anything else (KeyboardInterrupt, MemoryError, a DB error, a bug)
|
||||
# rolls back too, so the DB is never left half-imported. Re-raised so
|
||||
# the traceback survives.
|
||||
conn.rollback()
|
||||
conn.close()
|
||||
raise
|
||||
|
||||
# Stamp generator metadata (#855, #856) so the pre-push hook can detect a
|
||||
# stale snapshot. Written BEFORE the coverage gate: the stamp records which
|
||||
# code produced the DB, not whether the data is complete (#860).
|
||||
if not dry_run:
|
||||
try:
|
||||
stamp.write_stamp(conn, "import_economics", *IMPORT_ECONOMICS_SOURCES)
|
||||
conn.commit()
|
||||
console.event("Stamped: import_economics (covers brand pipeline Rust sources)")
|
||||
except Exception as exc: # noqa: BLE001
|
||||
console.event(f"failed to write generator stamp: {exc}", level="warn")
|
||||
|
||||
coverage_errors = _coverage(conn, wiki_corps)
|
||||
conn.close()
|
||||
|
||||
summary = (
|
||||
f"{counts['links']} gate_links, {counts['commodities']} commodities, "
|
||||
f"{counts['chains']} chains, {counts['inputs']} inputs, "
|
||||
f"{counts['presence']} corp_presence, {counts['brands']} brand_products, "
|
||||
f"{counts['brand_inputs']} brand_inputs, {counts['fiscal']} system_fiscal"
|
||||
)
|
||||
if coverage_errors:
|
||||
for e in coverage_errors:
|
||||
console.event(e, level="error")
|
||||
raise ReachError(
|
||||
f"imported ({summary}) but the D-175 Phase 2 coverage gate is not met — "
|
||||
f"{len(coverage_errors)} gap(s) above. "
|
||||
+ ("Nothing was written (dry run)." if dry_run else "Data and stamp are committed."),
|
||||
fix="add corporations to wiki/corporations/ until the thresholds are met, then re-run",
|
||||
exit_code=2,
|
||||
)
|
||||
|
||||
written = "validated, nothing written (dry run)" if dry_run else "imported and stamped"
|
||||
console.verdict(f"ledger import: {written} — {summary}")
|
||||
|
||||
|
||||
def _import(
|
||||
conn: sqlite3.Connection, wiki_corps, dry_run: bool, strict_specialization: bool
|
||||
) -> dict[str, int]:
|
||||
"""The clear-then-reimport cycle, as one explicit transaction.
|
||||
|
||||
Any crash, validation error or interrupt between the first DELETE and the
|
||||
commit rolls everything back (the caller owns that): the DB never ends up
|
||||
half-cleared. On success it commits exactly once, right after structural
|
||||
validation passes. On a dry run the transaction is left open so the
|
||||
coverage check can still SELECT the imported rows; closing discards it.
|
||||
"""
|
||||
step = _Steps()
|
||||
conn.execute("BEGIN")
|
||||
|
||||
# Schema migration — idempotent, inside the tx so a crash leaves no
|
||||
# half-applied ALTER TABLE.
|
||||
step("Schema migration...")
|
||||
migration.apply_schema_migrations(conn)
|
||||
# Atlas index tables: canonical DDL + empty geometry (D-223, #951).
|
||||
atlas.ensure_atlas_index_schema(conn, dry_run)
|
||||
|
||||
# Clear economics tables in FK-safe order (children before parents).
|
||||
if not dry_run:
|
||||
db.clear_economics_tables(conn)
|
||||
|
||||
step("Importing gate links...")
|
||||
n_links = economy.import_gate_links(conn, dry_run)
|
||||
console.event(f"{n_links} rows (bidirectional)")
|
||||
|
||||
step("Importing commodities...")
|
||||
n_commodities = economy.import_commodities(conn, dry_run)
|
||||
console.event(f"{n_commodities} commodities")
|
||||
|
||||
step("Importing production chains...")
|
||||
n_chains, n_inputs = economy.import_chains(conn, dry_run)
|
||||
console.event(f"{n_chains} chains, {n_inputs} inputs")
|
||||
|
||||
# D-237 authored layer — after commodities (FK), before currency zones.
|
||||
# UPSERTs onto pre-existing system_economy / system_factions rows;
|
||||
# unauthored systems stay NULL.
|
||||
step("Importing system specialization (D-237)...")
|
||||
spec = specialization.import_system_specialization(conn, dry_run, strict=strict_specialization)
|
||||
console.event(
|
||||
f"vocab {spec['vocab']} | economic {spec['economic']} | "
|
||||
f"cultural {spec['cultural']} | faction {spec['faction']}"
|
||||
)
|
||||
if spec["missing_economy_row"]:
|
||||
console.event(
|
||||
f"{len(spec['missing_economy_row'])} authored system(s) lack a "
|
||||
f"system_economy row (values dropped): {spec['missing_economy_row']}",
|
||||
level="warn",
|
||||
)
|
||||
if spec["missing_faction_row"]:
|
||||
console.event(
|
||||
f"{len(spec['missing_faction_row'])} authored system(s) lack a "
|
||||
f"system_factions row (faction dropped): {spec['missing_faction_row']}",
|
||||
level="warn",
|
||||
)
|
||||
|
||||
step("Setting currency zones...")
|
||||
for zone, count in sorted(economy.set_currency_zones(conn, dry_run).items()):
|
||||
console.event(f"{zone}: {count}")
|
||||
|
||||
# D-186 — must run after currency zones.
|
||||
step("Setting gate energy connectivity...")
|
||||
for label, count in sorted(economy.set_gate_energy(conn, dry_run).items()):
|
||||
console.event(f"{label}: {count}")
|
||||
|
||||
# D-182: a name divergence is a hard error.
|
||||
step("Syncing corporations...")
|
||||
corp_errors = corporations.sync_corporations(conn, wiki_corps, dry_run)
|
||||
if corp_errors:
|
||||
console.event(f"name divergence detected (D-182) — {len(corp_errors)} error(s):", level="error")
|
||||
for e in corp_errors:
|
||||
console.event(e, level="error")
|
||||
console.event("update the wiki title or the DB proper_name to match, then re-run", level="error")
|
||||
raise ImportAborted()
|
||||
n_db_corps = conn.execute("SELECT COUNT(*) FROM corporations").fetchone()[0]
|
||||
console.event(f"{n_db_corps} corporations in DB ({len(wiki_corps)} from wiki)")
|
||||
|
||||
# D-242 Phase A — reset + derive every run (PR #177 H1/T2). After corp
|
||||
# sync (rows must exist), before corp_presence (reads headquarters_body
|
||||
# back), and before the city-presence tiebreak further down, which reads
|
||||
# atlas_city_names BEFORE this run's rebuild — the previous run's settled
|
||||
# state, by design.
|
||||
step("Importing corp specialization + HQ placement (D-242)...")
|
||||
corp_spec = corporations.populate_corp_specialization(conn, wiki_corps, dry_run)
|
||||
console.event(
|
||||
f"{corp_spec['specialized']}/{corp_spec['total']} corps specialized, "
|
||||
f"{corp_spec['hq_resolved']} headquarters_body resolved (recomputed every run; "
|
||||
f"{corp_spec['hq_overridden']} authored overrides)"
|
||||
)
|
||||
|
||||
step("Importing corp presence...")
|
||||
commodity_ids = {r[0] for r in conn.execute("SELECT commodity_id FROM commodities").fetchall()}
|
||||
n_presence = corporations.import_corp_presence(conn, wiki_corps, commodity_ids, dry_run)
|
||||
console.event(f"{n_presence} corp_presence rows")
|
||||
|
||||
# D-189, #827.
|
||||
step("Importing brand products and inputs...")
|
||||
n_brands, n_brand_inputs = brands.import_brands(conn, dry_run)
|
||||
console.event(f"{n_brands} brand_products, {n_brand_inputs} brand_inputs")
|
||||
|
||||
# D-189 §6.
|
||||
step("Populating system_fiscal...")
|
||||
n_fiscal = economy.import_system_fiscal(conn, dry_run)
|
||||
console.event(f"{n_fiscal} system_fiscal rows")
|
||||
|
||||
# D-204, #910.
|
||||
step("Populating body_radius_km fallback...")
|
||||
console.event(f"{bodies.populate_body_radius_km(conn, dry_run)} bodies updated")
|
||||
|
||||
# D-207, #908.
|
||||
step("Populating atlas_city_names from wiki content...")
|
||||
console.event(f"{atlas.populate_atlas_city_names(conn, dry_run)} city name rows")
|
||||
|
||||
# D-223, T-1169 — mirrors the city pool's shape over a different
|
||||
# names-only markers.json key set; independent of the settlement pool.
|
||||
step("Populating atlas_feature_names from wiki content...")
|
||||
console.event(f"{atlas.populate_atlas_feature_names(conn, dry_run)} feature name rows")
|
||||
|
||||
# D-242 Phase B (T-1074). SUPERSEDES the retired corp-HQ cross-reference
|
||||
# (D-207, #909), which inserted one city row per corp HQ with no
|
||||
# UNIQUE(body_id, name) — ten co-named Groombridge rows on GJ380c. Must
|
||||
# follow the city pool and the corp-specialization step.
|
||||
step("Emitting Standalone-HQ settlements + CityTenant links (D-242)...")
|
||||
hq = corporations.populate_standalone_hq_settlements(conn, dry_run)
|
||||
console.event(
|
||||
f"{hq['standalone_inserted']} Standalone-HQ settlements, "
|
||||
f"{hq['tenant_linked']} CityTenant links ({hq['tenant_unmatched']} unmatched)"
|
||||
)
|
||||
|
||||
# D-242, T-1075 — over the CORRECTED pool, so Standalone-HQ settlements
|
||||
# are part of the rank-size spread rather than bolted on after.
|
||||
step("Baking settlement population + settlement_class (D-242)...")
|
||||
pop = atlas.populate_settlement_population_class(conn, dry_run)
|
||||
console.event(
|
||||
f"{pop['cities_populated']} cities populated across {pop['bodies_spread']} bodies, "
|
||||
f"{pop['name_locked_applied']} NameLocked pins applied"
|
||||
)
|
||||
|
||||
# D-232, #993 — catalog first, then sparse per-body hero bias (FK).
|
||||
step("Baking trait_templates catalog (D-232)...")
|
||||
console.event(f"{traits.populate_trait_templates(conn, dry_run)} trait templates")
|
||||
step("Baking atlas_body_trait_bias hero pins (D-232)...")
|
||||
console.event(f"{traits.populate_atlas_body_trait_bias(conn, dry_run)} body trait-bias rows")
|
||||
|
||||
# T-1024, D-239 §2.
|
||||
step("Populating axial_tilt_deg from body-def frontmatter...")
|
||||
console.event(f"{bodies.populate_axial_tilt_deg(conn, dry_run)} bodies updated with axial_tilt_deg")
|
||||
|
||||
# D-247, T-1085 — frontmatter override + two-gate default.
|
||||
step("Populating biosphere_class (D-247)...")
|
||||
console.event(f"{bodies.populate_biosphere_class(conn, dry_run)} bodies updated with biosphere_class")
|
||||
|
||||
# T-988, D-235 — both follow trait_templates (V-TT-06 / V-TT-07 validate
|
||||
# against it).
|
||||
step("Baking architecture_zone_bias table (D-235)...")
|
||||
console.event(f"{traits.populate_architecture_zone_bias(conn, dry_run)} zone-bias rows")
|
||||
step("Baking color_register_bands table (D-235)...")
|
||||
console.event(f"{traits.populate_color_register_bands(conn, dry_run)} color register bands")
|
||||
|
||||
# Structural integrity (FK, chain refs, chain completeness). These mean
|
||||
# the imported data is broken — do NOT commit.
|
||||
step("Validating structural integrity...")
|
||||
struct_errors = validators.validate(conn) + brands.validate_brands(conn)
|
||||
if struct_errors:
|
||||
console.event(f"STRUCTURAL ERRORS ({len(struct_errors)}) — rolling back:", level="error")
|
||||
for e in struct_errors:
|
||||
console.event(e, level="error")
|
||||
raise ImportAborted()
|
||||
console.event("FK integrity, chain completeness, and brand layer (V-B01–V-B06) OK")
|
||||
|
||||
# Commit before the coverage check: coverage is a Phase 2 gate (D-175),
|
||||
# and the data should be queryable so the gaps can be reported clearly.
|
||||
if not dry_run:
|
||||
conn.commit()
|
||||
console.event("Data committed.")
|
||||
else:
|
||||
console.event("Dry run — no changes written.")
|
||||
|
||||
if step.n != STEPS:
|
||||
# A step added without bumping STEPS would make every progress
|
||||
# fraction wrong without failing anything. Cheap to refuse here.
|
||||
raise AssertionError(f"STEPS is {STEPS} but the import ran {step.n} steps")
|
||||
|
||||
return {
|
||||
"links": n_links,
|
||||
"commodities": n_commodities,
|
||||
"chains": n_chains,
|
||||
"inputs": n_inputs,
|
||||
"presence": n_presence,
|
||||
"brands": n_brands,
|
||||
"brand_inputs": n_brand_inputs,
|
||||
"fiscal": n_fiscal,
|
||||
}
|
||||
|
||||
|
||||
def _coverage(conn: sqlite3.Connection, wiki_corps) -> list[str]:
|
||||
"""The D-175 Phase 2 gate. Runs after the commit, so the data is usable."""
|
||||
console.event("Validating coverage (D-175 Phase 2 gate)...")
|
||||
commodity_ids = {r[0] for r in conn.execute("SELECT commodity_id FROM commodities").fetchall()}
|
||||
errors = validators.validate_commodity_coverage(conn, wiki_corps, commodity_ids)
|
||||
errors += validators.validate_system_coverage(conn, wiki_corps)
|
||||
if not errors:
|
||||
console.event("All coverage thresholds met — Phase 2 gate PASSED.")
|
||||
return errors
|
||||
@@ -1,377 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Import economics data into systems.db.
|
||||
|
||||
Reads TOML/JSON source files and populates the economics tables:
|
||||
- gate_links from docs/design/star-map.json (335 edges, bidirectional)
|
||||
- commodities from wiki/economics/commodities.toml (36 types)
|
||||
- production_chains + chain_inputs from wiki/economics/production_chains.toml
|
||||
- currency_zone on star_systems (default TRACTUS_PRIMARY)
|
||||
- gate_energy_connected on star_systems (D-186: false for MARK_PRIMARY zones)
|
||||
- corporations from wiki/corporations/*.md (sync + insert new records)
|
||||
- corp_specialization/hq_placement/headquarters_body/headquarters_city_id
|
||||
on corporations (D-242: HQ-placement key + baked
|
||||
CityTenant link, from corp_hq_placement.toml)
|
||||
- corp_presence from wiki/corporations/*.md (headquarters location data)
|
||||
- atlas_city_names Standalone-HQ settlement rows (D-242, T-1074)
|
||||
|
||||
Validation (hard errors, non-zero exit on any failure):
|
||||
- Wiki corporation names must match DB proper_name records (D-182 sync constraint)
|
||||
- Chain completeness: every intermediate commodity has at least one production chain
|
||||
- Commodity coverage: 3+ corporations per major commodity type (D-175)
|
||||
- System coverage: 1+ corporation per inhabited system with population > 100K (D-175)
|
||||
|
||||
This file is the CLI entrypoint and single-transaction orchestrator; the import
|
||||
steps live in the economy_import package (T-1067). The stamped source set is
|
||||
defined in tooling/generator_sources.py.
|
||||
|
||||
Usage:
|
||||
python3 tooling/economy-db/import_economics.py
|
||||
python3 tooling/economy-db/import_economics.py --dry-run
|
||||
python3 tooling/economy-db/import_economics.py --db path/to/systems.db
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# tooling/economy-db is not a package (hyphenated dir) — put it on sys.path so
|
||||
# the economy_import package resolves; its __init__ adds tooling/ for
|
||||
# generator_sources and schema_version.
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
|
||||
from economy_import import ( # noqa: E402
|
||||
atlas,
|
||||
bodies,
|
||||
brands,
|
||||
corporations,
|
||||
db,
|
||||
economy,
|
||||
migration,
|
||||
specialization,
|
||||
stamp,
|
||||
traits,
|
||||
validators,
|
||||
)
|
||||
from economy_import.errors import ImportAborted # noqa: E402
|
||||
from economy_import.paths import DB_PATH # noqa: E402
|
||||
from generator_sources import IMPORT_ECONOMICS_SOURCES # noqa: E402
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Import economics data into systems.db")
|
||||
parser.add_argument("--db", default=str(DB_PATH), help="Path to systems.db")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Validate without writing")
|
||||
parser.add_argument(
|
||||
"--strict-specialization", action="store_true",
|
||||
help="Treat D-237 completeness gates (V-SES-02, V-FAC-01) as hard errors. "
|
||||
"Off by default until the #1014 fallback and #1016 content pass land.",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
db_path = Path(args.db)
|
||||
if not db_path.exists():
|
||||
print(f"error: {db_path} not found", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
print("\n Economics Import Pipeline")
|
||||
print(f" DB: {db_path}")
|
||||
if args.dry_run:
|
||||
print(" Mode: DRY RUN")
|
||||
print()
|
||||
|
||||
# Load wiki corps before opening DB — allows early exit on parse failures
|
||||
print(" Loading wiki corporations...")
|
||||
wiki_corps = corporations.load_wiki_corps()
|
||||
print(f" {len(wiki_corps)} corporation files parsed")
|
||||
|
||||
# Regenerate generated_brands.toml via the Rust binary before the Python
|
||||
# import reads it. Single pipeline, single stamp — resolves review T2/H3
|
||||
# ("on-behalf stamping" coupling) by folding brand generation into
|
||||
# import_economics' flow rather than having the caller (Makefile / user)
|
||||
# remember to run it first. Skipped on --dry-run to avoid a disk
|
||||
# side-effect during validation.
|
||||
if not args.dry_run:
|
||||
try:
|
||||
brands.regenerate_brands()
|
||||
except ImportAborted:
|
||||
sys.exit(1)
|
||||
|
||||
conn = db.connect(db_path)
|
||||
|
||||
# The clear-then-reimport cycle below runs as a single explicit
|
||||
# transaction. Any crash, validation error, or KeyboardInterrupt
|
||||
# between the first DELETE and the final commit rolls everything
|
||||
# back — the DB never ends up half-cleared with stale rows in some
|
||||
# tables and empty rows in others. On success we commit exactly
|
||||
# once, immediately after structural validation passes.
|
||||
conn.execute("BEGIN")
|
||||
try:
|
||||
# 1. Migrate schema (idempotent, inside the tx so a crash here
|
||||
# leaves no half-applied ALTER TABLE.)
|
||||
print(" [1/10] Schema migration...")
|
||||
migration.apply_schema_migrations(conn)
|
||||
# Atlas index tables: apply canonical DDL + empty geometry (D-223, #951)
|
||||
atlas.ensure_atlas_index_schema(conn, args.dry_run)
|
||||
print(" tables and columns ready")
|
||||
|
||||
# Clear economics tables in FK-safe order (children before parents)
|
||||
if not args.dry_run:
|
||||
db.clear_economics_tables(conn)
|
||||
|
||||
# 2. Gate links
|
||||
print(" [2/10] Importing gate links...")
|
||||
n_links = economy.import_gate_links(conn, args.dry_run)
|
||||
print(f" {n_links} rows (bidirectional)")
|
||||
|
||||
# 3. Commodities
|
||||
print(" [3/10] Importing commodities...")
|
||||
n_commodities = economy.import_commodities(conn, args.dry_run)
|
||||
print(f" {n_commodities} commodities")
|
||||
|
||||
# 4. Production chains
|
||||
print(" [4/10] Importing production chains...")
|
||||
n_chains, n_inputs = economy.import_chains(conn, args.dry_run)
|
||||
print(f" {n_chains} chains, {n_inputs} inputs")
|
||||
|
||||
# 4b. System specialization (D-237 authored layer) — after commodities
|
||||
# (FK) and before currency zones. UPSERTs onto pre-existing
|
||||
# system_economy / system_factions rows; unauthored systems stay NULL.
|
||||
print(" [4b/10] Importing system specialization (D-237)...")
|
||||
spec = specialization.import_system_specialization(
|
||||
conn, args.dry_run, strict=args.strict_specialization
|
||||
)
|
||||
print(
|
||||
f" vocab {spec['vocab']} | economic {spec['economic']} | "
|
||||
f"cultural {spec['cultural']} | faction {spec['faction']}"
|
||||
)
|
||||
if spec["missing_economy_row"]:
|
||||
print(
|
||||
f" WARNING: {len(spec['missing_economy_row'])} authored "
|
||||
f"system(s) lack a system_economy row (values dropped): "
|
||||
f"{spec['missing_economy_row']}"
|
||||
)
|
||||
if spec["missing_faction_row"]:
|
||||
print(
|
||||
f" WARNING: {len(spec['missing_faction_row'])} authored "
|
||||
f"system(s) lack a system_factions row (faction dropped): "
|
||||
f"{spec['missing_faction_row']}"
|
||||
)
|
||||
|
||||
# 5. Currency zones
|
||||
print(" [5/10] Setting currency zones...")
|
||||
zones = economy.set_currency_zones(conn, args.dry_run)
|
||||
for zone, count in sorted(zones.items()):
|
||||
print(f" {zone}: {count}")
|
||||
|
||||
# 6. Gate energy connectivity (D-186) — must run after currency zones
|
||||
print(" [6/10] Setting gate energy connectivity...")
|
||||
energy = economy.set_gate_energy(conn, args.dry_run)
|
||||
for label, count in sorted(energy.items()):
|
||||
print(f" {label}: {count}")
|
||||
|
||||
# 7. Sync corporations from wiki (D-182: hard error on name divergence)
|
||||
print(" [7/10] Syncing corporations...")
|
||||
corp_errors = corporations.sync_corporations(conn, wiki_corps, args.dry_run)
|
||||
if corp_errors:
|
||||
print(f" ERRORS ({len(corp_errors)}) — name divergence detected (D-182):")
|
||||
for e in corp_errors:
|
||||
print(f" - {e}")
|
||||
print(" Fix: update wiki title or DB proper_name to match, then re-run.")
|
||||
raise ImportAborted()
|
||||
n_db_corps = conn.execute("SELECT COUNT(*) FROM corporations").fetchone()[0]
|
||||
print(f" {n_db_corps} corporations in DB ({len(wiki_corps)} from wiki)")
|
||||
|
||||
# 7b. Corp specialization + HQ placement (D-242, T-1074) — Phase A.
|
||||
# Reset + derive on every run (PR #177 H1/T2): the importer-owned
|
||||
# corporations columns are NULLed and re-derived from source, never
|
||||
# kept from a prior run. Must run after corp sync (rows must exist),
|
||||
# before corp_presence (which reads headquarters_body back off
|
||||
# corporations), and before step 12 (the H2 city-presence tiebreak
|
||||
# reads atlas_city_names BEFORE this run's clear/rebuild — the
|
||||
# previous run's settled state, by design).
|
||||
print(" [7b/10] Importing corp specialization + HQ placement (D-242)...")
|
||||
corp_spec = corporations.populate_corp_specialization(conn, wiki_corps, args.dry_run)
|
||||
print(
|
||||
f" {corp_spec['specialized']}/{corp_spec['total']} corps specialized, "
|
||||
f"{corp_spec['hq_resolved']} headquarters_body resolved (recomputed every run; "
|
||||
f"{corp_spec['hq_overridden']} authored overrides)"
|
||||
)
|
||||
|
||||
# 8. Corp presence from wiki headquarters data
|
||||
print(" [8/10] Importing corp presence...")
|
||||
commodity_ids = {
|
||||
r[0] for r in conn.execute("SELECT commodity_id FROM commodities").fetchall()
|
||||
}
|
||||
n_presence = corporations.import_corp_presence(
|
||||
conn, wiki_corps, commodity_ids, args.dry_run
|
||||
)
|
||||
print(f" {n_presence} corp_presence rows")
|
||||
|
||||
# 9. Brand products and inputs (D-189, #827)
|
||||
print(" [9/10] Importing brand products and inputs...")
|
||||
n_brands, n_brand_inputs = brands.import_brands(conn, args.dry_run)
|
||||
print(f" {n_brands} brand_products, {n_brand_inputs} brand_inputs")
|
||||
|
||||
# 10. System fiscal parameters (D-189 section 6)
|
||||
print(" [10/13] Populating system_fiscal...")
|
||||
n_fiscal = economy.import_system_fiscal(conn, args.dry_run)
|
||||
print(f" {n_fiscal} system_fiscal rows")
|
||||
|
||||
# 11. body_radius_km fallback from planet_class (D-204, #910)
|
||||
print(" [11/13] Populating body_radius_km fallback...")
|
||||
n_radius = bodies.populate_body_radius_km(conn, args.dry_run)
|
||||
print(f" {n_radius} bodies updated")
|
||||
|
||||
# 12. atlas_city_names from wiki markers.json (D-207, #908)
|
||||
print(" [12/13] Populating atlas_city_names from wiki content...")
|
||||
n_cities = atlas.populate_atlas_city_names(conn, args.dry_run)
|
||||
print(f" {n_cities} city name rows")
|
||||
|
||||
# 12b. atlas_feature_names (rivers/mountains) from wiki markers.json
|
||||
# (D-223, T-1169) — mirrors step 12's pool-load shape exactly, over a
|
||||
# different names-only markers.json key set. Independent of the
|
||||
# settlement pool, so order relative to step 13 doesn't matter; placed
|
||||
# here to stay adjacent to its sibling pool-load step.
|
||||
print(" [12b/13] Populating atlas_feature_names from wiki content...")
|
||||
n_features = atlas.populate_atlas_feature_names(conn, args.dry_run)
|
||||
print(f" {n_features} feature name rows")
|
||||
|
||||
# 13. Standalone-HQ settlements + CityTenant city links (D-242, T-1074) — Phase B.
|
||||
# SUPERSEDES the retired corp-HQ cross-reference (D-207, #909) — that
|
||||
# step inserted one atlas_city_names row per corp HQ with no
|
||||
# UNIQUE(body_id, name), producing duplicate co-named "cities" (10
|
||||
# Groombridge rows on GJ380c). Must run after populate_atlas_city_names
|
||||
# (the city pool Standalone HQs join and CityTenant HQs tenant) and
|
||||
# after step 7b (corp_specialization/hq_placement/headquarters_body).
|
||||
print(" [13/13] Emitting Standalone-HQ settlements + CityTenant links (D-242)...")
|
||||
hq_settlements = corporations.populate_standalone_hq_settlements(conn, args.dry_run)
|
||||
print(
|
||||
f" {hq_settlements['standalone_inserted']} Standalone-HQ settlements, "
|
||||
f"{hq_settlements['tenant_linked']} CityTenant links "
|
||||
f"({hq_settlements['tenant_unmatched']} unmatched)"
|
||||
)
|
||||
|
||||
# 13b. Per-settlement population + settlement_class bake (D-242, T-1075).
|
||||
# Runs over the CORRECTED pool — after both T-1074 steps, so
|
||||
# Standalone-HQ settlements are included in the rank-size spread, not
|
||||
# bolted on after.
|
||||
print(" [13b/13] Baking settlement population + settlement_class (D-242)...")
|
||||
pop_bake = atlas.populate_settlement_population_class(conn, args.dry_run)
|
||||
print(
|
||||
f" {pop_bake['cities_populated']} cities populated across "
|
||||
f"{pop_bake['bodies_spread']} bodies, {pop_bake['name_locked_applied']} "
|
||||
f"NameLocked pins applied"
|
||||
)
|
||||
|
||||
# 14. Architecture-flavor trait templates (D-232, #993). Catalog first,
|
||||
# then sparse per-body hero bias (FK -> trait_templates + bodies).
|
||||
print(" [14/15] Baking trait_templates catalog (D-232)...")
|
||||
n_templates = traits.populate_trait_templates(conn, args.dry_run)
|
||||
print(f" {n_templates} trait templates")
|
||||
print(" [15/15] Baking atlas_body_trait_bias hero pins (D-232)...")
|
||||
n_bias = traits.populate_atlas_body_trait_bias(conn, args.dry_run)
|
||||
print(f" {n_bias} body trait-bias rows")
|
||||
|
||||
# 16. axial_tilt_deg from body-def frontmatter (T-1024, D-239 §2)
|
||||
print(" [16/17] Populating axial_tilt_deg from body-def frontmatter...")
|
||||
n_tilt = bodies.populate_axial_tilt_deg(conn, args.dry_run)
|
||||
print(f" {n_tilt} bodies updated with axial_tilt_deg")
|
||||
|
||||
# 17. biosphere_class from body frontmatter override + two-gate default (D-247, T-1085)
|
||||
print(" [17/19] Populating biosphere_class (D-247)...")
|
||||
n_bio = bodies.populate_biosphere_class(conn, args.dry_run)
|
||||
print(f" {n_bio} bodies updated with biosphere_class")
|
||||
|
||||
# 18. Architecture zone-type bias table (T-988, D-235 step 2) — must
|
||||
# follow trait_templates (V-TT-06 validates against its visual_bundle).
|
||||
print(" [18/19] Baking architecture_zone_bias table (D-235)...")
|
||||
n_zone_bias = traits.populate_architecture_zone_bias(conn, args.dry_run)
|
||||
print(f" {n_zone_bias} zone-bias rows")
|
||||
|
||||
# 19. Color register bands (T-988, D-235) — numeric HSV sampling bands
|
||||
# per trait-template color_register; also follows trait_templates (V-TT-07).
|
||||
print(" [19/19] Baking color_register_bands table (D-235)...")
|
||||
n_color_bands = traits.populate_color_register_bands(conn, args.dry_run)
|
||||
print(f" {n_color_bands} color register bands")
|
||||
|
||||
# Validate structural integrity (FK, chain refs, chain completeness).
|
||||
# These errors indicate broken imported data — do NOT commit.
|
||||
print("\n Validating structural integrity...")
|
||||
struct_errors = validators.validate(conn)
|
||||
struct_errors.extend(brands.validate_brands(conn))
|
||||
if struct_errors:
|
||||
print(f" STRUCTURAL ERRORS ({len(struct_errors)}) — rolling back:")
|
||||
for e in struct_errors:
|
||||
print(f" - {e}")
|
||||
raise ImportAborted()
|
||||
print(" FK integrity, chain completeness, and brand layer (V-B01–V-B06) OK")
|
||||
|
||||
# Commit all imported data (corps, presence, etc.) before coverage check.
|
||||
# Coverage validation is a Phase 2 gate (D-175) — data should be persisted
|
||||
# so tools can query it and report gaps clearly.
|
||||
if not args.dry_run:
|
||||
conn.commit()
|
||||
print(" Data committed.")
|
||||
else:
|
||||
# Dry-run: leave the transaction open so the coverage check below
|
||||
# can still SELECT against the in-memory imported data. The
|
||||
# transaction is discarded when conn.close() runs on exit.
|
||||
print(" Dry run — no changes written.")
|
||||
except ImportAborted:
|
||||
conn.rollback()
|
||||
conn.close()
|
||||
sys.exit(1)
|
||||
except BaseException:
|
||||
# Any other exception (KeyboardInterrupt, MemoryError, DB error,
|
||||
# programmer error) triggers a rollback so the DB is never left in
|
||||
# a half-imported state. Re-raise so the user sees the traceback.
|
||||
conn.rollback()
|
||||
conn.close()
|
||||
raise
|
||||
|
||||
# Stamp generator metadata (#855, #856): record source SHAs so the
|
||||
# pre-push hook can detect stale DB snapshots. Written BEFORE the
|
||||
# coverage gate — the stamp records generator execution (code version),
|
||||
# not data completeness. Coverage gaps (#860) are pre-existing data
|
||||
# issues and must not prevent the stamp from landing.
|
||||
if not args.dry_run:
|
||||
try:
|
||||
stamp.write_stamp(conn, "import_economics", *IMPORT_ECONOMICS_SOURCES)
|
||||
conn.commit()
|
||||
print(" Stamped: import_economics (covers brand pipeline Rust sources)")
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(f" WARNING: failed to write generator stamp: {exc}", file=sys.stderr)
|
||||
|
||||
# Validate coverage (hard errors per D-175, but after commit so data is usable).
|
||||
print("\n Validating coverage (D-175 Phase 2 gate)...")
|
||||
coverage_errors: list[str] = []
|
||||
commodity_ids_for_coverage = {
|
||||
r[0] for r in conn.execute("SELECT commodity_id FROM commodities").fetchall()
|
||||
}
|
||||
coverage_errors.extend(
|
||||
validators.validate_commodity_coverage(conn, wiki_corps, commodity_ids_for_coverage)
|
||||
)
|
||||
coverage_errors.extend(validators.validate_system_coverage(conn, wiki_corps))
|
||||
|
||||
if coverage_errors:
|
||||
print(f" COVERAGE ERRORS ({len(coverage_errors)}) — Phase 2 gate not met:")
|
||||
for e in coverage_errors:
|
||||
print(f" - {e}")
|
||||
print("\n Data committed but Phase 2 gate is NOT met. "
|
||||
"Add corporations to meet coverage thresholds and re-run.")
|
||||
conn.close()
|
||||
sys.exit(2) # exit 2 = coverage warning (data+stamp committed); exit 1 = real error
|
||||
else:
|
||||
print(" All coverage thresholds met — Phase 2 gate PASSED.")
|
||||
|
||||
conn.close()
|
||||
|
||||
print(f"\n Done: {n_links} gate_links, {n_commodities} commodities, "
|
||||
f"{n_chains} chains, {n_inputs} inputs, {n_presence} corp_presence, "
|
||||
f"{n_brands} brand_products, {n_brand_inputs} brand_inputs, "
|
||||
f"{n_fiscal} system_fiscal\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -5,20 +5,21 @@ generator_sources — single source of truth for systems.db generator source set
|
||||
Each generator that writes a stamp row to systems.db's ``meta`` table records the
|
||||
SHA-1 of the concatenated bytes of its source files (sorted by path). This module
|
||||
is the ONE place that defines which files belong to each generator's stamped set
|
||||
(T-1067). It replaces the formerly triplicated lists in:
|
||||
|
||||
- ``IMPORT_ECONOMICS_SOURCES`` in tooling/economy-db/import_economics.py
|
||||
- ``GENERATOR_SOURCES`` in tooling/check-systems-db-stamp
|
||||
- the hardcoded watch list in .claude/skills/pr-process/SKILL.md
|
||||
(now derived via ``python3 tooling/generator_sources.py --list``)
|
||||
(T-1067). It replaced three formerly separate lists (in the importer, in the
|
||||
old check-systems-db-stamp script, and in the pr-process skill).
|
||||
|
||||
Consumers:
|
||||
- tooling/economy-db (the importer) — imports ``IMPORT_ECONOMICS_SOURCES`` and
|
||||
``file_sha1`` to write the stamp after a successful run.
|
||||
- tooling/check-systems-db-stamp — imports ``GENERATOR_SOURCES`` and
|
||||
- the ledger importer (tooling/domains/ledger/) — imports
|
||||
``IMPORT_ECONOMICS_SOURCES`` and ``file_sha1`` to write the stamp after a
|
||||
successful run.
|
||||
- ``reach check systems-db-stamp`` — imports ``GENERATOR_SOURCES`` and
|
||||
``file_sha1`` to verify the stamp before a push.
|
||||
- ``reach pr watchlist-diff`` — reads ``GENERATOR_SOURCES`` for its watch set.
|
||||
- /pr-process (skill) — shells out to the ``--list`` CLI for its watch list.
|
||||
|
||||
Stays a module rather than a verb (tooling/DOMAINS.md): it is a list of paths
|
||||
read by gates, and nothing types it.
|
||||
|
||||
This file is itself a member of the stamped source set: a registry change (adding
|
||||
or removing a source) must flip the stamp and force ``make regen-db``, otherwise
|
||||
edits to the list would be invisible to staleness detection.
|
||||
@@ -44,10 +45,10 @@ REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
|
||||
# Entrypoint + split module package (T-1067).
|
||||
IMPORT_ECONOMICS_ENTRYPOINT: Path = (
|
||||
REPO_ROOT / "tooling" / "economy-db" / "import_economics.py"
|
||||
REPO_ROOT / "tooling" / "domains" / "ledger" / "service.py"
|
||||
)
|
||||
ECONOMY_IMPORT_PACKAGE_DIR: Path = (
|
||||
REPO_ROOT / "tooling" / "economy-db" / "economy_import"
|
||||
REPO_ROOT / "tooling" / "domains" / "ledger" / "economy_import"
|
||||
)
|
||||
|
||||
# Rust sources for the generate_brands subroutine. import_economics runs the
|
||||
@@ -163,7 +164,7 @@ def _corporation_pages() -> tuple[Path, ...]:
|
||||
|
||||
# Canonical source set for import_economics' meta stamp. Covers the Python
|
||||
# entrypoint and its module package, the Rust binary it invokes (generate_brands
|
||||
# main.rs + names.rs + surname corpus + wrapper script), the shared schema
|
||||
# main.rs + names.rs + surname corpus + core/process.py), the shared schema
|
||||
# version constant, the authored data TOMLs, the corp wiki pages whose
|
||||
# frontmatter the importer bakes (T1, PR #177), and this registry itself.
|
||||
IMPORT_ECONOMICS_SOURCES: tuple[Path, ...] = (
|
||||
@@ -175,7 +176,7 @@ IMPORT_ECONOMICS_SOURCES: tuple[Path, ...] = (
|
||||
GENERATE_BRANDS_NAMES_RS,
|
||||
GENERATE_BRANDS_SURNAMES_RS,
|
||||
REPO_ROOT / "tooling" / "core" / "process.py", # runs the brand binary (T-1286)
|
||||
REPO_ROOT / "tooling" / "schema_version.py",
|
||||
REPO_ROOT / "tooling" / "domains" / "ledger" / "schema_version.py",
|
||||
SPECIALIZATION_VOCAB_TOML,
|
||||
SYSTEM_SPECIALIZATION_TOML,
|
||||
CORP_HQ_PLACEMENT_TOML,
|
||||
@@ -198,7 +199,7 @@ IMPORT_ECONOMICS_SOURCES: tuple[Path, ...] = (
|
||||
# The atlas geometry generator (generate_atlas) was retired in #951 (D-223).
|
||||
# import_economics is now the sole regen-db generator that writes systems.db; it
|
||||
# owns the atlas index (names-only pool + empty geometry tables). The surviving
|
||||
# planet-gen importers (import_heightmaps, import_province_boundaries) are
|
||||
# `reach atlas planet` importers (import-heightmaps, import-provinces) are
|
||||
# one-time build imports baked into the committed DB, not part of regen-db, so
|
||||
# they are intentionally not stamped here.
|
||||
GENERATOR_SOURCES: dict[str, tuple[Path, ...]] = {
|
||||
|
||||
@@ -59,6 +59,10 @@ DOMAINS: dict[str, tuple[str, str]] = {
|
||||
"tooling.domains.atlas.router:app",
|
||||
"The spatial ladder — authoring, inspection and the DB",
|
||||
),
|
||||
"ledger": (
|
||||
"tooling.domains.ledger.router:app",
|
||||
"The economics pipeline — the import that builds systems.db",
|
||||
),
|
||||
"godot": (
|
||||
"tooling.domains.godot.router:app",
|
||||
"Does the client parse, and does it parse cold",
|
||||
|
||||
+4
-5
@@ -17,14 +17,14 @@ would drift from the real FK web this populator depends on). Wiki content
|
||||
(`wiki/star-systems/*/bodies/*/markers.json`) is read directly from the repo
|
||||
— read-only input, safe to reuse as-is; only the DB connection is scratch.
|
||||
Deliberately calls the two populator functions directly rather than the full
|
||||
`import_economics.py` CLI: the CLI's first step shells out to the Rust
|
||||
`reach ledger import`: its first step shells out to the Rust
|
||||
`generate_brands` binary and overwrites `generated_brands.toml` on disk
|
||||
(`brands.regenerate_brands`), a real side effect on shared repo content that
|
||||
has nothing to do with atlas name-pool idempotency and would make this test
|
||||
depend on a Rust build.
|
||||
|
||||
Stdlib only (unittest) — run directly or via `make test-tooling`:
|
||||
python3 tooling/economy-db/test_atlas_idempotency.py
|
||||
.venv/bin/python tooling/test_ledger_atlas_idempotency.py
|
||||
"""
|
||||
|
||||
import shutil
|
||||
@@ -34,11 +34,10 @@ import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||
|
||||
from economy_import import atlas # noqa: E402
|
||||
from economy_import.paths import DB_PATH # noqa: E402
|
||||
from tooling.domains.ledger.economy_import import atlas # noqa: E402
|
||||
from tooling.domains.ledger.economy_import.paths import DB_PATH # noqa: E402
|
||||
|
||||
|
||||
class AtlasNamePoolIdempotencyTests(unittest.TestCase):
|
||||
@@ -9,7 +9,7 @@ shape/coverage checks (T-988) — the `make test-tooling` dry-run only
|
||||
exercises the happy path against the committed, already-valid content.
|
||||
|
||||
Stdlib only (unittest) — run directly or via `make test-tooling`:
|
||||
python3 tooling/economy-db/test_traits.py
|
||||
.venv/bin/python tooling/test_ledger_traits.py
|
||||
"""
|
||||
|
||||
import json
|
||||
@@ -19,11 +19,10 @@ import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||
|
||||
from economy_import import traits # noqa: E402
|
||||
from economy_import.errors import ImportAborted # noqa: E402
|
||||
from tooling.domains.ledger.economy_import import traits # noqa: E402
|
||||
from tooling.domains.ledger.economy_import.errors import ImportAborted # noqa: E402
|
||||
|
||||
|
||||
def write_registry(tmp: Path, body: str) -> Path:
|
||||
@@ -247,7 +246,7 @@ class CatalogCrossCheckTests(unittest.TestCase):
|
||||
catalog.write_text(catalog_toml, encoding="utf-8")
|
||||
traits.ARCHITECTURE_TRAIT_CATALOG_TOML = catalog
|
||||
out = io.StringIO()
|
||||
with contextlib.redirect_stdout(out):
|
||||
with contextlib.redirect_stderr(out): # console events go to stderr (T-1289)
|
||||
with self.assertRaises(ImportAborted):
|
||||
traits.populate_trait_templates(self.conn, dry_run=True)
|
||||
return out.getvalue()
|
||||
@@ -365,7 +364,7 @@ class ZoneBiasValidationTests(unittest.TestCase):
|
||||
import io
|
||||
|
||||
out = io.StringIO()
|
||||
with contextlib.redirect_stdout(out):
|
||||
with contextlib.redirect_stderr(out): # console events go to stderr (T-1289)
|
||||
with self.assertRaises(ImportAborted):
|
||||
self.populate(
|
||||
'[bias.test_template.industrial_freight]\n'
|
||||
@@ -397,7 +396,7 @@ class ZoneBiasValidationTests(unittest.TestCase):
|
||||
|
||||
for bad in ("wall = 15000\n", 'wall = ["brick_wall"]\n'):
|
||||
out = io.StringIO()
|
||||
with contextlib.redirect_stdout(out):
|
||||
with contextlib.redirect_stderr(out): # console events go to stderr (T-1289)
|
||||
with self.assertRaises(ImportAborted):
|
||||
self.populate(f"[bias.test_template.commercial_market]\n{bad}")
|
||||
self.assertIn("V-TT-06", out.getvalue())
|
||||
@@ -461,7 +460,7 @@ class ColorRegisterBandTests(unittest.TestCase):
|
||||
import io
|
||||
|
||||
out = io.StringIO()
|
||||
with contextlib.redirect_stdout(out):
|
||||
with contextlib.redirect_stderr(out): # console events go to stderr (T-1289)
|
||||
with self.assertRaises(ImportAborted):
|
||||
# A band for a DIFFERENT register — neutral_grey (referenced by
|
||||
# test_template above) has no band at all.
|
||||
Reference in New Issue
Block a user