import_economics.py 2,620 → 309 lines — a thin orchestrator keeping the
exact CLI, single-transaction/rollback contract, and exit codes. The 16
import steps, MIGRATION_SQL, brands shell-out, validators, and stamp
write now live in tooling/economy-db/economy_import/ (db, migration,
economy, corporations, brands, bodies, atlas, specialization, traits,
validators, stamp, paths, errors). Full type hints throughout.
tooling/generator_sources.py replaces the triplicated source registry
(importer / stamp checker / pr-process watch list — the skill now derives
its list via --list). The registry stamps itself, and economy_import/
modules are globbed fail-closed, so a future module is stamped the moment
it exists — closing the silently-weakened-stamp failure mode.
Rider: connector config helpers centralized in tooling/db/common.py.
Byte-identical behavior proven: full-import table dump diff EMPTY over
107,843 lines / 37 tables (volatile timestamp fields excluded); dry-run
output parity; generated_brands.toml sha unchanged. make test-tooling
PASS; ruff clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- workers/pool.rs: 5 new tests; catch_unwind keeps worker threads alive on
handler panic (in-flight request loss unchanged, pinned by test + #843
docs); stubs.rs no longer falsely claims the pool is tested
- save/load: execute_save_load pinned .after(Storyteller) so the scheduler
cannot legally save pre-Input state; exclusive-system exception recorded
in tick_phases.rs rules
- surname corpus extracted to bin/shared/surname_corpus.rs (both economy
generators import it; byte-identical output verified on 23.6MB+1.45MB
TOMLs); all three stamp/watch registries updated
- generator_spike gated behind non-default 'generator-spike' feature
- economy.rs: 11 new D-181 signal-derivation tests on the new
econ_sim Simulation::from_economy in-memory constructor
- perception exemption comments now state the consumer sort contract;
unused bytemuck removed; rayon comment corrected; the 22 allow(dead_code)
documented as serde schema enforcement
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wires the D-237 authored specialization layer into import_economics.py:
- import_system_specialization(): reads specialization_vocabulary.toml
(FK-validated against commodities; repopulates specialization_vocabulary)
and system_specialization.toml (UPSERTs economic_specialization +
cultural_specialization onto system_economy, dominant_faction onto
system_factions for authored systems only). Hard errors on bad commodity
FK / projected enum / unknown system_id abort the transaction; prints a
coverage report + missing-row warnings.
- Called as step 4b in main() (after commodities so the FK resolves, before
currency zones). economic/cultural columns are importer-owned and cleared
to NULL first for idempotency; dominant_faction only overwritten for
authored systems (never globally cleared — shared with other derivation).
- specialization_vocabulary added to MIGRATION_SQL (after commodities for FK)
so the migration path on existing DBs creates the table, not just fresh
systems-schema.sql builds — this is what #1011 missed. Also cleared before
commodities in the FK-safe clear block.
- Both source TOMLs added to IMPORT_ECONOMICS_SOURCES and mirrored in
check-systems-db-stamp GENERATOR_SOURCES so editing them flips the stamp.
Validated: full non-dry-run import on a copy of the live systems.db exits 0;
27 vocab rows, 27/27/27 economic/cultural/faction set, no missing rows;
spot checks Ran=breadbasket/agrarian, Vuurkloof=independent,
financial_hub=NonPhysical/Specialist. Dry-run also exits 0.
V-SES CI guardrail suite remains #1015; code note flags that V-SES-03
(equal-or-higher) must not hard-fail HUB specializations.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The procedural server cascade (Phase 4) and the frozen names-only pool
supersede the Python atlas geometry generator and the LLM namer. Retire:
- generate_atlas.py (geometry production — cities/roads/rivers placement)
- gemma_naming.py, naming_core.py + tests (test_batch_naming,
test_register_selection, qa_naming) and run-atlas-naming.sh (the LLM
place-namer; its output is now the frozen pool)
- apply_name_fixes.py (name-field patches), fix_fewshot_bleed.py /
prune_atlas_features.py (geometry tools)
- import_city_names.py (redundant with import_economics name-pool path)
Pipeline updates: drop the generate_atlas step + atlas-generate /
test-atlas-determinism targets from the Makefile; remove generate_atlas
from the stamp registry (import_economics is the sole regen-db generator);
drop run-atlas-determinism from tests/run-all; refresh stale references in
schema_version, backfill_cultural_corridor, earth_blocklist (kept as
reference data), populate_terrain_reference, and heightmap.rs.
The Gemma prompting methodology is preserved in
docs/gemma-naming-methodology.md (separate commit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add gemma_naming.py and naming_core.py to generate_atlas source
tracking in check-systems-db-stamp. Mirror in pr-push watch list.
Prevents naming helper changes from bypassing stale-stamp detection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. decisions_sync.py: fix refs_created inflation (check rowcount),
remove dead IntegrityError except block
2. Extract SCHEMA_VERSION to shared tooling/schema_version.py —
both generators import from single source of truth
3. generate_atlas.py: narrow bare except to OperationalError +
"duplicate column" check
4. check-systems-db-stamp: add cross-generator schema_version
agreement assertion (defense-in-depth)
5. decision wrapper: add show + orphan-tickets to usage text
6. Add schema_version.py to all three source watch lists
(GENERATOR_SOURCES, IMPORT_ECONOMICS_SOURCES, generate_atlas
_write_stamp) — prevents silent staleness on version bump
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace SHA-1 hash in meta.schema_version with an orderable semver
string ("1.0.0"). SHA preserved in new schema_sha column for tamper
detection. Enables savegame migration lineage in Phase 5+ — saves can
record their schema version and determine which migrations to apply.
Updated both generators, check-systems-db-stamp validation (rejects
old SHA-hex values), schema DDL, and asset-pipeline docs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Addresses all blocking + minor items from PR #136 review.
Architectural change (T2/H3 — the review's main complaint):
generate_brands was previously a separate Rust binary that produced a TOML
artifact, with its stamp written "on behalf" by import_economics.py at the
end of its own run. Reviewers flagged the invisible coupling: two sources
of truth in a system designed to have one, and no way to tell from the
stamp that one "generator" was really a subroutine of the other.
import_economics now invokes tooling/generate-brands as the first step of
its main() flow, before opening its own DB connection. The TOML artefact
is still produced and still committed (useful for diff-review of brand
changes), but there's now one pipeline owner. The meta table carries two
rows (import_economics, generate_atlas) not three; the Rust binary's
source SHA folds into import_economics' stamp via IMPORT_ECONOMICS_SOURCES.
A MIGRATION_SQL DELETE cleans up pre-merge DBs that still have the
orphan generate_brands row.
Other review items addressed in-line:
H1 generate_atlas._write_stamp no longer commits — transaction ownership
stays with the caller (matches import_economics pattern). Stamp +
atlas data now commit atomically; a failed stamp rolls back the
atlas data rather than leaving a stamp-missing-data intermediate.
H2 _file_sha1 (in both import_economics, generate_atlas,
check-systems-db-stamp) raises FileNotFoundError on missing sources
instead of silently contributing an empty-bytes hash. A ghost-SHA
convergence could otherwise produce vacuous "fresh" passes.
H4 pre-push no-meta-table warning rephrased — was "run after next
regeneration", now "run now if this DB was generated by you".
T1 asset-pipeline.md determinism claim softened: the stamp is
deterministic (same source → same recorded SHA), the DB binary is
not (generated_at + SQLite rowids/freelist churn).
T3 asset-pipeline.md gains a "migration escape hatch" section naming
MIGRATION_SQL in import_economics.py as the only sanctioned path
for direct writes, and forbidding hand-run sqlite-exec / one-off
patch scripts / SQLite-GUI edits.
T4 Makefile regen-db now runs as a single shell with `set -e`. A
failure in one generator halts the pipeline immediately, preventing
the "stale data, fresh stamp" state where a later step stamped a
DB whose earlier step had failed. import_economics' exit code 2
(coverage gate warning) remains explicitly tolerated.
T5 pre-push stamp check now runs on a branch's first push too —
compares against origin/main instead of origin/$BRANCH, closing
the gap where a new branch could ship a stale DB via the first push.
T6 check-systems-db-stamp fails closed on unknown generator_names in
meta — a future branch adding a new generator without registering
it in GENERATOR_SOURCES will now be rejected, not silently skipped.
T7 /pr-push watch list gains a mutual cross-reference comment with
GENERATOR_SOURCES in check-systems-db-stamp, plus the missing
names.rs source file, so the two lists cannot silently drift.
Follow-up tickets created:
#887 T8 decisions-orphan-tickets CLI — surfaces tickets whose
decision_ref points at a non-existent D-record.
#888 T9 meta.schema_version monotonic semver — for savegame migration
lineage in Phase 5+ (SHA comparison can't be ordered).
Verified:
make regen-db end-to-end — OK
make check-systems-db — OK, 2 generator(s) up to date
STALE detection — OK, verified by touching generate_atlas.py
/pr-push watch list — OK, flags this branch's changed sources
decision show D-159 — OK, structured output with tickets + refs
Refs: #855#856#857#858#859 PR #136
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds systems.db regeneration discipline (#855) via a `meta` table (#856)
stamped by every generator, a pre-push hook that rejects stale DBs (#857),
and the top-level `make regen-db` / `make check-systems-db` targets that
drive the whole pipeline.
The stamp stores SHA-1 of generator source + schema, so the pre-push hook
can cheaply detect "you changed a generator but forgot to regen the DB"
before a binary merge conflict lands. Sprint 36 hit that class of conflict
on two branches touching systems.db simultaneously — this is the systemic
fix.
Regenerated systems.db is stamped; `make check-systems-db` passes.
Refs: #855#856#857
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
jpmschweitzerandClaude Fable 5