Merge remote-tracking branch 'origin/main' into sprint-37/server

# Conflicts:
#	CHANGELOG.md
#	Makefile
#	server/data/systems.db
This commit is contained in:
2026-04-22 09:01:43 +02:00
15 changed files with 827 additions and 12 deletions
+173
View File
@@ -0,0 +1,173 @@
# Asset Pipeline — Source-Canonical Rule
`server/data/systems.db` is a **read-only, deterministic snapshot** produced by the
generator pipeline. It is checked in to the repo as a build artefact so the Godot
client can ship it without a build step, but **it is never the source of truth**.
---
## The Golden Rule
> **Edit sources, not the DB.**
If you need to change economics data, modify the TOML/JSON source files.
If you need to change atlas markers, modify the `markers.json` files.
Never run `UPDATE` or `INSERT` directly on `server/data/systems.db` outside of a
migration — those changes will be silently overwritten by the next `make regen-db`.
---
## What produces systems.db
Two generators write to `systems.db`:
| Generator | Command | Source files (all contribute to the meta stamp SHA) |
|-----------|---------|--------------|
| `import_economics` | `python3 tooling/economy-db/import_economics.py` | `tooling/economy-db/import_economics.py` + the Rust brand binary sources it invokes: `server/src/bin/generate_brands/main.rs`, `server/src/bin/generate_brands/names.rs`, `tooling/generate-brands` |
| `generate_atlas` | `python3 tooling/planet-gen/generate_atlas.py --seed 42` | `tooling/planet-gen/generate_atlas.py` |
`import_economics` shells out to the Rust `generate_brands` binary as its first
step to refresh `wiki/economics/corporations/generated_brands.toml`, then reads
the TOML and imports brand data into the DB. The Rust binary is a subroutine
of the Python importer, not an independent generator — changes to its source
invalidate the `import_economics` meta stamp even though the Python file
itself didn't change.
`make regen-db` runs both in the correct order (economics first, atlas second).
---
## The meta table stamp (#855, #856)
After every successful non-dry-run, each generator writes a row to the `meta` table:
```sql
CREATE TABLE meta (
generator_name TEXT PRIMARY KEY, -- 'import_economics' | 'generate_atlas'
schema_version TEXT NOT NULL, -- SHA-1 of server/data/systems-schema.sql at generation time
generator_sha TEXT NOT NULL, -- SHA-1 of the generator source file(s)
generated_at TEXT NOT NULL DEFAULT (datetime('now'))
);
```
The `generator_sha` is the SHA-1 of the concatenated bytes of the generator's
source files (sorted by path, so order is deterministic). If any source file
changes and `make regen-db` is not re-run, the stamped SHA will differ from the
recomputed current SHA — this is what the pre-push hook detects.
**What's deterministic:** the stored SHA (same sources → same recorded SHA).
**What's NOT deterministic:** the DB binary itself. `meta.generated_at` uses
`datetime('now')`, SQLite `rowid`/`autoincrement` values drift across runs, and
transaction ordering can reshape freelist pages — two consecutive `make regen-db`
calls produce byte-different SQLite files even with identical inputs. This is
fine: the freshness guarantee comes from the stamp, not from bytewise DB equality.
---
## How to make a DB change
### Normal data changes (economics, atlas markers)
1. Edit the source files (TOML, JSON, markers.json).
2. Run `make regen-db`.
3. Run `make check-systems-db` to confirm the stamp is fresh.
4. Stage and commit:
```bash
git add server/data/systems.db
git commit -m "chore(db): regen systems.db — <what changed>"
```
### Schema changes (new tables or columns)
1. Add the DDL to `server/data/systems-schema.sql`.
2. Add migration SQL to `MIGRATION_SQL` in `import_economics.py` if the change
affects existing DBs (idempotent `CREATE TABLE IF NOT EXISTS` or `ALTER TABLE`).
3. Run `make regen-db`.
4. Stage `server/data/systems-schema.sql` and `server/data/systems.db` together.
---
## Pre-push hook (#857)
`.config/hooks/pre-push` (installed via `make install-hooks`) checks that whenever
`server/data/systems.db` is in the push, its meta stamp matches the current generator
source SHAs. If not, the push is rejected with:
```
systems.db is stale — run `make regen-db` before pushing.
Stale generators: ['import_economics']
```
Fix: run `make regen-db`, stage `server/data/systems.db`, amend or add a commit.
Or use `/pr-push` — it detects stale generator sources and reruns `make regen-db`
automatically before pushing.
The check script is `tooling/check-systems-db-stamp`. Run it interactively with
`make check-systems-db` or `python3 tooling/check-systems-db-stamp --verbose`. The
`GENERATOR_SOURCES` dict at the top of that script is the single registry — when
you add a new generator or source file, update it there and mirror the change in
the `/pr-push` skill's source-file watch list.
---
## /pr-push integration (#858)
The `/pr-push` skill checks whether any generator source files are modified on the
branch. If they are, it automatically runs `make regen-db` and stages the updated
`server/data/systems.db` before pushing — preventing pre-push hook rejections on
branches that modify generators without regenerating.
---
## Why direct DB edits are forbidden
Two branches that both commit `server/data/systems.db` changes produce a binary
merge conflict. Git cannot diff or merge binary SQLite files. Sprint 36 hit this
exact class of problem. The meta stamp + pre-push hook is the systematic fix:
- The stamp is deterministic (same generator source → same recorded SHA)
- Only one branch modifies generator sources at a time (per team scope rules)
- The pre-push hook is a hard blocker before the binary conflict can land
## The migration escape hatch
The rule above says "never run UPDATE or INSERT directly on systems.db outside
of a migration." Here's what a legitimate migration looks like, and what isn't
one:
**Sanctioned path: the `MIGRATION_SQL` block in `import_economics.py`.** That
string is executed at the top of every import run (inside the same transaction
that clears + reimports data) and contains idempotent `CREATE TABLE IF NOT
EXISTS` / `CREATE INDEX IF NOT EXISTS` statements, plus `ALTER TABLE` additions
handled via the `COLUMN_MIGRATIONS` list. When you need a new table, column,
or index on systems.db, add it there. It'll run on the next `make regen-db`
and the meta stamp will flip because `import_economics.py` changed.
**Also legitimate:** edits to `server/data/systems-schema.sql` (the canonical
DDL used by fresh builds) paired with matching entries in `MIGRATION_SQL` for
existing DBs. The stamp's `schema_version` field records the schema file's
SHA at generation time — change the schema, commit both files together, and
the stamp picks it up automatically.
**NOT legitimate and forbidden:**
- Running `tooling/db/sqlite-exec` (or any raw SQL) against `systems.db` by
hand. Any changes you make are silently reverted by the next `regen-db` run
— your edits die, not the pipeline's.
- One-off patch scripts that open `systems.db` and modify rows.
- Editing the DB file with a SQLite GUI.
- Committing `systems.db` alone, without the corresponding source change that
would explain the diff on regen.
If you think you need an exception, the right move is to make the source
change explicit instead: either edit the wiki TOMLs / JSONs that feed the
generators, or edit `MIGRATION_SQL` / `systems-schema.sql` directly. There is
no hand-edit path that survives regen.
---
## Future: savegame migration lineage
The `meta.schema_version` field records the schema SHA at generation time. When the
savegame system is built (Phase 5+), a save file can record which systems.db snapshot
it derives from, enabling forward migration without branching the DB file itself.
+81
View File
@@ -26,6 +26,20 @@ current branch — never touches main.
## Workflow
### 0. Dry-run mode check
If the user invokes `/pr-push --dry-run`:
- Print: "Dry-run mode — inspecting state, nothing will be pushed or committed."
- Run steps 1 through 4a in **inspect-only** mode:
- Step 4: run `make check-systems-db` to check current stamp freshness (no merge)
- Step 4a: report which watched files changed vs origin/main; show whether `make regen-db`
would be triggered; do NOT run the regen, stage, or commit
- Print a summary: watched files changed (list), regen needed (yes/no), DB stamp fresh (yes/no)
- Print "Dry run complete — use /pr-push to apply."
- Stop. Do not push or create a PR.
---
### 1. Validate branch
```bash
@@ -201,6 +215,73 @@ git merge origin/main --no-edit
If merge conflicts, **stop and report** — let the user resolve.
If clean, continue.
### 4a. Regen systems.db if generator sources or data changed (#858)
Check whether any file in the **source-file watch list** was modified on this branch
versus `origin/main`. This list covers generator code AND the data files that feed them.
The generator-source paths below **must stay in sync** with `GENERATOR_SOURCES` in
`tooling/check-systems-db-stamp` (PR #136 review T7) — if you add a new source file
to the stamp, add it here too, and vice versa. Drift between the two lists reintroduces
exactly the silent-stale-DB class of bug this skill exists to prevent.
```bash
git diff --name-only origin/main...HEAD -- \
tooling/economy-db/import_economics.py \
tooling/planet-gen/generate_atlas.py \
server/src/bin/generate_brands/main.rs \
server/src/bin/generate_brands/names.rs \
tooling/generate-brands \
server/data/systems-schema.sql \
wiki/star-systems/ \
wiki/economics/ \
content/economics/
```
**If output is empty:** skip this step entirely.
**If any files appear in the output:** the DB must be regenerated on top of the
current main. Perform the following:
1. **Integrate main.** Step 4 merged main into the branch. If you find yourself
on a branch that was NOT yet merged with main in step 4, do it now:
```bash
git fetch origin
git merge origin/main --no-edit
```
If there are merge conflicts in source files, **stop and report which files
conflict**. Ask the user to resolve manually — do not attempt to auto-resolve
generator source conflicts.
2. **Regenerate the DB:**
```bash
make regen-db
```
`make regen-db` runs all three generators and stamps the meta table. It tolerates
coverage gate failures (exit 2 = data quality warning, not an error). If it exits
with any other non-zero code, stop and report the stderr output — do not push.
3. **Stage the updated DB:**
```bash
git add server/data/systems.db
```
4. **Commit only if the DB actually changed:**
```bash
git diff --cached --stat -- server/data/systems.db
```
- If the diff shows changes: commit with `/git-commit`, message:
`chore(db): regen systems.db against rebased sources`
- If no diff (regen produced identical output — sources were self-consistent):
unstage the file (`git restore --staged server/data/systems.db`) and skip the
commit. The source changes alone are the PR content.
**In dry-run mode** (from step 0): report which watch-list files changed and
whether regen would be triggered. Do NOT run the regen or modify any files.
This step prevents the pre-push hook from rejecting a push where the branch modifies
a generator source or data file but did not regenerate the DB.
### 5. Push
```bash
+17
View File
@@ -365,11 +365,25 @@ using `TaskUpdate` with `addBlockedBy`.
For each agent from the `**Agents:**` line, spawn a teammate in the
background. Spawn all agents in parallel (one message, multiple Task calls):
**Model pin (MANDATORY for team members):** every team-mode spawn —
i.e. any `Task` with a `team_name` argument — must pass `model: "sonnet"`.
Sprint 37 observed Opus 4.7 teammates ignoring scope rules, leaving
tasks half-done, and failing to report back via SendMessage. Sonnet 4.6
follows literal rules block discipline better. The **team lead**
(this session, running `/sprint-start`) stays on whatever model the
user has selected — typically Opus.
**Inline (non-team) Agent spawns are exempt.** One-shot reviewers
(`/pr-review`), research subagents, and other `Task` calls without a
`team_name` keep their default model. The pin applies to the
long-running team-coordination path specifically, not every Agent call.
```
Task(
subagent_type: "{name_lowercase}",
team_name: "sprint-{N}-{team}",
name: "{name_lowercase}",
model: "sonnet",
prompt: "You are on the {team} team for Sprint {N}.
Branch: `sprint-{N}/{team}`
@@ -423,6 +437,9 @@ Task(
1. Read the sprint briefing: docs/sprints/sprint-{N}/{team}.md
2. Read the decision files referenced in the briefing.
If your work touches systems.db sources (markers.json, TOML files,
or generator code), read .claude/rules/asset-pipeline.md before
modifying anything.
3. Check TaskList for available work.
4. Claim an unblocked task (TaskUpdate with owner: your name),
mark it in_progress, and implement it.
+1
View File
@@ -61,6 +61,7 @@ Use the Task tool to spawn each agent as a teammate. Each call should:
- Set `team_name` to the workshop team name
- Set `name` to the agent name (e.g., "gestalt")
- Set `subagent_type` to the matching agent type (same as name — see reference table)
- **Set `model: "sonnet"`** — team-mode participants pin to Sonnet 4.6 for literal-rule-following discipline. Opus 4.7 was observed ignoring scope rules and failing to report back in team mode (Sprint 37). The *team lead* (this session) stays on whatever model the user has selected.
- Provide a prompt telling the agent to check TaskList for their assigned task
Spawn all agents in parallel (one Task call per agent in a single message). Agents will appear as teammates in the Claude Code UI and pick up their tasks from the shared task list.
+34
View File
@@ -138,6 +138,40 @@ else
echo "pre-push: no JSON changes — skipping"
fi
# --- systems.db stamp check (#857) ---
# If the branch touches server/data/systems.db and the meta stamp does not
# match current generator sources, reject the push. Prevents pushing a
# stale DB snapshot where generator source was modified but the DB was not
# regenerated.
#
# Runs whenever systems.db was modified in ANY branch commit vs. main —
# including on a branch's very first push (review T5: the previous version
# skipped the check for new branches because it compared against origin/$BRANCH,
# which didn't exist yet, leaving a gap where a stale DB could ship via the
# first push). We compare against origin/main — which always exists — so the
# check covers the first-push case.
DB_IN_PUSH=$(git diff --name-only origin/main...HEAD -- server/data/systems.db 2>/dev/null | wc -l)
if [ "$DB_IN_PUSH" -gt 0 ] && [ -f "$REPO_ROOT/tooling/check-systems-db-stamp" ]; then
echo "pre-push: checking systems.db stamp..."
rc=0
python3 "$REPO_ROOT/tooling/check-systems-db-stamp" || rc=$?
if [ "$rc" -eq 1 ]; then
# rc=1 means stale / unknown generator / missing source; message on stderr
echo " Fix: run 'make regen-db' then stage server/data/systems.db"
echo " Or use /pr-push — it handles regen automatically before pushing."
ERRORS=$((ERRORS + 1))
elif [ "$rc" -eq 2 ]; then
# rc=2 means no meta table — treat as unstamped, warn but don't block.
# This is legitimate immediately after the meta table is introduced;
# the next `make regen-db` will populate it (H4).
echo "pre-push: WARNING — systems.db has no meta stamp — run 'make regen-db' now if this DB was generated by you"
else
echo "pre-push: systems.db stamp — OK"
fi
else
echo "pre-push: systems.db not in push — skipping stamp check"
fi
if [ "$ERRORS" -gt 0 ]; then
echo ""
echo "pre-push: $ERRORS check(s) failed. Push aborted."
+6
View File
@@ -7,6 +7,11 @@ Format based on [Keep a Changelog](https://keepachangelog.com/).
## [Unreleased]
### Added
- **Asset pipeline discipline** (#854, #855, #856, #857, #858, #859) — `systems.db` is now a source-canonical snapshot with a `meta` table stamped by every generator (SHA of source + schema). Pre-push hook rejects stale DBs; `/pr-push` auto-runs `make regen-db` when generator sources change. Full rules in `.claude/rules/asset-pipeline.md`
- **`make regen-db`** — runs the two DB-writing generators (import_economics, generate_atlas) and stamps the meta table. import_economics now invokes the Rust generate_brands binary internally as its first step, so the brand pipeline is owned by a single stamp.
- **`make check-systems-db`** — verifies the meta stamp matches current generator sources
- **`make install-hooks`** — installs pre-push and pre-commit hooks in one step
- **`tooling/db/decision show <D-NNN>`** (#723) — drill-down view of a decision with implementing tickets and cross-refs
- **Atlas determinism smoke test** (#847) — `make test-atlas-determinism` runs `generate_atlas.process_body()` twice with a fixed seed and diffs the output to catch determinism regressions in terrain analysis, city placement, A* routing, and naming
- **SelectedBookmark save/load** (#863) — bookmark and starting-location choice now persist across save/load; replaces the v0.2-deferred TODO on `SelectedBookmark`
- **`BookmarkPlugin::new(registry)` injection** (#862) — test-friendly plugin construction for future TOML bookmark loading; default constructor still wires the canonical tycoon registry
@@ -14,6 +19,7 @@ Format based on [Keep a Changelog](https://keepachangelog.com/).
- **Atlas naming corridor-scoped dedup, compass-direction filter, river vocab filter, infra pair-naming** (#853) — city and mountain names deduplicate across bodies within a corridor; compass-direction defaults blocked in the few-shot prompt; navigational vocabulary (`Flow`, `Current`) rejected for rivers; unnamed roads and railroads receive deterministic `{CityA}{CityB} {corridor_suffix}` names
### Changed
- `decisions-coverage` Makefile target now lists implementing ticket IDs per decision instead of aggregate counts
- **Economy coverage gate** (#860) now passes end-to-end — closes the 21 raw-commodity / system gaps that blocked Phase 2 demand simulation
- Tag updates on 15 existing corporation wiki pages to match commodity coverage needs
+7 -1
View File
@@ -25,6 +25,12 @@ Full annotated tree: `.claude/rules/project-structure.md`
See [docs/DEVOPS.md](docs/DEVOPS.md) for build, test, lint, and CI procedures. All development operations go through the top-level `Makefile` — run `make` for a summary of targets.
### Asset pipeline
`server/data/systems.db` is a read-only canonical snapshot produced by the generator
pipeline — never edit it directly. To regenerate: `make regen-db`. Full rules in
`.claude/rules/asset-pipeline.md`.
## Development Cascade — First Things First
Development follows a strict cascade. Each phase has a concrete deliverable. **Do NOT discuss, design, or implement detail from a later phase while an earlier phase is incomplete.** If you encounter references to later-phase detail (room grammar, NPC bundles, heritage tokens, etc.) in documents or decisions, either ignore them silently and stay at the correct level, or flag that the reference is dragging attention to the wrong scope level and suggest it be rephrased or moved.
@@ -89,7 +95,7 @@ The ticketing database (`settledreach.db`) is accessed via `SR_DB_PATH` env var
| Sprints | `tooling/db/sprint status`, `start-work`, `prepare` | `/sprint-start` skill |
| SQL queries | `tooling/db/sqlite-query "SELECT ..."` | — |
| SQL writes | `tooling/db/sqlite-exec "UPDATE ..."` | — |
| Decisions | `tooling/db/decision next`, `claim`, `check-dupes` | — |
| Decisions | `tooling/db/decision show`, `next`, `claim`, `check-dupes` | — |
### Testing preferences
+32 -4
View File
@@ -2,8 +2,8 @@ GODOT := $(shell command -v godot4 2>/dev/null || command -v godot 2>/dev/null)
.PHONY: help setup build check-protocol client server game stop test lint lint-python setup-venv ci ci-client ci-server clean \
decisions-sync decisions-coverage decisions-active decisions-orphan \
db-backup db-install validate-content check-fact-ids setup-hooks \
audit deny atlas-verify economy-db atlas-generate \
db-backup db-install validate-content check-fact-ids setup-hooks install-hooks \
audit deny atlas-verify economy-db atlas-generate regen-db check-systems-db \
pre-pr pre-pr-lint pre-pr-build pre-pr-test pre-pr-validate pre-pr-fixtures \
pre-pr-server pre-pr-client pre-pr-content \
fixtures-client fixtures-gauntlet golden-diff golden-update \
@@ -46,7 +46,7 @@ help:
@echo " make db-install Restore shared database from backup"
@echo ""
@echo " make decisions-sync Sync decisions/*.md into SQLite"
@echo " make decisions-coverage Decision-to-ticket coverage by domain"
@echo " make decisions-coverage Each decision with its implementing ticket(s)"
@echo " make decisions-active List active decisions"
@echo " make decisions-orphan Decisions without implementing tickets"
@echo " make audit Run cargo audit (security advisory check)"
@@ -58,6 +58,9 @@ help:
@echo " make check-star-map Assert star_map_data.json is up to date (part of pre-pr-client)"
@echo " make economy-db Import economics data into systems.db (TOML/JSON → SQLite)"
@echo " make atlas-generate Generate atlas city/road/rail markers for all inhabited bodies"
@echo " make regen-db Regenerate systems.db from all sources + stamp meta table (#855)"
@echo " make check-systems-db Verify systems.db meta stamp matches current generator sources"
@echo " make install-hooks Install pre-push + pre-commit git hooks (once per clone)"
@echo " make test-atlas-determinism Determinism smoke test for generate_atlas.py (#847)"
@echo " make fixtures-client Generate GDScript->Rust cross-encoder fixtures (#475)"
@echo " make golden-diff Show diff if golden file output has changed"
@@ -111,6 +114,10 @@ setup-hooks:
@git config core.hooksPath .config/hooks
@echo "Git hooks path set to .config/hooks"
install-hooks: setup-hooks
@chmod +x .config/hooks/pre-push .config/hooks/pre-commit
@echo "Hooks installed — pre-push and pre-commit are active."
setup-venv:
@python3 -m venv .venv
@.venv/bin/pip install -e ".[dev]" --quiet
@@ -351,6 +358,27 @@ atlas-generate: ## Generate atlas markers (cities, roads, rail) for all inhabit
echo " [guard] $$count bodies with terrain_reference — proceeding."
@python3 tooling/planet-gen/generate_atlas.py --seed 42
regen-db: ## Regenerate systems.db from all sources and stamp meta table (#855, #856)
@# Run as a single shell so `set -e` covers all steps. Without this
@# each recipe line was a fresh shell and a failure in step 1 did not
@# halt step 2, which could produce stale data with a fresh stamp
@# (PR #136 review T4). import_economics' exit code 2 is a valid
@# coverage-gate-warning state (DB and stamp committed), not an error,
@# so it's explicitly tolerated. Any other non-zero exit halts the
@# pipeline immediately.
@set -e; \
echo " [regen-db] Importing economics data (runs generate_brands internally)..."; \
ec=0; python3 tooling/economy-db/import_economics.py || ec=$$?; \
if [ $$ec -ne 0 ] && [ $$ec -ne 2 ]; then exit $$ec; fi; \
echo " [regen-db] Running atlas generator..."; \
python3 tooling/planet-gen/generate_atlas.py --seed 42; \
echo ""; \
echo " regen-db complete — systems.db is up to date and stamped."; \
echo " Stage it with: git add server/data/systems.db"
check-systems-db: ## Verify systems.db meta stamp matches current generator sources (#857)
@python3 tooling/check-systems-db-stamp --verbose
econ-sim: ## Build the economics simulation binary (Layer 1+2: Leontief + tâtonnement trade)
@cargo build --manifest-path tooling/econ-sim/Cargo.toml --release
@echo "Built: tooling/econ-sim/target/release/econ-sim"
@@ -368,7 +396,7 @@ decisions-sync:
@tooling/db/decisions-sync
decisions-coverage:
@tooling/db/sqlite-query "SELECT d.domain, COUNT(DISTINCT d.id) as decisions, COUNT(DISTINCT t.decision_ref) as with_tickets FROM decisions d LEFT JOIN tickets t ON d.id = t.decision_ref WHERE d.status='active' AND d.type='confirmed' GROUP BY d.domain"
@tooling/db/sqlite-query "SELECT d.id, d.domain, d.title, COALESCE(GROUP_CONCAT(t.id, ', '), '') as implementing_tickets FROM decisions d LEFT JOIN tickets t ON d.id = t.decision_ref WHERE d.status='active' AND d.type='confirmed' GROUP BY d.id ORDER BY d.domain, d.id"
decisions-active:
@tooling/db/sqlite-query "SELECT id, domain, title FROM decisions WHERE status='active' AND type='confirmed' ORDER BY domain, id"
+52 -5
View File
@@ -207,13 +207,55 @@ Schema: `content/_schema/checklist.schema.json`. The checklist format feeds into
- **Advisory** — when knowledge catalogs (`content/global/knowledge/*.yaml`) have no fact definitions yet: lists referenced fact_ids and exits cleanly.
- **Enforcing** — when catalogs are populated: fails on any `fact_id` reference that doesn't match a canonical definition.
## Pre-commit Hooks
## Asset Pipeline — Generator-Driven DB (#855, #856, #857)
`server/data/systems.db` is a **read-only canonical snapshot** produced by three
generators. It is committed to the repo so the client can ship it, but it is never
the source of truth. Direct edits are forbidden — they are silently overwritten by
the next regeneration.
### Generators
| Generator | Source | Runs via |
|-----------|--------|----------|
| `generate_brands` | `server/src/bin/generate_brands/main.rs` | `tooling/generate-brands` |
| `import_economics` | `tooling/economy-db/import_economics.py` | `python3 tooling/economy-db/import_economics.py` |
| `generate_atlas` | `tooling/planet-gen/generate_atlas.py` | `python3 tooling/planet-gen/generate_atlas.py --seed 42` |
Run all three at once with:
```bash
make regen-db
```
### Meta table stamp
After every successful non-dry-run, each generator writes a row to the `meta` table in
`systems.db` recording the SHA-1 of its source file(s) and the schema file.
```bash
make check-systems-db # Verify the stamp is fresh (exit 1 = stale)
```
### Making a DB change
1. Edit source files (TOML, JSON, `markers.json`).
2. `make regen-db`
3. `git add server/data/systems.db`
4. Commit with `chore(db): regen systems.db — <reason>`
For schema changes, also update `server/data/systems-schema.sql` and add migration DDL
to `MIGRATION_SQL` in `import_economics.py`.
See `.claude/rules/asset-pipeline.md` for the full rule set.
## Pre-commit and Pre-push Hooks
Git hooks are stored in `.config/hooks/` (version-controlled). Activate them with:
```bash
make setup # Includes hook installation
make setup-hooks # Just hooks
make install-hooks # Just hooks (also makes them executable)
```
Or manually:
@@ -224,9 +266,14 @@ git config core.hooksPath .config/hooks
Active checks:
| Check | Script | Behavior |
|-------|--------|----------|
| fact_id validation | `tooling/check-fact-ids` | Warns if catalogs are stubs; fails on unknown fact_ids when populated |
| Hook | Check | Script | Behavior |
|------|-------|--------|----------|
| pre-commit | fact_id validation | `tooling/check-fact-ids` | Warns if catalogs are stubs; fails on unknown fact_ids when populated |
| pre-push | GDScript parse | internal | Fails on any SCRIPT ERROR |
| pre-push | Rust lint | internal | fmt + clippy |
| pre-push | Python lint | internal | ruff |
| pre-push | JSON syntax | internal | python3 -m json.tool |
| pre-push | systems.db stamp | `tooling/check-systems-db-stamp` | Rejects stale DB when pushed (#857) |
The `core.hooksPath` setting uses a relative path (`.config/hooks`) that resolves per worktree, so it works correctly across all worktrees in the repository.
+17
View File
@@ -480,6 +480,23 @@ CREATE INDEX IF NOT EXISTS idx_corps_type ON corporations(corp_type);
CREATE INDEX IF NOT EXISTS idx_star_systems_currency ON star_systems(currency_zone);
CREATE INDEX IF NOT EXISTS idx_gate_links_from ON gate_links(from_system_id);
CREATE INDEX IF NOT EXISTS idx_gate_links_to ON gate_links(to_system_id);
-- Generator metadata stamp (#855, #856)
-- One row per generator, updated on each successful non-dry-run.
-- schema_version: SHA-1 of server/data/systems-schema.sql content at generation time
-- generator_sha: SHA-1 of the generator source file(s) content
-- generated_at: ISO-8601 UTC timestamp of the run
--
-- Used by:
-- tooling/check-systems-db-stamp — verifies freshness before push (#857)
-- .config/hooks/pre-push — rejects pushes with stale DB (#857)
-- /pr-push skill — triggers make regen-db if stale (#858)
CREATE TABLE IF NOT EXISTS meta (
generator_name TEXT PRIMARY KEY, -- 'import_economics' | 'generate_atlas' | 'generate_brands'
schema_version TEXT NOT NULL, -- SHA-1 hex of systems-schema.sql content
generator_sha TEXT NOT NULL, -- SHA-1 hex of generator source file(s) content
generated_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE INDEX IF NOT EXISTS idx_commodities_tier ON commodities(tier);
CREATE INDEX IF NOT EXISTS idx_production_chains_output ON production_chains(output_commodity_id);
CREATE INDEX IF NOT EXISTS idx_chain_inputs_commodity ON chain_inputs(input_commodity_id);
Binary file not shown.
+154
View File
@@ -0,0 +1,154 @@
#!/usr/bin/env python3
"""
check-systems-db-stamp — verify that server/data/systems.db is up to date.
Reads the meta table from systems.db and checks that the stored SHA-1 of each
generator's source file(s) matches the current file content on disk.
Exit codes:
0 — DB is stamped and all generator SHAs match current sources
1 — DB is stale, has an unknown generator, or references a missing source file
2 — DB does not have a meta table (treat as unstamped — run make regen-db)
Usage (called by .config/hooks/pre-push):
tooling/check-systems-db-stamp
Usage (interactive):
tooling/check-systems-db-stamp --verbose
Decision refs: #855 (generator versioning), #857 (pre-push hook)
"""
import hashlib
import sqlite3
import sys
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parent.parent
DB_PATH = REPO_ROOT / "server" / "data" / "systems.db"
# Maps generator_name (as stored in meta.generator_name) to the source
# file(s) whose SHA is stamped. The SHA is computed as SHA-1 of the
# concatenated bytes of all files in sorted order.
#
# import_economics' source set includes the Rust generate_brands binary it now
# invokes as a subroutine (#136 review T2/H3). Keep this list in sync with
# IMPORT_ECONOMICS_SOURCES in tooling/economy-db/import_economics.py.
GENERATOR_SOURCES: dict[str, list[Path]] = {
"import_economics": [
REPO_ROOT / "tooling" / "economy-db" / "import_economics.py",
REPO_ROOT / "server" / "src" / "bin" / "generate_brands" / "main.rs",
REPO_ROOT / "server" / "src" / "bin" / "generate_brands" / "names.rs",
REPO_ROOT / "tooling" / "generate-brands",
],
"generate_atlas": [
REPO_ROOT / "tooling" / "planet-gen" / "generate_atlas.py",
],
}
def file_sha1(*paths: Path) -> str:
"""SHA-1 of concatenated file contents (sorted paths).
Missing files raise FileNotFoundError rather than silently contributing
an empty-string hash (H2): a ghost SHA could mask real breakage when
stored and current SHAs converge on the empty-bytes digest.
"""
h = hashlib.sha1()
for p in sorted(paths):
if not p.exists():
raise FileNotFoundError(f"generator source not found: {p}")
h.update(p.read_bytes())
return h.hexdigest()
def check(verbose: bool = False) -> int:
"""Return exit code: 0 = fresh, 1 = stale, 2 = no meta table."""
if not DB_PATH.exists():
if verbose:
print(f"check-systems-db-stamp: {DB_PATH} not found — skipping check")
return 0
try:
conn = sqlite3.connect(str(DB_PATH))
rows = conn.execute(
"SELECT generator_name, generator_sha FROM meta"
).fetchall()
conn.close()
except sqlite3.OperationalError:
# meta table does not exist
if verbose:
print("check-systems-db-stamp: no meta table — systems.db has not been stamped")
print(" Run: make regen-db")
return 2
if not rows:
if verbose:
print("check-systems-db-stamp: meta table is empty — systems.db has not been stamped")
print(" Run: make regen-db")
return 2
stale: list[str] = []
unknown: list[str] = []
for generator_name, stored_sha in rows:
sources = GENERATOR_SOURCES.get(generator_name)
if sources is None:
# Unknown generator — fail closed (T6). A future branch adding a
# new generator without registering it here must update this map
# before the check will pass, preventing the "silent no-op" trap.
unknown.append(generator_name)
continue
try:
current_sha = file_sha1(*sources)
except FileNotFoundError as exc:
# Source file moved/deleted — explicit failure instead of
# silent empty-hash (H2).
print(
f"check-systems-db-stamp: BROKEN — {generator_name}: {exc}",
file=sys.stderr,
)
return 1
if current_sha != stored_sha:
stale.append(generator_name)
if verbose:
print(
f"check-systems-db-stamp: STALE — {generator_name}"
f"\n stored: {stored_sha}"
f"\n current: {current_sha}"
)
if unknown:
print(
"check-systems-db-stamp: UNKNOWN generator(s) in meta table: "
f"{unknown}",
file=sys.stderr,
)
print(
" Update GENERATOR_SOURCES in tooling/check-systems-db-stamp to "
"register them before pushing.",
file=sys.stderr,
)
return 1
if stale:
if not verbose:
print(
"systems.db is stale — run `make regen-db` before pushing.",
file=sys.stderr,
)
print(f" Stale generators: {stale}", file=sys.stderr)
return 1
if verbose:
print(f"check-systems-db-stamp: OK — {len(rows)} generator(s) up to date")
return 0
def main() -> None:
verbose = "--verbose" in sys.argv or "-v" in sys.argv
sys.exit(check(verbose=verbose))
if __name__ == "__main__":
main()
+49
View File
@@ -414,6 +414,49 @@ def claim_id(cfg, prefix, domain, title):
conn.close()
def show_decision(cfg, decision_id):
"""Show a single decision with full details including linked tickets and cross-refs."""
conn = get_connection(cfg)
try:
row = conn.execute(
"SELECT * FROM decisions WHERE id = ?",
(decision_id,),
).fetchone()
if not row:
return {"ok": False, "error": f"Decision not found: {decision_id}"}
decision = dict(row)
# Implementing tickets: tickets where decision_ref = this ID
ticket_rows = conn.execute(
"SELECT id, title, status, type FROM tickets"
" WHERE decision_ref = ? ORDER BY id",
(decision_id,),
).fetchall()
decision["implementing_tickets"] = [dict(t) for t in ticket_rows]
# Cross-refs outbound: references from this decision to others
refs_out = conn.execute(
"SELECT target_id, ref_type, note FROM decision_refs"
" WHERE source_id = ? ORDER BY target_id",
(decision_id,),
).fetchall()
decision["refs_out"] = [dict(r) for r in refs_out]
# Cross-refs inbound: other decisions referencing this one
refs_in = conn.execute(
"SELECT source_id, ref_type, note FROM decision_refs"
" WHERE target_id = ? ORDER BY source_id",
(decision_id,),
).fetchall()
decision["refs_in"] = [dict(r) for r in refs_in]
return {"ok": True, "decision": decision}
finally:
conn.close()
def check_dupes(cfg):
"""Check for duplicate decision IDs across all markdown files."""
# Pre-existing collisions too deeply embedded to renumber (139+ references).
@@ -461,6 +504,7 @@ Settled Reach Decisions Sync & ID Management
Usage:
decisions_sync.py sync Parse decisions/*.md and upsert into SQLite
decisions_sync.py show <D-NNN> Show a decision with linked tickets + refs
decisions_sync.py next [D|Q|R] Show next available ID (all prefixes or one)
decisions_sync.py claim <D|Q|R> <domain> [title] Claim next ID and insert placeholder
decisions_sync.py check-dupes Check for duplicate IDs across markdown files
@@ -493,6 +537,11 @@ def main():
if cmd == "sync":
result = sync(cfg)
elif cmd == "show":
if len(sys.argv) < 3:
result = {"ok": False, "error": "Usage: show <decision_id> e.g. show D-159"}
else:
result = show_decision(cfg, sys.argv[2])
elif cmd == "next":
result = next_id(cfg, sys.argv[2] if len(sys.argv) > 2 else None)
elif cmd == "claim":
+137 -1
View File
@@ -24,6 +24,7 @@ Usage:
"""
import argparse
import hashlib
import json
import re
import sqlite3
@@ -41,6 +42,101 @@ SCHEMA_SQL = REPO_ROOT / "server" / "data" / "systems-schema.sql"
CORPORATIONS_DIR = REPO_ROOT / "wiki" / "corporations"
BRANDS_TOML = REPO_ROOT / "wiki" / "economics" / "corporations" / "brands.toml"
GENERATED_BRANDS_TOML = REPO_ROOT / "wiki" / "economics" / "corporations" / "generated_brands.toml"
# Rust sources for the generate_brands subroutine. import_economics shells out to
# tooling/generate-brands as part of its normal flow (see regenerate_brands()), so
# both Rust files contribute to this script's effective source SHA: any change to
# either must invalidate the meta stamp even though Python hasn't changed.
GENERATE_BRANDS_RS = REPO_ROOT / "server" / "src" / "bin" / "generate_brands" / "main.rs"
GENERATE_BRANDS_NAMES_RS = REPO_ROOT / "server" / "src" / "bin" / "generate_brands" / "names.rs"
GENERATE_BRANDS_WRAPPER = REPO_ROOT / "tooling" / "generate-brands"
def _file_sha1(*paths: Path) -> str:
"""Return SHA-1 hex of the concatenated content of one or more files.
Files are sorted by path for determinism. Missing files raise FileNotFoundError
rather than silently contributing an empty-string hash a ghost SHA masks real
breakage (review comment H2: da39a3ee convergence could produce vacuous passes).
"""
h = hashlib.sha1()
for p in sorted(paths):
if not p.exists():
raise FileNotFoundError(f"generator source not found: {p}")
h.update(p.read_bytes())
return h.hexdigest()
# Canonical source set for import_economics' meta stamp. Covers its own .py file
# plus the Rust binary it invokes (generate_brands main.rs + names.rs + wrapper
# script) so any change to the brand generation pipeline flips the stamp. Keep
# this list in sync with GENERATOR_SOURCES["import_economics"] in
# tooling/check-systems-db-stamp.
IMPORT_ECONOMICS_SOURCES: tuple[Path, ...] = (
Path(__file__),
GENERATE_BRANDS_RS,
GENERATE_BRANDS_NAMES_RS,
GENERATE_BRANDS_WRAPPER,
)
def _write_stamp(conn: sqlite3.Connection, generator_name: str, *source_files: Path) -> None:
"""Upsert a row in the meta table recording this generator's current source SHA.
Called after every successful non-dry-run commit. Idempotent: running
twice on the same sources writes the same sha with an updated timestamp.
Only one stamp is written by this module: ``import_economics``, whose source
set includes the Rust binary it invokes (see IMPORT_ECONOMICS_SOURCES).
generate_atlas writes its own stamp. generate_brands does NOT write a stamp
of its own it's a subroutine of import_economics, not an independent DB
writer (PR #136 review T2/H3).
The meta table is created by the MIGRATION_SQL block above; this
function assumes it exists (caller must run migrations first).
"""
schema_sha = _file_sha1(SCHEMA_SQL)
generator_sha = _file_sha1(*source_files)
conn.execute(
"""INSERT OR REPLACE INTO meta (generator_name, schema_version, generator_sha, generated_at)
VALUES (?, ?, ?, datetime('now'))""",
(generator_name, schema_sha, generator_sha),
)
def regenerate_brands() -> None:
"""Run the Rust generate_brands binary to refresh generated_brands.toml.
Invoked as the first step of import_economics' main flow so the TOML on disk
always matches the current Rust source before the Python import reads it.
This replaces the former split (tooling/generate-brands run separately by
make regen-db) with a single, coherent brand pipeline owned by one stamp.
The wrapper script builds the binary on demand and runs it with the default
canonical seed=1; callers that need non-canonical seeds must still invoke
the wrapper directly (experimentation only committed output must be seed=1).
"""
import subprocess
if not GENERATE_BRANDS_WRAPPER.exists():
raise FileNotFoundError(
f"generate_brands wrapper not found at {GENERATE_BRANDS_WRAPPER}"
)
print(" [pre/10] Running generate_brands (Rust) to refresh generated_brands.toml...")
result = subprocess.run(
[str(GENERATE_BRANDS_WRAPPER)],
cwd=str(REPO_ROOT),
capture_output=True,
text=True,
)
if result.returncode != 0:
print(result.stdout, file=sys.stderr)
print(result.stderr, file=sys.stderr)
raise _ImportAborted()
# Print the Rust binary's own summary lines (brands generated, coverage).
# Indent so they fold under the pre-step heading.
for line in result.stdout.splitlines():
if line.strip():
print(f" {line}")
class _ImportAborted(Exception):
@@ -170,6 +266,21 @@ CREATE INDEX IF NOT EXISTS idx_production_chains_output ON production_chains(out
CREATE INDEX IF NOT EXISTS idx_chain_inputs_commodity ON chain_inputs(input_commodity_id);
CREATE INDEX IF NOT EXISTS idx_corp_presence_corp ON corp_presence(corp_id);
CREATE INDEX IF NOT EXISTS idx_corp_presence_location ON corp_presence(location_id);
-- Generator metadata stamp (#855, #856)
CREATE TABLE IF NOT EXISTS meta (
generator_name TEXT PRIMARY KEY,
schema_version TEXT NOT NULL,
generator_sha TEXT NOT NULL,
generated_at TEXT NOT NULL DEFAULT (datetime('now'))
);
-- Drop the pre-merge 'generate_brands' stamp row if it exists (PR #136 review T2/H3).
-- The Rust brand binary is now a subroutine of import_economics its source
-- SHA contributes to the 'import_economics' stamp so it no longer merits its
-- own meta row. This DELETE makes the check-systems-db-stamp "unknown generator"
-- path (fail-closed per T6) compatible with older DBs that still have the row.
DELETE FROM meta WHERE generator_name = 'generate_brands';
"""
# Columns to add to existing tables (ALTER TABLE is idempotent via try/except)
@@ -1010,6 +1121,18 @@ def main():
wiki_corps = load_wiki_corps()
print(f" {len(wiki_corps)} corporation files parsed")
# Regenerate generated_brands.toml via the Rust binary before the Python
# import reads it. Single pipeline, single stamp — resolves review T2/H3
# ("on-behalf stamping" coupling) by folding brand generation into
# import_economics' flow rather than having the caller (Makefile / user)
# remember to run it first. Skipped on --dry-run to avoid a disk
# side-effect during validation.
if not args.dry_run:
try:
regenerate_brands()
except _ImportAborted:
sys.exit(1)
conn = sqlite3.connect(str(db_path))
conn.execute("PRAGMA foreign_keys=ON")
@@ -1136,6 +1259,19 @@ def main():
conn.close()
raise
# Stamp generator metadata (#855, #856): record source SHAs so the
# pre-push hook can detect stale DB snapshots. Written BEFORE the
# coverage gate — the stamp records generator execution (code version),
# not data completeness. Coverage gaps (#860) are pre-existing data
# issues and must not prevent the stamp from landing.
if not args.dry_run:
try:
_write_stamp(conn, "import_economics", *IMPORT_ECONOMICS_SOURCES)
conn.commit()
print(" Stamped: import_economics (covers brand pipeline Rust sources)")
except Exception as exc: # noqa: BLE001
print(f" WARNING: failed to write generator stamp: {exc}", file=sys.stderr)
# Validate coverage (hard errors per D-175, but after commit so data is usable).
print("\n Validating coverage (D-175 Phase 2 gate)...")
coverage_errors: list[str] = []
@@ -1154,7 +1290,7 @@ def main():
print("\n Data committed but Phase 2 gate is NOT met. "
"Add corporations to meet coverage thresholds and re-run.")
conn.close()
sys.exit(1)
sys.exit(2) # exit 2 = coverage warning (data+stamp committed); exit 1 = real error
else:
print(" All coverage thresholds met — Phase 2 gate PASSED.")
+67 -1
View File
@@ -88,6 +88,48 @@ _ATLAS_SCHEMA_BEGIN_MARKER = "-- BEGIN ATLAS INDEX"
_ATLAS_SCHEMA_END_MARKER = "-- END ATLAS INDEX"
# ---------------------------------------------------------------------------
# Generator metadata stamp (#855, #856)
# ---------------------------------------------------------------------------
def _file_sha1(*paths: Path) -> str:
"""Return SHA-1 hex of the concatenated content of one or more files.
Files are sorted by path for determinism. Missing files raise
FileNotFoundError rather than silently skip a ghost hash (empty-bytes
digest) can mask real breakage when stored and current SHAs converge
(#136 review H2).
"""
h = hashlib.sha1()
for p in sorted(paths):
if not p.exists():
raise FileNotFoundError(f"generator source not found: {p}")
h.update(p.read_bytes())
return h.hexdigest()
def _write_stamp(conn: sqlite3.Connection) -> None:
"""Upsert a meta row for generate_atlas after a successful run.
Idempotent: running twice on the same source files writes the same SHA
with an updated timestamp. The meta table is created by the atlas schema
migration executed in ensure_atlas_schema(); this function assumes it
exists.
Transaction ownership stays with the caller (matches the import_economics
pattern) no inner commit here. Review H1 flagged the prior behaviour as
a double-commit with the atlas data write that precedes it.
"""
schema_sha = _file_sha1(SYSTEMS_SCHEMA_PATH)
generator_sha = _file_sha1(Path(__file__))
conn.execute(
"""INSERT OR REPLACE INTO meta
(generator_name, schema_version, generator_sha, generated_at)
VALUES ('generate_atlas', ?, ?, datetime('now'))""",
(schema_sha, generator_sha),
)
def _load_atlas_schema() -> str:
"""Return the atlas_* DDL block from systems-schema.sql.
@@ -121,8 +163,20 @@ def ensure_atlas_schema(conn: sqlite3.Connection) -> None:
Idempotent: all statements inside the block use CREATE TABLE / INDEX
IF NOT EXISTS, so running this on an already-migrated DB is a no-op.
Also ensures the meta stamp table exists (#855, #856).
"""
conn.executescript(_load_atlas_schema())
conn.executescript(
"""
CREATE TABLE IF NOT EXISTS meta (
generator_name TEXT PRIMARY KEY,
schema_version TEXT NOT NULL,
generator_sha TEXT NOT NULL,
generated_at TEXT NOT NULL DEFAULT (datetime('now'))
);
"""
)
def _first_int(values, default: int = 0) -> int:
@@ -1333,7 +1387,19 @@ def main():
print(f" [{i+1}/{len(bodies)}] {body_id:20s} ERROR: {result.get('message', '')}")
if not args.dry_run:
conn.commit()
# Stamp generator metadata (#855, #856) together with the atlas data
# in a single commit — atlas data + stamp land atomically, and the
# stamp function itself no longer commits (H1). Failure of the stamp
# write rolls back the atlas data too rather than leaving a stamped-
# but-missing-data intermediate state.
try:
_write_stamp(conn)
conn.commit()
print(" Stamped: generate_atlas")
except Exception as exc: # noqa: BLE001
conn.rollback()
print(f" WARNING: failed to write generator stamp: {exc}", file=sys.stderr)
print(" Atlas data NOT committed — regen required.", file=sys.stderr)
conn.close()
elapsed_total = time.time() - t_total