docs(governance): D-262 — the wiki generator flow has one canonical map

The relationship between wiki/, the generators, systems.db and the runtime is
a directed graph with two edges running opposite to the obvious direction and
one running backwards into its own producer. Prose renders that badly: every
document that has described it states a single ownership direction and is
therefore wrong about part of the tree. D-262 makes the diagram the source of
truth and points CLAUDE.md, Skill(wiki), project-structure.md and
wiki/GOVERNANCE.md at it.

The correction that matters most: body pages were described everywhere as
machine-owned and reverted on sync. They are not. scaffold_bodies.py writes
one once and never overwrites it, and import_economics then reads that
frontmatter directly as input — so a hand-edit is not reverted, it is obeyed,
and silently changes world generation. Worse than being overwritten, and the
actual reason GOVERNANCE.md forbids the edit.

New: tooling/check-dataflow-graph.py, wired into the Makefile and the pre-push
hook. It asserts every repo path named in a hand-authored diagram still
resolves — and its docstring states plainly what it cannot do: verify that an
edge still MEANS what it says. If wiki_sync.py stopped writing body pages
tomorrow, every path would still exist and the check would still pass. Edge
semantics stay a human check against the tool's source, so nobody reads a green
gate as a verified map.

Verified by breaking it: pointing one label at a moved path fails with exit 1
naming that path; restoring it passes. Building the checker also caught two
real vaguenesses in the diagram — "GJ-*/index.md" and "bodies/{id}/index.md"
were written without their wiki/star-systems/ prefix, which is precisely the
ambiguity this map exists to remove. Generated star-map .d2 files are excluded
by name; their correctness belongs to their generator under D-223.

Also files Q-124 + T-1246 (tooling): whether the 123 Python files under
tooling/ should become one Rust CLI of pql's calibre. The friction is real and
mostly not about the language — the permission gate prefix-matches whole
command strings and a blanket Bash(python3 *) grant is forbidden, so each tool
prompts near-individually, while a single binary is one allowlist entry. The
record requires pricing the cheap alternative (a Python dispatcher entrypoint)
before recommending Rust, and flags the hard constraint: import_economics is
stamped by source SHA, so any port must keep that contract intact through the
transition rather than disabled during it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-20 01:56:12 +02:00
co-authored by Claude Opus 5
parent 0817befcba
commit 6949f800dc
13 changed files with 324 additions and 91 deletions
+5
View File
@@ -1,5 +1,10 @@
# Project Structure (detailed) # Project Structure (detailed)
For how `wiki/`, the generators, `systems.db` and the runtime relate — which
way the arrows point for any given file — the canonical map is the diagram
`docs/diagrams/data-flow/wiki-generator-flow.d2` (**D-262**), not the tree
below. The tree says where things live; the diagram says who writes them.
``` ```
wiki/ # THE SEED (11,864 files, the largest tree here). Authored wiki/ # THE SEED (11,864 files, the largest tree here). Authored
# world content the generator varies from — NOT a # world content the generator varies from — NOT a
+6
View File
@@ -23,6 +23,12 @@ should read its own docs for detail:
Two ownership models, running OPPOSITE ways. Get this wrong and your work Two ownership models, running OPPOSITE ways. Get this wrong and your work
vanishes on the next sync, without an error. vanishes on the next sync, without an error.
**The canonical answer is a diagram, not this text (D-262):**
`docs/diagrams/data-flow/wiki-generator-flow.d2` — view it with
`clide draw --file docs/diagrams/data-flow/wiki-generator-flow.d2`. Every edge
in it was read in the tool's own source. What follows here is a summary; where
the two disagree, the diagram is right.
### `star-systems/` — DB owns structure, wiki owns prose (OUTPUT) ### `star-systems/` — DB owns structure, wiki owns prose (OUTPUT)
3,262 pages, 93% of the wiki. Generated FROM `server/data/systems.db` and 3,262 pages, 93% of the wiki. Generated FROM `server/data/systems.db` and
+14
View File
@@ -316,6 +316,20 @@ if [ -f "$REPO_ROOT/tooling/check-canvas-version" ]; then
fi fi
fi fi
# --- Data-flow diagram paths (D-262) ---
# A diagram that names files goes stale SILENTLY — nothing fails when a path
# moves, so the map keeps asserting a layout that is no longer true. This checks
# only that the paths still resolve; whether an EDGE still means what it says is
# a human check against the tool's source. Cheap (no subprocess beyond python).
if [ -f "$REPO_ROOT/tooling/check-dataflow-graph.py" ]; then
echo "pre-push: checking data-flow diagram paths..."
if python3 "$REPO_ROOT/tooling/check-dataflow-graph.py"; then
:
else
fail_check "a path named in a data-flow diagram no longer resolves"
fi
fi
# --- Clerk review (D-221) --- # --- Clerk review (D-221) ---
# DISABLED 2026-05-23 (#965) pending rework. Two problems made it net-negative: # DISABLED 2026-05-23 (#965) pending rework. Two problems made it net-negative:
# 1. Non-exhaustive — a single run reports ~the first contradiction it finds # 1. Non-exhaustive — a single run reports ~the first contradiction it finds
+1
View File
@@ -5,3 +5,4 @@ INSERT INTO ticket_idmap (record_id, ticket_id, created_at, updated_at, deleted_
INSERT INTO ticket_idmap (record_id, ticket_id, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G0P0F9C3RKKK23ZHHTANCPZW', 'T-1243', '2026-08-16 14:37:49.794', '2026-08-16 14:37:49.794', NULL, '0dbba61f1c75851f6e0cca78402da745', 2) ON CONFLICT(record_id) DO UPDATE SET ticket_id=excluded.ticket_id, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= ticket_idmap.updated_at; INSERT INTO ticket_idmap (record_id, ticket_id, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G0P0F9C3RKKK23ZHHTANCPZW', 'T-1243', '2026-08-16 14:37:49.794', '2026-08-16 14:37:49.794', NULL, '0dbba61f1c75851f6e0cca78402da745', 2) ON CONFLICT(record_id) DO UPDATE SET ticket_id=excluded.ticket_id, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= ticket_idmap.updated_at;
INSERT INTO ticket_idmap (record_id, ticket_id, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G1R8WZ987YCP1ARANSA70R0W', 'T-1244', '2026-08-19 22:28:11.726', '2026-08-19 22:28:11.726', NULL, 'fd1f9d38036692125ba7c7de0198bfc8', 2) ON CONFLICT(record_id) DO UPDATE SET ticket_id=excluded.ticket_id, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= ticket_idmap.updated_at; INSERT INTO ticket_idmap (record_id, ticket_id, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G1R8WZ987YCP1ARANSA70R0W', 'T-1244', '2026-08-19 22:28:11.726', '2026-08-19 22:28:11.726', NULL, 'fd1f9d38036692125ba7c7de0198bfc8', 2) ON CONFLICT(record_id) DO UPDATE SET ticket_id=excluded.ticket_id, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= ticket_idmap.updated_at;
INSERT INTO ticket_idmap (record_id, ticket_id, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G1RB7GSJA4379EAQDZPQQDN8', 'T-1245', '2026-08-19 22:38:22.413', '2026-08-19 22:38:22.413', NULL, '34a0757e14a868c4d9227c372c013e90', 2) ON CONFLICT(record_id) DO UPDATE SET ticket_id=excluded.ticket_id, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= ticket_idmap.updated_at; INSERT INTO ticket_idmap (record_id, ticket_id, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G1RB7GSJA4379EAQDZPQQDN8', 'T-1245', '2026-08-19 22:38:22.413', '2026-08-19 22:38:22.413', NULL, '34a0757e14a868c4d9227c372c013e90', 2) ON CONFLICT(record_id) DO UPDATE SET ticket_id=excluded.ticket_id, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= ticket_idmap.updated_at;
INSERT INTO ticket_idmap (record_id, ticket_id, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G1RWR7M1H1PC3TY2RT7AC174', 'T-1246', '2026-08-19 23:54:55.776', '2026-08-19 23:54:55.776', NULL, 'f3663ab4236d777635d78a5cf33e65e1', 2) ON CONFLICT(record_id) DO UPDATE SET ticket_id=excluded.ticket_id, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= ticket_idmap.updated_at;
+9
View File
@@ -1176,3 +1176,12 @@ the same root cause — measuring VALUES without reading the CODE that produced
them. First correction: tectonics is derived. Second: the whole continuous/ them. First correction: tectonics is derived. Second: the whole continuous/
categorical split is an artefact of the resolution tiers. Both were available in categorical split is an artefact of the resolution tiers. Both were available in
a docstring the original pass grepped but never read.', 'backlog', 'medium', NULL, 'server', NULL, '2026-08-19 22:28:11.722', '2026-08-19 22:52:46.636', NULL, '65db5461d46f8f84aed68526113025a6', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at; a docstring the original pass grepped but never read.', 'backlog', 'medium', NULL, 'server', NULL, '2026-08-19 22:28:11.722', '2026-08-19 22:52:46.636', NULL, '65db5461d46f8f84aed68526113025a6', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G1RWR7M1H1PC3TY2RT7AC174', 'story', '06FBPPMZNNEV052DBYYY3A897C', 'Investigate retooling Python tooling into a Rust CLI (Q-124)', 'Answer Q-124. Investigation and costing FIRST — do not open with a rewrite.
Price the cheap alternative before recommending Rust: a single Python dispatcher entrypoint (one command, many subcommands) would resolve the permission-prompt friction and most of the startup cost without touching the numerics. If that is enough, say so and close Q-124 with it.
Scope order if Rust is chosen: port the check/gate family first (check-client-version, check-canvas-version, check-systems-db-stamp, check-dataflow-graph) — small, pure, no numerics, highest prompt-friction, runs on every push. Leave planet-gen numerics (numpy/scipy/PIL) in Python behind a stable interface.
Hard constraint: import_economics is stamped by source SHA via tooling/generator_sources.py. Any port must keep the stamp/pre-push contract intact through the transition, not disabled during it.
Outcome: a recommendation with costs, then a D-record and Jeroen''s ruling. Epics only after that.', 'backlog', 'low', NULL, 'tooling', 'Q-124', '2026-08-19 23:54:55.776', '2026-08-19 23:54:55.776', NULL, 'cfb08832c72e06f85b075a46cd3223f2', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
+6 -1
View File
@@ -12,7 +12,7 @@ GODOT := $(shell command -v godot4 2>/dev/null || command -v godot 2>/dev/null)
perf-baseline debug-schedule \ perf-baseline debug-schedule \
test-ipc-fixtures test-ipc-protocol test-ipc-integration test-ipc-benchmark \ test-ipc-fixtures test-ipc-protocol test-ipc-integration test-ipc-benchmark \
screenshot visual-movie test-visual visual-update \ screenshot visual-movie test-visual visual-update \
manifest diagrams check-diagrams manifest diagrams check-diagrams check-dataflow-graph
# --- Configuration --- # --- Configuration ---
@@ -561,6 +561,11 @@ check-diagrams:
test $$fail -eq 0 || { echo "Run 'make diagrams'."; exit 1; }; \ test $$fail -eq 0 || { echo "Run 'make diagrams'."; exit 1; }; \
echo "All diagrams rendered." echo "All diagrams rendered."
# D-262: every repo path named in a hand-authored data-flow diagram must still
# resolve. Catches the stale half of a diagram; edge MEANING is a human check.
check-dataflow-graph:
@python3 tooling/check-dataflow-graph.py
# --- Clean --- # --- Clean ---
clean: clean:
@@ -59,8 +59,8 @@ db: "systems.db" {class: store}
smap: "star-map.json" {class: store} smap: "star-map.json" {class: store}
# ---- generated into wiki/ (grey): do not hand-edit ---- # ---- generated into wiki/ (grey): do not hand-edit ----
sys: "GJ-*/index.md\nREAD-ONLY blocks" {class: generated} sys: "wiki/star-systems/GJ-*/index.md\nREAD-ONLY blocks" {class: generated}
body: "bodies/{id}/index.md\nfrontmatter = body definition" {class: generated} body: "wiki/star-systems/GJ-*/bodies/{id}/index.md\nfrontmatter = body definition" {class: generated}
assets: "heightmap · reliefmap · globe\nterrain.npz · markers.json" {class: generated} assets: "heightmap · reliefmap · globe\nterrain.npz · markers.json" {class: generated}
brands: "generated_brands.toml" {class: generated} brands: "generated_brands.toml" {class: generated}
File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 44 KiB

After

Width:  |  Height:  |  Size: 44 KiB

+2
View File
@@ -327,6 +327,7 @@ line in place — keep the Q-record for the audit trail rather than deleting it.
- [D-259: Enclosed settlements — open-air requires a breathable atmosphere](decisions/architecture.md#d-259-enclosed-settlements--open-air-requires-a-breathable-atmosphere) — _architecture_ - [D-259: Enclosed settlements — open-air requires a breathable atmosphere](decisions/architecture.md#d-259-enclosed-settlements--open-air-requires-a-breathable-atmosphere) — _architecture_
- [D-260: Generator scope — Sol is authored, not generated](decisions/architecture.md#d-260-generator-scope--sol-is-authored-not-generated) — _architecture_ - [D-260: Generator scope — Sol is authored, not generated](decisions/architecture.md#d-260-generator-scope--sol-is-authored-not-generated) — _architecture_
- [D-261: River rendering — a stroke, not a scaled feature](decisions/architecture.md#d-261-river-rendering--a-stroke-not-a-scaled-feature) — _architecture_ - [D-261: River rendering — a stroke, not a scaled feature](decisions/architecture.md#d-261-river-rendering--a-stroke-not-a-scaled-feature) — _architecture_
- [D-262: The wiki↔generator data flow has one canonical map, and it is a diagram](decisions/architecture.md#d-262-the-wikigenerator-data-flow-has-one-canonical-map-and-it-is-a-diagram) — _architecture_
## Open questions ## Open questions
@@ -411,6 +412,7 @@ line in place — keep the Q-record for the audit trail rather than deleting it.
- [Q-121: Where does cultural dress canon live — TOML registers now, prose pages in Phase 6?](questions/content.md#q-121-where-does-cultural-dress-canon-live--toml-registers-now-prose-pages-in-phase-6) — _content_ - [Q-121: Where does cultural dress canon live — TOML registers now, prose pages in Phase 6?](questions/content.md#q-121-where-does-cultural-dress-canon-live--toml-registers-now-prose-pages-in-phase-6) — _content_
- [Q-122: Third-party asset sourcing policy — when to use CC0/purchased bases vs Trellis generation, per asset class?](questions/content.md#q-122-third-party-asset-sourcing-policy--when-to-use-cc0purchased-bases-vs-trellis-generation-per-asset-class) — _content_ - [Q-122: Third-party asset sourcing policy — when to use CC0/purchased bases vs Trellis generation, per asset class?](questions/content.md#q-122-third-party-asset-sourcing-policy--when-to-use-cc0purchased-bases-vs-trellis-generation-per-asset-class) — _content_
- [Q-123: Believability gate threshold calibration](questions/architecture.md#q-123-believability-gate-threshold-calibration) — _architecture_ - [Q-123: Believability gate threshold calibration](questions/architecture.md#q-123-believability-gate-threshold-calibration) — _architecture_
- [Q-124: Should the Python tooling be retooled into a single Rust CLI?](questions/architecture.md#q-124-should-the-python-tooling-be-retooled-into-a-single-rust-cli) — _architecture_
## Resolved questions ## Resolved questions
+18 -1
View File
@@ -2514,6 +2514,23 @@ Technical foundation decisions that constrain implementation: engine, client-ser
- **Cross-reference:** [D-255](#d-255) (stepped ladder — premise (1) server-determines-content / premise (2) client-is-a-map-art-function, the split this record applies; and the 2026-07-26 extent inversion that made screen-pixel reasoning coherent per rung), [D-258](#d-258) (rung 0.5 — its amendment records the 375-course / 458-pixel measurement this record is built on), [D-227](#d-227) (derive-don't-store — culling is presentation, not derivation, so it is outside that rule). - **Cross-reference:** [D-255](#d-255) (stepped ladder — premise (1) server-determines-content / premise (2) client-is-a-map-art-function, the split this record applies; and the 2026-07-26 extent inversion that made screen-pixel reasoning coherent per rung), [D-258](#d-258) (rung 0.5 — its amendment records the 375-course / 458-pixel measurement this record is built on), [D-227](#d-227) (derive-don't-store — culling is presentation, not derivation, so it is outside that rule).
- **Dissent:** None. - **Dissent:** None.
### D-262: The wiki↔generator data flow has one canonical map, and it is a diagram
- **Date:** 2026-08-20
- **Decision:** `docs/diagrams/data-flow/wiki-generator-flow.d2` (rendered to `.svg` beside it) is the **source of truth for how `wiki/`, the generators, `systems.db` and the runtime relate**. It answers one question — *for any file under `wiki/`, which way do the arrows point?* — and that question has no other home. Prose descriptions of the flow in `CLAUDE.md`, `Skill(wiki)`, `.claude/rules/project-structure.md` and `wiki/GOVERNANCE.md` remain, but each now points here, and where any of them disagrees with the diagram, **the diagram is right and the prose is a stale copy**.
- **Why a diagram and not prose.** The relationship is a directed graph with two edges running *opposite* to the obvious direction and one running *backwards into its own producer*. Prose renders that badly: every document that has tried states a single ownership direction and is therefore wrong about part of the tree. The graph is the natural shape of the data, so it is the honest representation.
- **What the diagram records that nothing else does:**
- **Two paths run BOTH ways.** `wiki_sync.py` renders DB→wiki (`generate_wiki()`, routine) *and* parses wiki→DB (`import_from_wiki()`, bootstrap/migration only). Neither direction alone is "the" direction.
- **The trap.** `scaffold_bodies.py` writes a body page **once and never overwrites it**; `import_economics` then reads that frontmatter directly as input. So a hand-edited body page is **not reverted — it is obeyed**, and silently changes world generation. This is strictly worse than being overwritten, and it is the real reason `wiki/GOVERNANCE.md` forbids the edit. Every document that described body pages as "machine-owned, reverted on sync" was wrong in the direction that matters.
- **Two routes into the game.** `systems.db` ships with the build; the per-body terrain PNGs are read at **runtime** via `bodies.terrain_reference`, never baked in.
- **Provenance is part of the artefact.** Every edge was read in the tool's own source, not inferred from filenames or grep. The header comment says so. A diagram whose provenance is unstated is a guess with good typography, and this one exists because an earlier statistical scan of the same tree produced confident, wrong conclusions.
- **Enforcement — and its explicit limit.** `tooling/check-dataflow-graph.py` (`make check-dataflow-graph`, run by the pre-push hook) asserts that **every repo path named in the diagram still resolves**. That is the cheap half. It **cannot** verify that an edge still *means* what it says: if `wiki_sync.py` stopped writing body pages tomorrow, every path would still exist and the check would still pass. **Edge semantics are a human check against the tool's source**, and this limit is written into the script's docstring so nobody mistakes a green gate for a verified map. The failure mode being prevented is silence — a path moves, nothing fails, and the map keeps asserting a layout that is no longer true.
- **Scope: hand-authored diagrams only.** `docs/diagrams/design/star-map-*.d2` are **generator output** (`tooling/generate-star-map.py`), so their correctness is the generator's problem under the same source-canonical rule as [D-223](#d-223)/`asset-pipeline.md`, not this checker's. They are excluded by name.
- **Consequence for format.** Diagrams are now `.d2` + `.svg`, not `.png` — d2 emits SVG natively while its PNG path needs a ~150 MB headless-Chromium download that this system does not have. `make diagrams` renders, `make check-diagrams` gates. Note the trap this exposed: a d2 `|md` block compiles to an SVG `<foreignObject>`, which ImageMagick and `flutter_svg` both silently drop — text that exists in the file and is invisible in every viewer except a browser. Legends must be plain labels. Recorded in `.claude/rules/diagrams.md`.
- **Raised by:** Jeroen, 2026-08-20: *"add this to the docs and relevant d records that this is where the source of truth for this component lies (this may be dangerous since it may be weirdly shaped data so we need some sort of graph solution you can use)"* — the "weirdly shaped data" instinct was correct, and is why this is a graph rather than another table.
- **Cross-reference:** [D-223](#d-223) (source-canonical asset pipeline — the rule the generated star-maps fall under instead), `Skill(wiki)`, `.claude/rules/diagrams.md`, `wiki/GOVERNANCE.md`.
- **Dissent:** None.
--- ---
*115 decisions (D-001 through D-261, excluding gaps). Last updated: 2026-07-28 (D-261river rendering: a fixed 5 px screen-space stroke through cell centres, never over water, culled below 15 px of on-screen length so the km threshold derives per rung from the stroke itself; client-side because the cull is measured in pixels the server cannot know).* *116 decisions (D-001 through D-262, excluding gaps). Last updated: 2026-08-20 (D-262the wiki↔generator data flow has one canonical map, `docs/diagrams/data-flow/wiki-generator-flow.d2`; a path checker runs on push, but edge MEANING stays a human check against the tool's source).*
+19 -1
View File
@@ -491,6 +491,24 @@ Technical foundation questions: engine, protocols, data structures, performance,
7. **Pass budget + ratchet** — the % of land probes that must pass to ship (budgeted), and the trigger to ratchet advisory → strict (hard push-gate block). 7. **Pass budget + ratchet** — the % of land probes that must pass to ship (budgeted), and the trigger to ratchet advisory → strict (hard push-gate block).
- **Cross-reference:** [D-245](../decisions/architecture.md#d-245-nature-layer-believability-acceptance-gate) (the gate this calibrates), [D-239](../decisions/architecture.md#d-239) §8/§10 (laws + budget), [D-227](../decisions/architecture.md#d-227), [D-243](../decisions/architecture.md#d-243); tickets T-1083 (enforcer — produces the baselines), T-1079 (epic), T-1080/T-1081/T-1082/T-1084. - **Cross-reference:** [D-245](../decisions/architecture.md#d-245-nature-layer-believability-acceptance-gate) (the gate this calibrates), [D-239](../decisions/architecture.md#d-239) §8/§10 (laws + budget), [D-227](../decisions/architecture.md#d-227), [D-243](../decisions/architecture.md#d-243); tickets T-1083 (enforcer — produces the baselines), T-1079 (epic), T-1080/T-1081/T-1082/T-1084.
### Q-124: Should the Python tooling be retooled into a single Rust CLI?
- **Date:** 2026-08-20
- **Status:** Open
- **Question:** `tooling/` is **123 Python files** plus 36 extensionless executables. Should it become one Rust CLI of the same calibre as `pql` — a single binary, subcommand-structured, no interpreter and no virtualenv — or should it stay Python and have its friction fixed in place?
- **The friction is concrete, recurring, and mostly not about Python the language:**
1. **Permission prompts.** Every invocation is `python3 tooling/<script>`, and the permission gate prefix-matches whole command strings. A blanket `Bash(python3 *)` grant is explicitly forbidden as "an unbounded write grant" (`docs/agent-operation.md`), so each tool prompts more or less individually. A single binary with subcommands (`pql`-style) is one allowlist entry covering the whole surface — this is the same reason `pql` is frictionless today.
2. **The venv.** 5 tooling modules resolve a virtualenv explicitly (`tooling/db/common.py` and friends), `make setup-venv` exists to create it, and non-interactive shells — agents, git hooks — do not source `/etc/profile.d`, so interpreter and dependency resolution differ between a human shell and a hook. A static binary has no such split.
3. **Startup cost in the push gate.** Several checks now run per push (`check-client-version`, `check-canvas-version`, `check-systems-db-stamp`, `check-dataflow-graph`), each paying interpreter start plus imports.
- **What must NOT be lost, and is the real argument against:**
- `import_economics` is the sole generator of `systems.db` and is **stamped by source SHA** (`tooling/generator_sources.py`); a rewrite changes every SHA and must be sequenced so the stamp/pre-push contract survives the transition rather than being disabled through it.
- The planet-gen path (`import_heightmaps`, `planet_simulation`) leans on numpy/scipy/PIL. Porting that is a genuine numerical-equivalence problem, not a transliteration — heightmaps are committed build artefacts and goldens depend on them.
- Python is where one-off analysis gets written. A Rust CLI must not make ad-hoc investigation expensive, or it trades a small recurring cost for a large occasional one.
- **Therefore the likely shape, if pursued:** not a big-bang rewrite. Port the **check/gate family first** (small, pure, no numerics, highest prompt-friction, most push-frequent), leave the numerics in Python behind a stable interface, and judge from there. A partial answer is a legitimate outcome of this question.
- **Also worth asking:** whether the fix is a CLI at all, or a shared `tooling/` entrypoint dispatcher in Python — one command, many subcommands — which would resolve friction (1) and most of (3) at a fraction of the cost, without touching the numerics. **This is the cheap alternative the investigation must price before recommending Rust**, or the record will have argued for the expensive option by never costing the other one.
- **Raised by:** Jeroen, 2026-08-20: *"those python files are a pain. they throw permission prompts all over the place and the venv dependencies are annoying... We should ticket a Q record to retool that to rust probably. maybe make it into an actual cli of the quality level of pql. one can dream"*
- **Cross-reference:** [D-223](../decisions/architecture.md#d-223) and `.claude/rules/asset-pipeline.md` (the stamp contract a rewrite must preserve), [D-262](../decisions/architecture.md#d-262) (`check-dataflow-graph`, the most recent addition to the per-push Python set), `.claude/rules/ticket-cli.md` (`pql` as the quality bar being referenced).
--- ---
*65 questions (13 resolved, 1 partially resolved, 51 open). Last updated: 2026-06-28 (Q-123believability gate threshold calibration, alongside D-245).* *66 questions (13 resolved, 1 partially resolved, 52 open). Last updated: 2026-08-20 (Q-124retool the 123-file Python tooling into a Rust CLI, or fix the friction in place with a single dispatcher).*
+149
View File
@@ -0,0 +1,149 @@
#!/usr/bin/env python3
"""Assert that every repo path named in a hand-authored data-flow diagram exists.
A diagram that names files goes stale SILENTLY nothing fails when a path
moves, so the map keeps asserting a layout that is no longer true. This closes
the cheap half of that gap.
WHAT IT CAN CHECK: that each path mentioned in a `.d2` node label still resolves
on disk (allowing `*`/`{...}` globs and `·`-separated lists).
WHAT IT CANNOT CHECK: whether an EDGE still means what it says. If
`wiki_sync.py` stops writing body pages tomorrow, every path here still exists
and this script still passes. Edge semantics are verified by reading the tool's
source, which is a human job see D-262.
Generated `.d2` files are skipped: their correctness is the generator's problem
(the same source-canonical rule as .claude/rules/asset-pipeline.md), and they
name star-system ids rather than repo paths.
Usage:
python3 tooling/check-dataflow-graph.py [--verbose]
"""
from __future__ import annotations
import argparse
import re
import sys
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parent.parent
DIAGRAM_ROOT = REPO_ROOT / "docs" / "diagrams"
# Hand-authored diagrams whose labels name real repo paths. Add a diagram here
# when it starts naming files; a diagram absent from this list is not checked.
CHECKED_DIAGRAMS = [
"data-flow/wiki-generator-flow.d2",
]
# Generated sources — skipped even if listed above. See .claude/rules/diagrams.md.
GENERATED_PREFIXES = ("design/star-map-",)
# A token looks like a path if it contains a slash and a plausible name char.
# Node labels in these diagrams carry paths such as:
# "wiki/economics/\nTOMLs + 37 commodity pages"
# "heightmap · reliefmap · globe\nterrain.npz · markers.json"
# "tooling/atlas add-body\nTHE bodies catalog origin"
PATH_TOKEN = re.compile(r"[A-Za-z0-9_.\-*{}/]*/[A-Za-z0-9_.\-*{}/]+")
# A token only counts as a path if its first segment is a real top-level entry
# in the repo. Without this, legend prose such as "dashed one-time or
# bootstrap" yields the token `one-time/bootstrap` and fails the check.
TOP_LEVEL = {p.name for p in REPO_ROOT.iterdir()}
def label_strings(d2_text: str) -> list[str]:
"""Every double-quoted label in the d2 source."""
return re.findall(r'"((?:[^"\\]|\\.)*)"', d2_text)
def candidate_paths(label: str) -> list[str]:
"""Extract path-looking tokens from one label."""
# Labels use \n for line breaks and · to separate sibling files.
flat = label.replace("\\n", " ").replace("·", " ")
out = []
for tok in PATH_TOKEN.findall(flat):
tok = tok.strip(".,;:")
if not tok or tok.split("/", 1)[0] not in TOP_LEVEL:
continue
out.append(tok)
return out
def resolves(token: str) -> bool:
"""True if the token resolves on disk, treating * and {..} as wildcards."""
direct = REPO_ROOT / token
if direct.exists():
return True
# `bodies/{id}/index.md` -> `bodies/*/index.md`; then glob it.
pattern = re.sub(r"\{[^}]*\}", "*", token)
if "*" in pattern:
try:
return any(REPO_ROOT.glob(pattern))
except (ValueError, OSError):
return False
# A bare filename inside a directory that was named elsewhere in the
# diagram (e.g. `terrain.npz` under a body dir) — search narrowly.
return False
def check(diagram: str, verbose: bool) -> list[str]:
path = DIAGRAM_ROOT / diagram
if not path.exists():
return [f"{diagram}: diagram not found"]
failures = []
checked = 0
for label in label_strings(path.read_text(encoding="utf-8")):
for token in candidate_paths(label):
checked += 1
if resolves(token):
if verbose:
print(f" ok {token}")
else:
failures.append(f"{diagram}: path does not resolve: {token}")
if checked == 0:
# A diagram listed for checking that yields no paths means the label
# format changed and this script silently stopped checking anything.
failures.append(
f"{diagram}: no path-like tokens found — the checker is not "
f"actually checking this diagram"
)
elif verbose:
print(f" {checked} path tokens checked in {diagram}")
return failures
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--verbose", action="store_true")
args = ap.parse_args()
failures: list[str] = []
for diagram in CHECKED_DIAGRAMS:
if diagram.startswith(GENERATED_PREFIXES):
continue
failures.extend(check(diagram, args.verbose))
if failures:
print("check-dataflow-graph: FAILED", file=sys.stderr)
for f in failures:
print(f" {f}", file=sys.stderr)
print(
"\nA path named in a diagram no longer exists. Either the path "
"moved (update the diagram) or the diagram was always wrong.",
file=sys.stderr,
)
return 1
print(f"check-dataflow-graph: OK — {len(CHECKED_DIAGRAMS)} diagram(s)")
return 0
if __name__ == "__main__":
sys.exit(main())
+7
View File
@@ -7,6 +7,13 @@
## Two Ownership Models ## Two Ownership Models
> **The canonical map of this is a diagram (D-262):**
> `docs/diagrams/data-flow/wiki-generator-flow.d2`. Every edge in it was read in
> the generator's own source. The two models below are a summary; the diagram
> also records the parts that do not fit a two-way split — `wiki_sync.py` runs
> in **both** directions, and body pages are read back into the generator as
> input. Where this text and the diagram disagree, the diagram is right.
The wiki has two kinds of content with different sources of truth (see The wiki has two kinds of content with different sources of truth (see
`tooling/db/wiki_sync.py`): `tooling/db/wiki_sync.py`):