diff --git a/.claude/skills/reach/SKILL.md b/.claude/skills/reach/SKILL.md new file mode 100644 index 000000000..602378581 --- /dev/null +++ b/.claude/skills/reach/SKILL.md @@ -0,0 +1,155 @@ +--- +name: reach +description: Use and extend `reach`, the one CLI for all repo tooling in settled-reach (D-263). Use when running any repo tool (gates, generators, the atlas, the ledger import, asset connectors, Blender payloads), when asked "is there a tool for X", or when adding or changing tooling — a new verb, a new domain, a port. Also use before writing ANY script under tooling/: new tooling is a reach verb, never a loose script. +--- + +# reach — the one door to repo tooling + +Everything under `tooling/` is one Python package behind one command, `reach` +(D-263). It is on PATH as a bare name everywhere — shells, hooks, agents — +via `make install-reach` (uv tool, editable: edit `tooling/`, run `reach`, no +reinstall). **Its primary user is an agent**, so help is the index, errors name +their fix, and output volume is treated as a cost. + +## Using it + +**Discover, don't guess.** `reach --help` lists the domains without importing +any of them; `reach --help` lists verbs; `reach --help` +shows options. An unknown domain or choice fails with the accepted set. + +| domain | what it is | +|---|---| +| `check` | the consistency gates the push hook runs (canvas-version, client-version, systems-db-stamp, fact-ids, dataflow-graph) | +| `validate` | content/checklist YAML against schemas, RON against Rust structs | +| `atlas` | the spatial ladder: `db ` fronts the Rust atlas binary; proposal workflow verbs; nested `atlas map` (Reach map data + SVG) and `atlas planet` (body generation) | +| `ledger` | `ledger import` — the sole generator of systems.db (`make regen-db` delegates here) | +| `wiki` | wiki fill rates, the GTTR hook | +| `assets` | Stable Audio / Gemini / Trellis connectors, `synth-ui` | +| `character` | brand logos, GLB utility-node strip, garment QA | +| `blender` | run a payload under Blender (`list`, `which`, `run`) | +| `godot`, `visual` | parse sweeps; capture diff/thumbnail/blank-check | +| `generate` | brands, corporations, the character manifest | +| `pr`, `dev`, `jobs` | review loop; environment + perf + clerk; detached runs | + +**Global options go BEFORE the domain:** `reach --verbose …` (debug events + +tracebacks), `reach --no-input …` (hooks and agents: never prompt), +`reach --detach …` (run in the background, print a job id; follow with +`reach jobs log --follow`, `reach jobs wait `). Anything that runs for +minutes — `ledger import`, `atlas planet batch`, `assets trellis batch` — wants +`--detach` from an agent, because a foreground Bash call gives up at two +minutes and sends nothing. + +**Reading the result.** stdout is the command's DATA (often JSON — pipe it); +stderr is the event stream (JSONL when piped, rendered on a TTY) ending in one +**verdict**. The exit status is the answer: 0 is success, and **every non-zero +exit carries a `fix`** — the command that resolves it. Read the status, never a +field like `"ok"`. A handful of verbs keep meaningful non-1 codes (e.g. +`ledger import` exits 2 for "imported and stamped, coverage gate unmet"; +`character qa` 2/3 for missing config/Godot). + +## Where things are + +``` +tooling/main.py routing only — the DOMAINS registry (lazy) +tooling/core/ cli, command (@command), console, errors (ReachError), + process (the one guarded exec), config, jobs, runtime +tooling/domains// router.py = transport; everything else = logic +tooling/scripts/blender/ Blender payloads — run by Blender's Python, NEVER imported +tooling/archive/ provenance only; README says why each must not run +tooling/test_*.py gate tests, run by `make test-tooling` +tooling/DOMAINS.md the domain map and every judgment call behind it +``` + +## Adding tooling — the rules a reviewer checks + +**There is no other place to add tooling** (CLAUDE.md). A new tool is a verb on +an existing domain, or a new domain. Not a loose script, not an extensionless +executable, not a `python3 tooling/.py` entrypoint, not a make target that +holds logic. The only exceptions: Blender payloads (they physically cannot +import the package) and the Rust crates. + +`tooling/test_conformance.py` enforces these mechanically — run it: + +1. **Transport isolation.** Only `router.py` / `main.py` / `core/cli.py` import + typer. A service must not know it was called from a CLI. +2. **One output path.** No `print()`, no `sys.stdout.write` — use + `core.console`: `console.out(data)` for the result, `console.event(msg, + level=…, phase=…, progress=…)` for progress, `console.verdict(msg)` once at + the end. +3. **Every command is `@command`**, placed UNDER `@app.command(...)`. It + composes logging, error handling and job identity; you cannot apply half. +4. **Every command has help** — the docstring. An agent finds verbs by reading it. +5. **Every `ReachError` passes `fix=`.** A failure that only says "no" is what + D-263 exists to replace. +6. **One guarded exec.** Nothing outside `core/process.py` imports + `subprocess` or calls `os.system`/`execv`. Use `process.run([argv…], + missing_fix="install X")` — an argv LIST, never a string. `process.cargo_binary(name, …)` + builds (if stale) and runs a server binary. +7. **Lazy.** Import heavy modules (numpy, PIL, the service) INSIDE the command + function, so `reach --help` stays flat as domains are added. + +### The failure contract, worked + +```python +from tooling.core.errors import ReachError, unknown_choice + +if not db_path.exists(): + raise ReachError( + f"{db_path} not found", + fix="make regen-db, or pass --db with an existing systems.db", + ) +if mode not in MODES: + raise unknown_choice("mode", mode, MODES) # names the accepted set +raise ReachError("coverage gate not met", fix="add corporations…", exit_code=2) +``` + +Name the cause AND the next command. When the cause is ambiguous, classify it +(the assets domain distinguishes "service is OFF" from "request rejected" from +"bad credentials" — see `domains/assets/endpoints.py`). + +### Add a verb + +```python +# tooling/domains//router.py +@app.command("thing") +@command +def thing(path: Path = typer.Argument(..., help="…")) -> None: + """One line an agent can act on.""" + from tooling.domains. import service # lazy + console.out(json.dumps(service.thing(path))) +``` + +Logic goes in a service module that takes plain arguments and returns data, so +a test can call it without a CLI. + +### Add a domain + +1. `tooling/domains//__init__.py` (a docstring saying what it is), + `router.py` with `app = cli.domain("", "one-line help")` and an + `@app.callback()` so a one-verb domain stays a group. +2. One line in `DOMAINS` in `tooling/main.py` — target and one-line help (the + help string lives there so `--help` never imports the domain). +3. A row in `tooling/DOMAINS.md`. +4. A nested group (like `atlas planet`) is `cli.domain(...)` plus + `parent.add_typer(child, name=...)`. + +### Porting or changing a tool: parity is the acceptance + +Before moving anything, capture a baseline from the OLD code. After, prove: +**exit codes match exactly, no fact is lost, failures name a remedy.** Compare +bytes wherever an output is byte-comparable (and decode first where it is not +— Ogg streams carry a random serial). A check counts only if it could have +failed: mutate the code once and watch the test go red. Grep for +`__file__`-relative roots before moving a file — use `config.repo_root()`. + +**Traps this repo has already paid for:** +- Anything in `tooling/generator_sources.py` is systems.db-stamped: moving or + editing it means `make regen-db` + `reach check systems-db-stamp` in the + same commit. +- Anything in `tooling/canvas_sources.py` (incl. that file) needs a + `project.yaml` version bump, even for a comment. No override. +- `make` keeps build/test orchestration only. A make target for a tool is + retired, or survives as a one-line delegate that says so (`make regen-db`). +- Some generators are archived because RUNNING them destroys committed data + (wiki_sync.generate_wiki, generate-star-map). Read `tooling/archive/README.md` + before resurrecting anything. diff --git a/.pql/changelog/ticket_history/2026-09.sql b/.pql/changelog/ticket_history/2026-09.sql index 0f0c9d8f7..e358bd230 100644 --- a/.pql/changelog/ticket_history/2026-09.sql +++ b/.pql/changelog/ticket_history/2026-09.sql @@ -622,3 +622,19 @@ Recurring finding across the ports: tools whose failure read as success. Exit 0 Two tools archived because RUNNING them is destructive against today''s committed data: wiki_sync.generate_wiki (T-1292) and generate-star-map (T-1294). In both cases the docs and rules were telling agents to run them.', NULL, '2026-09-23 18:05:18', '2026-09-23 18:05:18.760', '2026-09-23 18:05:18.760', NULL, '1ce5c9f62390a48d046d640c0f3ee46c', 2) ON CONFLICT(hash) DO NOTHING; INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G1S3D0M1TQW0GMFBBPQZG3ZM', 'status', 'in_progress', 'done', NULL, '2026-09-23 18:05:19', '2026-09-23 18:05:19.260', '2026-09-23 18:05:19.260', NULL, '4178a2e1f60ae93db618f7ee7d77efd6', 2) ON CONFLICT(hash) DO NOTHING; +INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G1S3DTRHSBYJS55NDT9YR4R0', 'description', 'The highest-frequency, highest-friction surface, and the one with a hard number attached. Baseline measured 2026-08-20: check-client-version 20 ms, check-dataflow-graph 38 ms, check-canvas-version 46 ms — about 104 ms for the three unconditional checks; check-systems-db-stamp runs only when systems.db is in the push. The ported gates must not exceed that. Pieces that become tickets: (1) port the four checks into domains/check/ — client-version, canvas-version, dataflow-graph, systems-db-stamp; (2) port validate-content, validate-checklist, validate-ron, check-fact-ids into domains/validate/; (3) rewire .config/hooks/pre-push to call reach, keeping the fail_check by-name reporting that hook deliberately has; (4) prove each gate still FAILS — a check that explains itself and exits 0 silently disables its own gate, which is exactly the clide failure recorded in D-263; test each one against a deliberately broken tree, not just a clean one; (5) the wall-clock assertion that turns the budget into a test; (6) confirm domains/check/ imports no pydantic — 87 ms against a 20 ms check is a 3.5x regression bought for nothing, four times per push. Depends on E1 (lazy registration) and E2 (the error contract) being real first. + +AMENDED 2026-08-20 — the 104 ms ceiling in the description above is WITHDRAWN. D-263 was amended the same day: the acceptance criterion is OUTPUT PARITY, not timing parity. A ported check must produce the same output and the same exit code as the script it replaces; it is not required to be as fast. Reason: the only automatic consumer is the pre-push hook, which on a server or client push already runs cargo test or the gdUnit4 suite (minutes), and on a governance-only push totals about a second — so a few hundred ms is invisible either way. The replacement budget is a ceiling with headroom: a reach invocation should feel instant to a human, under about 250 ms, and the unconditional gate set stays comfortably under a second. Item (5) of this ticket, the wall-clock assertion, is REPLACED by an import-graph assertion — a timing test is flaky on a loaded machine and reports that something got slow rather than what; asserting sys.modules after reach --help names the offender. Item (6), the no-pydantic-in-check rule, is DROPPED — pydantic at 87 ms is unremarkable against a 250 ms ceiling, and the carve-out bought 87 ms at the cost of an exception to the layering. + +FROM T-1262 (2026-08-31) — what ''output parity'' means was sharpened while porting the first gate, and it applies to every port in this epic. Byte-for-byte parity is IMPOSSIBLE under the streaming model and should not be attempted: D-263 puts the verdict on stderr as part of the event stream, while the scripts being replaced write their success line to stdout. Matching both would mean abandoning streaming or special-casing each ported gate. The enforceable definition, now in D-263: (a) EXIT CODES MATCH EXACTLY — the hook gates on this and it is the only part a caller can act on programmatically; (b) NO FACT IS LOST — every version number, path and reason the old message carried must appear in the new one, since a migration that silently drops a detail makes the failure harder to fix than before; (c) FAILURES NAME A REMEDY as a structured field, which the old scripts did only in prose. In text mode the success line is in fact byte-identical; only the stream differs. tooling/test_check_parity.py is the working pattern to copy for the other three checks — it builds a throwaway fixture repo (sentinel project.yaml plus client/project.godot), copies the OLD script into it so its __file__-relative root resolves there, points the new command at the same fixture via SR_REPO_ROOT, and compares. Note the asymmetry it exposes: the old scripts have NO root override, which is precisely why they are hard to test and part of why the port is worth doing.', 'The highest-frequency, highest-friction surface, and the one with a hard number attached. Baseline measured 2026-08-20: check-client-version 20 ms, check-dataflow-graph 38 ms, check-canvas-version 46 ms — about 104 ms for the three unconditional checks; check-systems-db-stamp runs only when systems.db is in the push. The ported gates must not exceed that. Pieces that become tickets: (1) port the four checks into domains/check/ — client-version, canvas-version, dataflow-graph, systems-db-stamp; (2) port validate-content, validate-checklist, validate-ron, check-fact-ids into domains/validate/; (3) rewire .config/hooks/pre-push to call reach, keeping the fail_check by-name reporting that hook deliberately has; (4) prove each gate still FAILS — a check that explains itself and exits 0 silently disables its own gate, which is exactly the clide failure recorded in D-263; test each one against a deliberately broken tree, not just a clean one; (5) the wall-clock assertion that turns the budget into a test; (6) confirm domains/check/ imports no pydantic — 87 ms against a 20 ms check is a 3.5x regression bought for nothing, four times per push. Depends on E1 (lazy registration) and E2 (the error contract) being real first. + +AMENDED 2026-08-20 — the 104 ms ceiling in the description above is WITHDRAWN. D-263 was amended the same day: the acceptance criterion is OUTPUT PARITY, not timing parity. A ported check must produce the same output and the same exit code as the script it replaces; it is not required to be as fast. Reason: the only automatic consumer is the pre-push hook, which on a server or client push already runs cargo test or the gdUnit4 suite (minutes), and on a governance-only push totals about a second — so a few hundred ms is invisible either way. The replacement budget is a ceiling with headroom: a reach invocation should feel instant to a human, under about 250 ms, and the unconditional gate set stays comfortably under a second. Item (5) of this ticket, the wall-clock assertion, is REPLACED by an import-graph assertion — a timing test is flaky on a loaded machine and reports that something got slow rather than what; asserting sys.modules after reach --help names the offender. Item (6), the no-pydantic-in-check rule, is DROPPED — pydantic at 87 ms is unremarkable against a 250 ms ceiling, and the carve-out bought 87 ms at the cost of an exception to the layering. + +FROM T-1262 (2026-08-31) — what ''output parity'' means was sharpened while porting the first gate, and it applies to every port in this epic. Byte-for-byte parity is IMPOSSIBLE under the streaming model and should not be attempted: D-263 puts the verdict on stderr as part of the event stream, while the scripts being replaced write their success line to stdout. Matching both would mean abandoning streaming or special-casing each ported gate. The enforceable definition, now in D-263: (a) EXIT CODES MATCH EXACTLY — the hook gates on this and it is the only part a caller can act on programmatically; (b) NO FACT IS LOST — every version number, path and reason the old message carried must appear in the new one, since a migration that silently drops a detail makes the failure harder to fix than before; (c) FAILURES NAME A REMEDY as a structured field, which the old scripts did only in prose. In text mode the success line is in fact byte-identical; only the stream differs. tooling/test_check_parity.py is the working pattern to copy for the other three checks — it builds a throwaway fixture repo (sentinel project.yaml plus client/project.godot), copies the OLD script into it so its __file__-relative root resolves there, points the new command at the same fixture via SR_REPO_ROOT, and compares. Note the asymmetry it exposes: the old scripts have NO root override, which is precisely why they are hard to test and part of why the port is worth doing. + +DONE 2026-09-23, audited against the items rather than assumed. (1)+(2) the gate ports landed as T-1281 (check) and T-1282 (validate). (3) .config/hooks/pre-push and pre-commit invoke only `reach --no-input …` for every gate (plus ruff and json.tool); no tooling/ script path survives in any hook. (4) gates proven to FAIL: test_check.py runs 19 cases across five gates, failures included; this session the canvas-version gate did its job for real, rejecting a push of a comment-only change to a registered file (hence 0.4.14). (5) replaced by the import-graph assertion, which exists: test_lazy_domains.py asserts that after `reach --help` no tooling.domains module and none of rich/pygments/numpy/scipy/pydantic/PIL is imported, with a positive control. (6) dropped by the 2026-08-20 amendment.', NULL, '2026-09-23 18:16:36', '2026-09-23 18:16:36.200', '2026-09-23 18:16:36.200', NULL, '36015bf18eb0617695d7b3d69838b857', 2) ON CONFLICT(hash) DO NOTHING; +INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G1S3F6SKZ6EQQ7TY0RKT2970', 'description', 'The riskiest epic, and the one with a hard sequencing constraint. tooling/generator_sources.py SHAs the concatenated bytes of the generator sources SORTED BY PATH, so renaming a file changes the stamp even when its content is byte-identical. The relocation must therefore land as ONE commit — registry paths updated, make regen-db run, stamp verified — never split across pushes, or the pre-push gate rejects an intermediate state that is in fact correct. Pieces that become tickets: (1) move import_economics.py and the economy_import package into domains/db/, update generator_sources.py in the same change, regen and verify the stamp; (2) the generate_brands Rust binary is a SUBROUTINE of the Python importer, not an independent generator — keep that relationship intact; (3) canvas_sources.py and schema_version.py move with their consumers, and check-canvas-version must keep resolving the registry; (4) planet-gen into domains/planet/ — import_heightmaps, import_province_boundaries, planet_simulation, scaffold_bodies, render_heightmap; these are one-time build imports baked into the committed DB, intentionally unstamped, and they lean on numpy/scipy/PIL; (5) garment-fit into domains/garment/ with its 23 blender_author_* files going to scripts/blender/ per the carve-out; (6) db/wiki_sync.py and the audio/image/trellis connectors, which resolve a venv explicitly today in tooling/db/common.py and should stop needing to; (7) test_sim_determinism and test_oasis_ring_scaling keep passing — they are the guard that the numerics did not move under us. NOTHING here is rewritten. Code is relocated and re-fronted; no numerical behaviour changes, which is precisely why Rust was rejected.', 'The riskiest epic, and the one with a hard sequencing constraint. tooling/generator_sources.py SHAs the concatenated bytes of the generator sources SORTED BY PATH, so renaming a file changes the stamp even when its content is byte-identical. The relocation must therefore land as ONE commit — registry paths updated, make regen-db run, stamp verified — never split across pushes, or the pre-push gate rejects an intermediate state that is in fact correct. Pieces that become tickets: (1) move import_economics.py and the economy_import package into domains/db/, update generator_sources.py in the same change, regen and verify the stamp; (2) the generate_brands Rust binary is a SUBROUTINE of the Python importer, not an independent generator — keep that relationship intact; (3) canvas_sources.py and schema_version.py move with their consumers, and check-canvas-version must keep resolving the registry; (4) planet-gen into domains/planet/ — import_heightmaps, import_province_boundaries, planet_simulation, scaffold_bodies, render_heightmap; these are one-time build imports baked into the committed DB, intentionally unstamped, and they lean on numpy/scipy/PIL; (5) garment-fit into domains/garment/ with its 23 blender_author_* files going to scripts/blender/ per the carve-out; (6) db/wiki_sync.py and the audio/image/trellis connectors, which resolve a venv explicitly today in tooling/db/common.py and should stop needing to; (7) test_sim_determinism and test_oasis_ring_scaling keep passing — they are the guard that the numerics did not move under us. NOTHING here is rewritten. Code is relocated and re-fronted; no numerical behaviour changes, which is precisely why Rust was rejected. + +DONE 2026-09-23 — delivered by the E3 ports, item by item. (1) import_economics + economy_import moved to domains/ledger/ (the map renamed db to ledger), generator_sources.py updated in the same commit, regen run, stamp verified: T-1289. (2) generate_brands kept as a subroutine of the importer, and the relationship is now tighter: cargo_binary rebuilds a stale binary, where before a Rust edit could be stamped over output from the old binary (fix commit e3daf561d). (3) schema_version.py moved with the ledger; canvas_sources.py stays a top-level registry module per the domain map, and check-canvas-version still resolves it. (4) planet-gen became domains/atlas/planet (T-1288); the one-time importers stayed unstamped. (5) garment-fit: 22 payloads to scripts/blender (T-1273) + make_logo to domains/character (T-1290). (6) wiki_sync + connectors to domains/wiki and domains/assets; ensure_venv is gone (T-1290). (7) test_sim_determinism and test_oasis_ring_scaling moved to tooling/test_planet_*.py and pass; globe renders were pixel-identical across the T-1274 cleanup. Nothing numeric changed: every byte-comparable generator output was checked identical before/after.', NULL, '2026-09-23 18:16:36', '2026-09-23 18:16:36.730', '2026-09-23 18:16:36.730', NULL, '94372e7d3f4ec83ccfda9cf5844e0b93', 2) ON CONFLICT(hash) DO NOTHING; +INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G1S3DTRHSBYJS55NDT9YR4R0', 'status', 'backlog', 'done', NULL, '2026-09-23 18:16:37', '2026-09-23 18:16:37.323', '2026-09-23 18:16:37.323', NULL, '3f5b676d54ad0b58a6f0879f9a5c7642', 2) ON CONFLICT(hash) DO NOTHING; +INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G1S3F6SKZ6EQQ7TY0RKT2970', 'status', 'backlog', 'done', NULL, '2026-09-23 18:16:37', '2026-09-23 18:16:37.331', '2026-09-23 18:16:37.331', NULL, '28db5755c043d2ee316b03594cbc8de1', 2) ON CONFLICT(hash) DO NOTHING; diff --git a/.pql/changelog/tickets/2026-09.sql b/.pql/changelog/tickets/2026-09.sql index d4218073f..8db535a57 100644 --- a/.pql/changelog/tickets/2026-09.sql +++ b/.pql/changelog/tickets/2026-09.sql @@ -770,3 +770,23 @@ DONE 2026-09-23. Every legacy file under tooling/ is now in one of four places: Recurring finding across the ports: tools whose failure read as success. Exit 0 with failures in batch --verify-determinism, import-provinces, audio batch, trellis batch and the garment-QA analyzer; a crash on trellis generate; glb strip re-reporting orphans. All fixed in the port that found them. Two tools archived because RUNNING them is destructive against today''s committed data: wiki_sync.generate_wiki (T-1292) and generate-star-map (T-1294). In both cases the docs and rules were telling agents to run them.', 'done', 'medium', NULL, NULL, 'D-263', '2026-08-20 00:23:58.880', '2026-09-23 18:05:19.260', NULL, 'c1e720e40b6c411fd917c8423d1c42d9', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at; +INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G1S3DTRHSBYJS55NDT9YR4R0', 'epic', '06G1S37Y1ARMV68RJT802Z3VPG', 'E4 — The gate family and the push hook, at output parity', 'The highest-frequency, highest-friction surface, and the one with a hard number attached. Baseline measured 2026-08-20: check-client-version 20 ms, check-dataflow-graph 38 ms, check-canvas-version 46 ms — about 104 ms for the three unconditional checks; check-systems-db-stamp runs only when systems.db is in the push. The ported gates must not exceed that. Pieces that become tickets: (1) port the four checks into domains/check/ — client-version, canvas-version, dataflow-graph, systems-db-stamp; (2) port validate-content, validate-checklist, validate-ron, check-fact-ids into domains/validate/; (3) rewire .config/hooks/pre-push to call reach, keeping the fail_check by-name reporting that hook deliberately has; (4) prove each gate still FAILS — a check that explains itself and exits 0 silently disables its own gate, which is exactly the clide failure recorded in D-263; test each one against a deliberately broken tree, not just a clean one; (5) the wall-clock assertion that turns the budget into a test; (6) confirm domains/check/ imports no pydantic — 87 ms against a 20 ms check is a 3.5x regression bought for nothing, four times per push. Depends on E1 (lazy registration) and E2 (the error contract) being real first. + +AMENDED 2026-08-20 — the 104 ms ceiling in the description above is WITHDRAWN. D-263 was amended the same day: the acceptance criterion is OUTPUT PARITY, not timing parity. A ported check must produce the same output and the same exit code as the script it replaces; it is not required to be as fast. Reason: the only automatic consumer is the pre-push hook, which on a server or client push already runs cargo test or the gdUnit4 suite (minutes), and on a governance-only push totals about a second — so a few hundred ms is invisible either way. The replacement budget is a ceiling with headroom: a reach invocation should feel instant to a human, under about 250 ms, and the unconditional gate set stays comfortably under a second. Item (5) of this ticket, the wall-clock assertion, is REPLACED by an import-graph assertion — a timing test is flaky on a loaded machine and reports that something got slow rather than what; asserting sys.modules after reach --help names the offender. Item (6), the no-pydantic-in-check rule, is DROPPED — pydantic at 87 ms is unremarkable against a 250 ms ceiling, and the carve-out bought 87 ms at the cost of an exception to the layering. + +FROM T-1262 (2026-08-31) — what ''output parity'' means was sharpened while porting the first gate, and it applies to every port in this epic. Byte-for-byte parity is IMPOSSIBLE under the streaming model and should not be attempted: D-263 puts the verdict on stderr as part of the event stream, while the scripts being replaced write their success line to stdout. Matching both would mean abandoning streaming or special-casing each ported gate. The enforceable definition, now in D-263: (a) EXIT CODES MATCH EXACTLY — the hook gates on this and it is the only part a caller can act on programmatically; (b) NO FACT IS LOST — every version number, path and reason the old message carried must appear in the new one, since a migration that silently drops a detail makes the failure harder to fix than before; (c) FAILURES NAME A REMEDY as a structured field, which the old scripts did only in prose. In text mode the success line is in fact byte-identical; only the stream differs. tooling/test_check_parity.py is the working pattern to copy for the other three checks — it builds a throwaway fixture repo (sentinel project.yaml plus client/project.godot), copies the OLD script into it so its __file__-relative root resolves there, points the new command at the same fixture via SR_REPO_ROOT, and compares. Note the asymmetry it exposes: the old scripts have NO root override, which is precisely why they are hard to test and part of why the port is worth doing. + +DONE 2026-09-23, audited against the items rather than assumed. (1)+(2) the gate ports landed as T-1281 (check) and T-1282 (validate). (3) .config/hooks/pre-push and pre-commit invoke only `reach --no-input …` for every gate (plus ruff and json.tool); no tooling/ script path survives in any hook. (4) gates proven to FAIL: test_check.py runs 19 cases across five gates, failures included; this session the canvas-version gate did its job for real, rejecting a push of a comment-only change to a registered file (hence 0.4.14). (5) replaced by the import-graph assertion, which exists: test_lazy_domains.py asserts that after `reach --help` no tooling.domains module and none of rich/pygments/numpy/scipy/pydantic/PIL is imported, with a positive control. (6) dropped by the 2026-08-20 amendment.', 'backlog', 'high', NULL, NULL, 'D-263', '2026-08-20 00:24:05.572', '2026-09-23 18:16:36.200', NULL, '4da4c39a424c323ff32d460d739cb130', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at; +INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G1S3F6SKZ6EQQ7TY0RKT2970', 'epic', '06G1S37Y1ARMV68RJT802Z3VPG', 'E5 — Generators and numerics, without breaking the systems.db stamp', 'The riskiest epic, and the one with a hard sequencing constraint. tooling/generator_sources.py SHAs the concatenated bytes of the generator sources SORTED BY PATH, so renaming a file changes the stamp even when its content is byte-identical. The relocation must therefore land as ONE commit — registry paths updated, make regen-db run, stamp verified — never split across pushes, or the pre-push gate rejects an intermediate state that is in fact correct. Pieces that become tickets: (1) move import_economics.py and the economy_import package into domains/db/, update generator_sources.py in the same change, regen and verify the stamp; (2) the generate_brands Rust binary is a SUBROUTINE of the Python importer, not an independent generator — keep that relationship intact; (3) canvas_sources.py and schema_version.py move with their consumers, and check-canvas-version must keep resolving the registry; (4) planet-gen into domains/planet/ — import_heightmaps, import_province_boundaries, planet_simulation, scaffold_bodies, render_heightmap; these are one-time build imports baked into the committed DB, intentionally unstamped, and they lean on numpy/scipy/PIL; (5) garment-fit into domains/garment/ with its 23 blender_author_* files going to scripts/blender/ per the carve-out; (6) db/wiki_sync.py and the audio/image/trellis connectors, which resolve a venv explicitly today in tooling/db/common.py and should stop needing to; (7) test_sim_determinism and test_oasis_ring_scaling keep passing — they are the guard that the numerics did not move under us. NOTHING here is rewritten. Code is relocated and re-fronted; no numerical behaviour changes, which is precisely why Rust was rejected. + +DONE 2026-09-23 — delivered by the E3 ports, item by item. (1) import_economics + economy_import moved to domains/ledger/ (the map renamed db to ledger), generator_sources.py updated in the same commit, regen run, stamp verified: T-1289. (2) generate_brands kept as a subroutine of the importer, and the relationship is now tighter: cargo_binary rebuilds a stale binary, where before a Rust edit could be stamped over output from the old binary (fix commit e3daf561d). (3) schema_version.py moved with the ledger; canvas_sources.py stays a top-level registry module per the domain map, and check-canvas-version still resolves it. (4) planet-gen became domains/atlas/planet (T-1288); the one-time importers stayed unstamped. (5) garment-fit: 22 payloads to scripts/blender (T-1273) + make_logo to domains/character (T-1290). (6) wiki_sync + connectors to domains/wiki and domains/assets; ensure_venv is gone (T-1290). (7) test_sim_determinism and test_oasis_ring_scaling moved to tooling/test_planet_*.py and pass; globe renders were pixel-identical across the T-1274 cleanup. Nothing numeric changed: every byte-comparable generator output was checked identical before/after.', 'backlog', 'medium', NULL, NULL, 'D-263', '2026-08-20 00:24:16.844', '2026-09-23 18:16:36.730', NULL, 'c49d623a9c7d0357d7c650ed17db5c3f', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at; +INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G1S3DTRHSBYJS55NDT9YR4R0', 'epic', '06G1S37Y1ARMV68RJT802Z3VPG', 'E4 — The gate family and the push hook, at output parity', 'The highest-frequency, highest-friction surface, and the one with a hard number attached. Baseline measured 2026-08-20: check-client-version 20 ms, check-dataflow-graph 38 ms, check-canvas-version 46 ms — about 104 ms for the three unconditional checks; check-systems-db-stamp runs only when systems.db is in the push. The ported gates must not exceed that. Pieces that become tickets: (1) port the four checks into domains/check/ — client-version, canvas-version, dataflow-graph, systems-db-stamp; (2) port validate-content, validate-checklist, validate-ron, check-fact-ids into domains/validate/; (3) rewire .config/hooks/pre-push to call reach, keeping the fail_check by-name reporting that hook deliberately has; (4) prove each gate still FAILS — a check that explains itself and exits 0 silently disables its own gate, which is exactly the clide failure recorded in D-263; test each one against a deliberately broken tree, not just a clean one; (5) the wall-clock assertion that turns the budget into a test; (6) confirm domains/check/ imports no pydantic — 87 ms against a 20 ms check is a 3.5x regression bought for nothing, four times per push. Depends on E1 (lazy registration) and E2 (the error contract) being real first. + +AMENDED 2026-08-20 — the 104 ms ceiling in the description above is WITHDRAWN. D-263 was amended the same day: the acceptance criterion is OUTPUT PARITY, not timing parity. A ported check must produce the same output and the same exit code as the script it replaces; it is not required to be as fast. Reason: the only automatic consumer is the pre-push hook, which on a server or client push already runs cargo test or the gdUnit4 suite (minutes), and on a governance-only push totals about a second — so a few hundred ms is invisible either way. The replacement budget is a ceiling with headroom: a reach invocation should feel instant to a human, under about 250 ms, and the unconditional gate set stays comfortably under a second. Item (5) of this ticket, the wall-clock assertion, is REPLACED by an import-graph assertion — a timing test is flaky on a loaded machine and reports that something got slow rather than what; asserting sys.modules after reach --help names the offender. Item (6), the no-pydantic-in-check rule, is DROPPED — pydantic at 87 ms is unremarkable against a 250 ms ceiling, and the carve-out bought 87 ms at the cost of an exception to the layering. + +FROM T-1262 (2026-08-31) — what ''output parity'' means was sharpened while porting the first gate, and it applies to every port in this epic. Byte-for-byte parity is IMPOSSIBLE under the streaming model and should not be attempted: D-263 puts the verdict on stderr as part of the event stream, while the scripts being replaced write their success line to stdout. Matching both would mean abandoning streaming or special-casing each ported gate. The enforceable definition, now in D-263: (a) EXIT CODES MATCH EXACTLY — the hook gates on this and it is the only part a caller can act on programmatically; (b) NO FACT IS LOST — every version number, path and reason the old message carried must appear in the new one, since a migration that silently drops a detail makes the failure harder to fix than before; (c) FAILURES NAME A REMEDY as a structured field, which the old scripts did only in prose. In text mode the success line is in fact byte-identical; only the stream differs. tooling/test_check_parity.py is the working pattern to copy for the other three checks — it builds a throwaway fixture repo (sentinel project.yaml plus client/project.godot), copies the OLD script into it so its __file__-relative root resolves there, points the new command at the same fixture via SR_REPO_ROOT, and compares. Note the asymmetry it exposes: the old scripts have NO root override, which is precisely why they are hard to test and part of why the port is worth doing. + +DONE 2026-09-23, audited against the items rather than assumed. (1)+(2) the gate ports landed as T-1281 (check) and T-1282 (validate). (3) .config/hooks/pre-push and pre-commit invoke only `reach --no-input …` for every gate (plus ruff and json.tool); no tooling/ script path survives in any hook. (4) gates proven to FAIL: test_check.py runs 19 cases across five gates, failures included; this session the canvas-version gate did its job for real, rejecting a push of a comment-only change to a registered file (hence 0.4.14). (5) replaced by the import-graph assertion, which exists: test_lazy_domains.py asserts that after `reach --help` no tooling.domains module and none of rich/pygments/numpy/scipy/pydantic/PIL is imported, with a positive control. (6) dropped by the 2026-08-20 amendment.', 'done', 'high', NULL, NULL, 'D-263', '2026-08-20 00:24:05.572', '2026-09-23 18:16:37.317', NULL, '4d2efdd656d6c8e09b09f1814417aeb1', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at; +INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G1S3F6SKZ6EQQ7TY0RKT2970', 'epic', '06G1S37Y1ARMV68RJT802Z3VPG', 'E5 — Generators and numerics, without breaking the systems.db stamp', 'The riskiest epic, and the one with a hard sequencing constraint. tooling/generator_sources.py SHAs the concatenated bytes of the generator sources SORTED BY PATH, so renaming a file changes the stamp even when its content is byte-identical. The relocation must therefore land as ONE commit — registry paths updated, make regen-db run, stamp verified — never split across pushes, or the pre-push gate rejects an intermediate state that is in fact correct. Pieces that become tickets: (1) move import_economics.py and the economy_import package into domains/db/, update generator_sources.py in the same change, regen and verify the stamp; (2) the generate_brands Rust binary is a SUBROUTINE of the Python importer, not an independent generator — keep that relationship intact; (3) canvas_sources.py and schema_version.py move with their consumers, and check-canvas-version must keep resolving the registry; (4) planet-gen into domains/planet/ — import_heightmaps, import_province_boundaries, planet_simulation, scaffold_bodies, render_heightmap; these are one-time build imports baked into the committed DB, intentionally unstamped, and they lean on numpy/scipy/PIL; (5) garment-fit into domains/garment/ with its 23 blender_author_* files going to scripts/blender/ per the carve-out; (6) db/wiki_sync.py and the audio/image/trellis connectors, which resolve a venv explicitly today in tooling/db/common.py and should stop needing to; (7) test_sim_determinism and test_oasis_ring_scaling keep passing — they are the guard that the numerics did not move under us. NOTHING here is rewritten. Code is relocated and re-fronted; no numerical behaviour changes, which is precisely why Rust was rejected. + +DONE 2026-09-23 — delivered by the E3 ports, item by item. (1) import_economics + economy_import moved to domains/ledger/ (the map renamed db to ledger), generator_sources.py updated in the same commit, regen run, stamp verified: T-1289. (2) generate_brands kept as a subroutine of the importer, and the relationship is now tighter: cargo_binary rebuilds a stale binary, where before a Rust edit could be stamped over output from the old binary (fix commit e3daf561d). (3) schema_version.py moved with the ledger; canvas_sources.py stays a top-level registry module per the domain map, and check-canvas-version still resolves it. (4) planet-gen became domains/atlas/planet (T-1288); the one-time importers stayed unstamped. (5) garment-fit: 22 payloads to scripts/blender (T-1273) + make_logo to domains/character (T-1290). (6) wiki_sync + connectors to domains/wiki and domains/assets; ensure_venv is gone (T-1290). (7) test_sim_determinism and test_oasis_ring_scaling moved to tooling/test_planet_*.py and pass; globe renders were pixel-identical across the T-1274 cleanup. Nothing numeric changed: every byte-comparable generator output was checked identical before/after.', 'done', 'medium', NULL, NULL, 'D-263', '2026-08-20 00:24:16.844', '2026-09-23 18:16:37.328', NULL, 'c67cf493bc8411754fdd7d1e7871440f', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at; diff --git a/CLAUDE.md b/CLAUDE.md index 047640735..df3c88b7b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -39,7 +39,19 @@ unreachable from here until 2026-08-20. ## DevOps -See [docs/DEVOPS.md](docs/DEVOPS.md) for build, test, lint, and CI procedures. All development operations go through the top-level `Makefile` — run `make` for a summary of targets. +See [docs/DEVOPS.md](docs/DEVOPS.md) for build, test, lint, and CI procedures. Build and test orchestration goes through the top-level `Makefile` — run `make` for a summary of targets. + +### Tooling: one CLI (D-263) + +All repo tooling is `reach` — `reach --help` is the index, and `Skill(reach)` +covers use and authoring. **No tooling is developed outside it**: a new tool is +a verb on an existing domain or a new domain under `tooling/domains/`, never a +loose script, an extensionless executable, a `python3 tooling/.py` +entrypoint, or a make target that holds logic. The tree fragmented into ~160 +files and four naming conventions precisely because nothing ruled out adding +one more. Two exceptions only: Blender payloads in `tooling/scripts/blender/` +(Blender's bundled Python cannot import the package) and the Rust crates under +`tooling/` (`econ-sim`, `line-previewer`, `test-client`). ### Asset pipeline diff --git a/docs/diagrams/architecture/reach-cli.d2 b/docs/diagrams/architecture/reach-cli.d2 new file mode 100644 index 000000000..8ce3fb8c8 --- /dev/null +++ b/docs/diagrams/architecture/reach-cli.d2 @@ -0,0 +1,68 @@ +# reach — one CLI for all repo tooling (D-263) +# +# WHAT THIS ANSWERS: when something invokes a repo tool, what does the call +# pass through, and where is each kind of code allowed to live? +# +# Verified 2026-09-23 against the source (tooling/main.py, core/, domains/, +# test_conformance.py) — module names are real; the arrows are the import and +# call paths, not inferred. The conformance test enforces the rules in the +# legend; this picture does not. +# +# View: clide draw --file docs/diagrams/architecture/reach-cli.d2 + +direction: down + +classes: { + caller: {style: {fill: "#1f6f43"; stroke: "#8fd9ae"; font-color: "#ffffff"; bold: true; border-radius: 4}} + door: {shape: hexagon; style: {fill: "#2b4c7e"; stroke: "#9dc0f0"; font-color: "#ffffff"; bold: true}} + core: {style: {fill: "#5c2d6e"; stroke: "#d3a8e6"; font-color: "#ffffff"; border-radius: 4}} + domain: {style: {fill: "#7a4b12"; stroke: "#e0b070"; font-color: "#ffffff"; border-radius: 4}} + outside: {style: {fill: "#3a3a42"; stroke: "#9a9aa4"; font-color: "#ffffff"; border-radius: 4; stroke-dash: 4}} + ext: {shape: cylinder; style: {fill: "#2a2a30"; stroke: "#9a9aa4"; font-color: "#ffffff"}} +} + +# ---- callers ---- +agent: "agents + humans\nreach " {class: caller} +hooks: "git hooks\nreach --no-input check …" {class: caller} +make: "make (orchestration only)\none-line delegates, e.g. regen-db" {class: caller} + +# ---- the door ---- +main: "tooling/main.py\nrouting only · DOMAINS registry\nlazy: --help imports no domain" {class: door} + +# ---- core ---- +command: "core/command.py @command\njob identity ⊃ handle_errors ⊃ logged" {class: core} +console: "core/console.py\nstdout = data · stderr = JSONL events + one verdict" {class: core} +errors: "core/errors.py ReachError\nevery raise carries fix=" {class: core} +process: "core/process.py — the ONE guarded exec\nargv lists · missing-binary remedy · cargo_binary" {class: core} +jobs: "core/jobs.py + domains/jobs\n--detach · log --follow · wait" {class: core} + +# ---- domains ---- +router: "domains//router.py\ntransport: args in, format out\nthe only place typer may be imported" {class: domain} +service: "domains//*.py (services)\nplain args in, data out\ncallable from tests without a CLI" {class: domain} +list: "14 domains\ncheck · validate · atlas {map, planet} · ledger · wiki\nassets · character · godot · visual · generate\nblender · pr · jobs · dev" {class: domain} + +# ---- outside the package ---- +payloads: "tooling/scripts/blender/ (41 payloads)\nrun under Blender's Python — never imported" {class: outside} +archive: "tooling/archive/\nprovenance — never run, not linted" {class: outside} + +# ---- external programs ---- +extbin: "cargo · godot · blender · ffmpeg · git · xvfb-run" {class: ext} + +agent -> main +hooks -> main +make -> main +main -> router: "loads ONE domain, on use" +router -> command: "wraps every verb" +router -> service: "delegates" +router -> list {style.stroke-dash: 3} +command -> errors +command -> jobs +service -> console: "the only output path" +errors -> console: "verdict + fix" +service -> process: "any external program" +process -> extbin +process -> payloads: "reach blender run" + +legend: "ENFORCED BY tooling/test_conformance.py\n\n1 typer only in router.py / main.py / core/cli.py\n2 no print(): everything through console\n3 every verb is @command, and has help\n4 every ReachError names a fix\n5 no subprocess outside core/process.py\n6 the Blender carve-out stays outside the package\n\nColour: green caller · blue the door · purple core\namber domains · grey dashed outside the package" { + style: {fill: "#1b1b22"; stroke: "#9a9aa4"; font-color: "#c8d0e0"} +} diff --git a/docs/diagrams/architecture/reach-cli.svg b/docs/diagrams/architecture/reach-cli.svg new file mode 100644 index 000000000..3454ef591 --- /dev/null +++ b/docs/diagrams/architecture/reach-cli.svg @@ -0,0 +1,108 @@ +agents + humansreach <domain> <verb>git hooksreach --no-input check …make (orchestration only)one-line delegates, e.g. regen-dbtooling/main.pyrouting only · DOMAINS registrylazy: --help imports no domaincore/command.py @commandjob identity ⊃ handle_errors ⊃ loggedcore/console.pystdout = data · stderr = JSONL events + one verdictcore/errors.py ReachErrorevery raise carries fix=core/process.py — the ONE guarded execargv lists · missing-binary remedy · cargo_binarycore/jobs.py + domains/jobs--detach · log --follow · waitdomains/<d>/router.pytransport: args in, format outthe only place typer may be importeddomains/<d>/*.py (services)plain args in, data outcallable from tests without a CLI14 domainscheck · validate · atlas {map, planet} · ledger · wikiassets · character · godot · visual · generateblender · pr · jobs · devtooling/scripts/blender/ (41 payloads)run under Blender's Python — never importedtooling/archive/provenance — never run, not lintedcargo · godot · blender · ffmpeg · git · xvfb-runENFORCED BY tooling/test_conformance.py 1 typer only in router.py / main.py / core/cli.py2 no print(): everything through console3 every verb is @command, and has help4 every ReachError names a fix5 no subprocess outside core/process.py6 the Blender carve-out stays outside the package Colour: green caller · blue the door · purple coreamber domains · grey dashed outside the package loads ONE domain, on usewraps every verbdelegates the only output pathverdict + fixany external programreach blender run + + + + + + + + + diff --git a/governance/decisions/architecture.md b/governance/decisions/architecture.md index c56e0eba9..0092dfe7f 100644 --- a/governance/decisions/architecture.md +++ b/governance/decisions/architecture.md @@ -2538,6 +2538,7 @@ Technical foundation decisions that constrain implementation: engine, client-ser - **Date:** 2026-08-20 - **Resolves:** [Q-124](../questions/architecture.md#q-124-should-the-python-tooling-be-retooled-into-a-single-rust-cli). The Rust option is recorded as [R-014](../rejected/architecture.md#r-014-rust-rewrite-of-the-python-tooling). - **Decision:** `tooling/` becomes **one installable Python package with one console entrypoint, `reach`**, structured into domain subcommands, installed as a bare name on PATH. It stays Python. Nothing is rewritten — the code is **relocated and re-fronted**. +- **Diagram:** [docs/diagrams/architecture/reach-cli.d2](../../docs/diagrams/architecture/reach-cli.d2) — the door, core, the domains, the carve-out, and the invariants the conformance test enforces (added T-1256, 2026-09-23, when the CLI it describes existed). **The four calls, fixed here so no ticket has to re-litigate them:**