Files

T

jpmschweitzerandClaude Opus 5.5 4537b71b92 refactor(tooling): T-1290 — the wiki domain, and the renderer that must not run

`reach wiki stats` and `reach wiki gttr-hook` replace tooling/db/wiki_sync.py
and populate_gttr_hook.py. Both are output-identical to the originals:
`stats` byte-for-byte, and all 301 extracted GTTR hooks line-for-line.

wiki_sync.py moved whole, but generate_wiki() and import_from_wiki() are NOT
verbs. Before porting, the old `--generate` was run against a clean tree to get
a parity baseline. It changed all 301 system pages, +940 / -10,761, and was
reverted at once. It deletes the Celestial Bodies / Stations blocks (owned by
the Rust atlas sync, which it does not know about), deletes the
Industries / Exports / Imports rows (nothing writes those any more), and
rewrites star types where systems.db and the pages disagree. D-262, CLAUDE.md
and the wiki skill all described it as the routine, prose-preserving render.
CLAUDE.md and the skill now say not to run it; D-262 needs amending — T-1292.

Provenance moves to tooling/archive/, with a README naming what each script
did and why it is not run:

- pql-migrate/ (the T-1271 ruling)
- wiki-bootstrap/: assign-astro-ids + its catalog, migrate-s-to-gj,
  patch-core-sector (hardcodes a dead path), fill-missing-globes,
  generate-stubs and find-stubs (finds 0 stubs — Phase 1 is done),
  backfill_cultural_corridor (a raw systems.db patch script, outside D-262),
  and process-wiki-system-changes, whose last step is the destructive render

Also:

- stats() printed "run import first" and exited 0 when a table was missing;
  it now fails with a remedy. generate_wiki() counted created pages after
  writing them, so `created` was always 0.
- tooling/godot-cold-parse and godot-parse-sweep were never retired after
  T-1283, and the pr-process skill still told agents to run them. Removed;
  the skill and parse_sweep.gd now name the reach verbs.
- systems.db re-stamped: schema comments changed, and the stamp records the
  schema file's SHA for tamper detection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-09-23 16:25:51 +02:00

14 KiB

Raw Blame History

title, description, type, status, created, updated, decision_refs, tags

title

description

type

status

created

updated

decision_refs

pql Migration Plan — decisions + ticketing

Status: approved, not yet executed. All work happens on the pql-migration branch; main is untouched until every verification in the merge gate passes.

Context

Project planning data lives in two places today:

Tickets — a binary SQLite DB at <repo-parent>/settledreach.db (gitignored), with a committed binary backup at docs/backups/settledreach.db.backup kept fresh by a main-only make db-backup ritual. Accessed via tooling/db/ticket (23 subcommands) + tooling/db/sqlite-query|exec. 1013 tickets, 481 dependency edges, 48 labels, 16 history rows, 2 milestones.
Decisions — markdown is source-of-truth in decisions/*.md (358 D/Q/R records), synced into the same SQLite via tooling/db/decisions_sync.py (358 decisions, 1040 cross-refs). ID claiming reserves the next id in the DB before the markdown edit.

Pain points: the binary DB is merge-conflict-prone; the backup ritual is fragile; the sqlite3 CLI crashes in this environment (std::bad_alloc); and docs/workshops/wiki context is consumed by grep, not structured query.

pql (v1.6.2, ~/.local/bin/pql; repo already pql init-ed — .pql/ exists, vault index built, pql.db empty) is purpose-built for this:

Decisions — markdown-sourced, pql decisions sync|validate|claim|list|show|read|refs|coverage.
Tickets — SQLite-native, versioned by a git-tracked text changelog (.pql/changelog/<table>/<YYYY-MM>.sql, last-writer-wins, idempotent replay). pql ticket new|list|show|status|assign|team|block|unblock|append|label|board|setparent|refine
- pql plan status|whatsnext|review|export|import|rebuild.
Vault — pql query|search|backlinks|related|context|meta|tags|schema over all ~4385 markdown files (frontmatter/tags/links/headings).

Outcome: decisions + tickets fully on pql; docs/workshops/wiki folded into the queryable vault with provenance links; the old SQLite tooling retired; this doc + pql-requirements.md authored.

Findings that shaped the plan

Milestones are vestigial — do NOT migrate as an entity. Live: 2 milestones (1 active "Phase 4" / 42 tickets, 1 completed / 0), milestone_deps 0 rows, 42/1013 tickets linked, cascade_phase + cross-milestone ranking never exercised. → the one active milestone becomes a label (phase:4); /whats-next → pql ticket list --label phase:4 --unblocked.
#N → T-N is a clean bijection — pql's ticket id is a TEXT column, so seed T-440 for old #440. We switch the convention to T-NNN and run a safe find-replace of #N → T-N across markdown + skills (git history keeps #N, numerically equal to T-N).
pql's git hooks would be dead here — repo uses core.hooksPath=.config/hooks; pql installs to .git/hooks/. Fold pql steps into .config/hooks/* manually; do NOT rely on pql init's hooks.
.gitignore blanket-ignores .pql/ — must narrow to keep .pql/index.db/.pql/pql.db ignored but track .pql/changelog/ (the durable artifact).
Duplicate D-035 (content.md + perception.md) and R-011 (rejected.md + economics.md ×2) exist; pql decisions sync silently LWW-collapses them while our check-dupes errors. Fix in markdown first.
pql plan import --legacy is retired — seed the changelog directly (format below).
docs/ ~94–100% frontmattered; workshops/wiki ~89%; cross-refs are mostly bare text (searchable, but not backlinks-able). Workshop→decision provenance is prose-only today.

Decisions locked

Cutover: big-bang after a verification gate (no dual-write window).
Decisions layout: restructure to the DQR governance tree (governance/{decisions,questions,rejected}/).
docs/workshops/wiki fold-in: in-scope for this migration (provenance frontmatter + link convention).
Ticket refs: switch to T-NNN, T-N ≡ #N, via a safe tested codebase find-replace.

Execution safety: branch-only until fully verified

All work on the pql-migration branch. settledreach.db, its committed backup, and the tooling/db/* CLIs stay intact and operational on main the whole time — rollback = don't merge.

Merge gate (ALL must pass on the branch before the PR merges):

Phase-2 ticket verification gate (row/status parity + spot-checks) green.
pql decisions validate + check-dupes clean; decision count parity.
Find-replace dry-run diff reviewed + applied; rg '#\d+' residue audited.
Full cargo/test + lint suite green; hooks fire (changelog staged, replay works).
A full /whats-next → /pr-process → /pr-review dry-run on pql with zero SQLite reads.
Standard /pr-review (Hoshe + Tyre) APPROVED on the migration PR.

Phase 6 (retire SQLite + delete old tooling) runs only after merge + a stable period, as a separate follow-up PR — never in the cutover merge, so rollback survives the first live days.

Phases

Phase 0 — Prep / unblock

Resolve dup D-035 (renumber one via a freshly-claimed id; fix inbound refs) and R-011 (delete the economics.md copies; keep rejected.md canonical). Re-run pql decisions sync + check-dupes until clean.
Narrow .gitignore: ignore .pql/index.db + .pql/pql.db, track .pql/changelog/.
Set the hook fold-in strategy (.config/hooks/*, not pql's installer).

Phase 1 — Decisions → DQR tree on pql

Restructure decisions/*.md → governance/{decisions,questions,rejected}/<domain>.md (set dqr_dir in .pql/config.yaml). Split mixed D/Q/R domain files into the three trees; carry the decisions/README.md index forward; preserve the per-record format + amendment-in-place convention.
Update literal decisions/ path references (CLAUDE.md, rules, skills). decision_refs are ID-based, so they are unaffected by the path move.
Swap workflow: tooling/db/decision claim/sync → pql decisions claim/sync; add pql decisions validate to pre-commit (the real replacement for the never-implemented check-decision-ids). Keep Python check-dupes wired until pql sync warns on dups (see pql-requirements.md).

Phase 2 — Ticket data migration (the one-way move)

Build a read-only seed script (Python; reuses tooling/db/common.py read path) exporting settledreach.db → .pql/changelog/{tickets,ticket_deps,ticket_labels,ticket_history}/<YYYY-MM>.sql in pql's proven UPSERT format:
```
INSERT INTO tickets (id,type,parent_id,title,description,status,priority,assigned_to,team,
  decision_ref,created_at,updated_at,deleted_at,hash,canonical_version)
VALUES ('T-440',...) ON CONFLICT(id) DO UPDATE SET ...
  WHERE excluded.updated_at > tickets.updated_at
     OR (excluded.updated_at = tickets.updated_at AND excluded.hash > tickets.hash);
```
Transforms: id→'T-'+id, parent_id→'T-'+parent_id; status/priority/type enums copied verbatim (identical between schemas); decision_ref preserved (D-010); deps/labels straight; history ON CONFLICT(hash) DO NOTHING; milestones→labels (ticket_milestones → phase:4 label rows; no milestones table emitted); compute per-row hash + canonical_version.
pql plan rebuild → verification gate: row-count parity (tickets 1013, deps 481, labels 48+, history 16), status distribution parity (backlog 193 / done 755 / cancelled 62 / in_progress 2 / review 1), and 5–10 spot-checks via pql ticket show T-N --with-context (a parent chain, a decision_ref ticket, the single comma-team ticket — pick its primary team).

Phase 3 — `#N → T-N` convention find-replace (safe, tested)

Dedicated transform across markdown + skills/rules/docs (NOT git history): rewrite ticket refs #<digits> → T-<digits> ONLY where they denote tickets. Guard against false positives — markdown headings (# , ## ), anchors (#d-010-…), any non-ticket #. Match roughly (?<![\w#])#(\d+)\b in prose, frontmatter ticket:/ticket_refs:, and the Ticket: #N decision lines. Dry-run diff first → human review → apply.
Update /pr-process to emit T-NNN; update rules/CLAUDE.md convention docs.

Phase 4 — Consumer cutover (single big-bang PR)

/whats-next — pql ticket list --label phase:4 --unblocked (drops the manual dep-walk + the empty milestone_deps query); keep priority/fan-out ranking (fan-out client-side or per requirements); WIP via pql ticket list --status in_progress | count (advisory — pql doesn't enforce); pql ticket status … in_progress; Si-refinement → pql ticket refine list/next/write.
/ticket — all 23 subcommands → pql ticket * / pql plan *; milestone subcommands → label ops; the sqlite-exec UPDATE description step → pql ticket append.
/pr-process — step 8 emits T-NNN, calls pql ticket status T-NNN review.
/pr-review, /git-commit — decision reads → pql decisions show/read.
clerk (tooling/clerk-review, #965) — re-point D-record/ticket-drift checks to pql decisions show/refs + pql ticket show; do this together with the #965 re-enable (don't rewrite dead SQLite-clerk code).
rules/ticket-cli.md, CLAUDE.md — rewrite the command surface to pql; document T-NNN, no-WIP-enforcement, no-state-machine, labels-as-milestones; drop the sqlite3-crash caveat.
Makefile — decision targets → pql decisions …/pql plan status; remove db-backup/db-install.
hooks — .config/hooks/pre-commit: git add .pql/changelog/ + pql decisions validate (+ Python check-dupes until pql warns); add .config/hooks/post-merge/post-checkout: pql plan import / pql plan rebuild + pql decisions sync.
.claude/settings.json — drop SR_DB_PATH; rely on git-root vault discovery (or set PQL_VAULT/PQL_DB); update permission allowlist Bash(tooling/db/*) → Bash(pql *).

Phase 5 — docs/workshops/wiki fold-in

Add decision_refs: frontmatter to docs/workshops/*/workshop-outcomes.md (workshop→decision provenance, currently prose-only) so pql backlinks surfaces the source workshop of a D-record.
Adopt a D-NNN link convention (wikilink [[D-NNN]] or relative md link) where it unlocks backlinks/related; leave generated wiki READ-ONLY sections untouched (frontmatter only).
Verify: pql backlinks governance/decisions/<domain>.md, pql related docs/briefings/<agent>.md, pql context <path> for Si-refinement bundles.

Phase 6 — Retire SQLite (post-merge follow-up PR)

Archive settledreach.db + docs/backups/settledreach.db.backup; delete superseded tooling/db/{ticket,decision,decisions_sync.py,sqlite_connector.py,sqlite-query,sqlite-exec}.

Migration scripting

Script	Input	Output	Idempotent?
`seed_pql_changelog.py` (Phase 2)	`settledreach.db` (read-only via `common.py`)	`.pql/changelog/{tickets,ticket_deps,ticket_labels,ticket_history}/<YYYY-MM>.sql`	Yes — UPSERT + `pql plan rebuild` converges
`convert_ticket_refs.py` (Phase 3)	markdown + skills/rules/docs	in-place `#N → T-N`	Dry-run → review → apply; re-run is a no-op

Both are read-mostly and gated by review before any destructive step. Verification is row/status parity + spot-checks (Phase 2) and a reviewed diff + rg audit (Phase 3).

Benefits

Text changelog kills binary-DB pain — no gitignored binary, no docs/backups/*.db.backup, no main-only make db-backup, no sqlite3-crash caveat. Merges become text-diffable, LWW, idempotent.
Vault queries replace grep in Si refinement / lore-librarian / pr-review: pql search, pql query, pql context <path>, pql related <path>, pql backlinks over the indexed vault.
pql plan whatsnext/review + pql ticket refine map onto existing skill steps — less bespoke logic.
--unblocked built-in replaces the manual dependency walk; ticket show --with-context/--tree gives the implementer bundle /whats-next assembles by hand.
Workshop→decision provenance via frontmatter + pql backlinks once outcomes are linked.

Verification

Decisions: pql decisions list count == markdown record count; pql decisions validate clean; check-dupes clean; spot-check pql decisions show D-010 --with-tickets.
Tickets: the Phase-2 gate; pql plan status matches; pql plan whatsnext returns the same next-batch /whats-next would.
Find-replace: reviewed dry-run diff; post-run rg '#\d+' shows only intended residue.
Hooks: a trivial ticket edit stages .pql/changelog/ in the commit; a simulated merge replays.
Fold-in: pql backlinks on a decision surfaces its source workshop.
End-to-end: a full /whats-next → /pr-process → /pr-review cycle on pql with no SQLite reads.

Critical files

.gitignore; .pql/config.yaml (dqr_dir)
decisions/*.md → governance/{decisions,questions,rejected}/; decisions/README.md
tooling/db/common.py (seed-script read path); new tooling/seed_pql_changelog.py, tooling/convert_ticket_refs.py
.claude/skills/{whats-next,ticket,pr-process,pr-review,git-commit}/SKILL.md
.claude/agents/clerk* + tooling/clerk-review (with #965)
.claude/rules/ticket-cli.md, CLAUDE.md, Makefile, .config/hooks/*, .claude/settings.json
docs/workshops/*/workshop-outcomes.md (provenance frontmatter)

Gaps for the pql team

See pql-requirements.md.

14 KiB Raw Blame History Unescape Escape