settled-reach/docs/architecture/pql-migration.md

title, description, type, status, created, updated, decision_refs, tags

title	description	type	status	created	updated	decision_refs	tags
pql Migration Plan — decisions + ticketing	Migrate the SQLite-backed ticket system and markdown decisions sync onto pql (markdown-vault indexer + planning tool)	plan	completed	2026-06-06	2026-06-06		migration tooling pql decisions ticketing

COMPLETED & ARCHIVED (2026-06-06). This plan shipped: Phases 0–5 in PR #154, the phases-as-hierarchy refactor + Phase-6 SQLite retirement in PR #155. It is kept as a historical record of why and how the planning system moved off the binary SQLite DB onto pql. Current usage is in CLAUDE.md, .claude/rules/ticket-cli.md, and governance/README.md; the one-shot transforms live in tooling/pql-migrate/. Open feedback for the pql team is in the root pql-requirements.md.

pql Migration Plan — decisions + ticketing

Status: approved, not yet executed. All work happens on the pql-migration branch; main is untouched until every verification in the merge gate passes.

Context

Project planning data lives in two places today:

• Tickets — a binary SQLite DB at <repo-parent>/settledreach.db (gitignored), with a committed binary backup at docs/backups/settledreach.db.backup kept fresh by a main-only make db-backup ritual. Accessed via tooling/db/ticket (23 subcommands) + tooling/db/sqlite-query|exec. 1013 tickets, 481 dependency edges, 48 labels, 16 history rows, 2 milestones.
• Decisions — markdown is source-of-truth in decisions/*.md (358 D/Q/R records), synced into the same SQLite via tooling/db/decisions_sync.py (358 decisions, 1040 cross-refs). ID claiming reserves the next id in the DB before the markdown edit.

Pain points: the binary DB is merge-conflict-prone; the backup ritual is fragile; the sqlite3 CLI crashes in this environment (std::bad_alloc); and docs/workshops/wiki context is consumed by grep, not structured query.

pql (v1.6.2, ~/.local/bin/pql; repo already pql init-ed — .pql/ exists, vault index built, pql.db empty) is purpose-built for this:

• Decisions — markdown-sourced, pql decisions sync|validate|claim|list|show|read|refs|coverage.
• Tickets — SQLite-native, versioned by a git-tracked text changelog (.pql/changelog/<table>/<YYYY-MM>.sql, last-writer-wins, idempotent replay). pql ticket new|list|show|status|assign|team|block|unblock|append|label|board|setparent|refine
  • pql plan status|whatsnext|review|export|import|rebuild.
• Vault — pql query|search|backlinks|related|context|meta|tags|schema over all ~4385 markdown files (frontmatter/tags/links/headings).

Outcome: decisions + tickets fully on pql; docs/workshops/wiki folded into the queryable vault with provenance links; the old SQLite tooling retired; this doc + pql-requirements.md authored.

Findings that shaped the plan

• Milestones are vestigial — do NOT migrate as an entity. Live: 2 milestones (1 active "Phase 4" / 42 tickets, 1 completed / 0), milestone_deps 0 rows, 42/1013 tickets linked, cascade_phase + cross-milestone ranking never exercised. → the one active milestone becomes a label (phase:4); /whats-next → pql ticket list --label phase:4 --unblocked.
• #N → T-N is a clean bijection — pql's ticket id is a TEXT column, so seed T-440 for old #440. We switch the convention to T-NNN and run a safe find-replace of #N → T-N across markdown + skills (git history keeps #N, numerically equal to T-N).
• pql's git hooks would be dead here — repo uses core.hooksPath=.config/hooks; pql installs to .git/hooks/. Fold pql steps into .config/hooks/* manually; do NOT rely on pql init's hooks.
• .gitignore blanket-ignores .pql/ — must narrow to keep .pql/index.db/.pql/pql.db ignored but track .pql/changelog/ (the durable artifact).
• Duplicate D-035 (content.md + perception.md) and R-011 (rejected.md + economics.md ×2) exist; pql decisions sync silently LWW-collapses them while our check-dupes errors. Fix in markdown first.
• pql plan import --legacy is retired — seed the changelog directly (format below).
• docs/ ~94–100% frontmattered; workshops/wiki ~89%; cross-refs are mostly bare text (searchable, but not backlinks-able). Workshop→decision provenance is prose-only today.

Decisions locked

1. Cutover: big-bang after a verification gate (no dual-write window).
2. Decisions layout: restructure to the DQR governance tree (governance/{decisions,questions,rejected}/).
3. docs/workshops/wiki fold-in: in-scope for this migration (provenance frontmatter + link convention).
4. Ticket refs: switch to T-NNN, T-N ≡ #N, via a safe tested codebase find-replace.

Execution safety: branch-only until fully verified

All work on the pql-migration branch. settledreach.db, its committed backup, and the tooling/db/* CLIs stay intact and operational on main the whole time — rollback = don't merge.

Merge gate (ALL must pass on the branch before the PR merges):

• Phase-2 ticket verification gate (row/status parity + spot-checks) green.
• pql decisions validate + check-dupes clean; decision count parity.
• Find-replace dry-run diff reviewed + applied; rg '#\d+' residue audited.
• Full cargo/test + lint suite green; hooks fire (changelog staged, replay works).
• A full /whats-next → /pr-process → /pr-review dry-run on pql with zero SQLite reads.
• Standard /pr-review (Hoshe + Tyre) APPROVED on the migration PR.

Phase 6 (retire SQLite + delete old tooling) runs only after merge + a stable period, as a separate follow-up PR — never in the cutover merge, so rollback survives the first live days.

Phases

Phase 0 — Prep / unblock

• Resolve dup D-035 (renumber one via a freshly-claimed id; fix inbound refs) and R-011 (delete the economics.md copies; keep rejected.md canonical). Re-run pql decisions sync + check-dupes until clean.
• Narrow .gitignore: ignore .pql/index.db + .pql/pql.db, track .pql/changelog/.
• Set the hook fold-in strategy (.config/hooks/*, not pql's installer).

Phase 1 — Decisions → DQR tree on pql

• Restructure decisions/*.md → governance/{decisions,questions,rejected}/<domain>.md (set dqr_dir in .pql/config.yaml). Split mixed D/Q/R domain files into the three trees; carry the decisions/README.md index forward; preserve the per-record format + amendment-in-place convention.
• Update literal decisions/ path references (CLAUDE.md, rules, skills). decision_refs are ID-based, so they are unaffected by the path move.
• Swap workflow: tooling/db/decision claim/sync → pql decisions claim/sync; add pql decisions validate to pre-commit (the real replacement for the never-implemented check-decision-ids). Keep Python check-dupes wired until pql sync warns on dups (see pql-requirements.md).

Phase 2 — Ticket data migration (the one-way move)

• Build a read-only seed script (Python; reuses tooling/db/common.py read path) exporting settledreach.db → .pql/changelog/{tickets,ticket_deps,ticket_labels,ticket_history}/<YYYY-MM>.sql in pql's proven UPSERT format:

  INSERT INTO tickets (id,type,parent_id,title,description,status,priority,assigned_to,team,
    decision_ref,created_at,updated_at,deleted_at,hash,canonical_version)
  VALUES ('T-440',...) ON CONFLICT(id) DO UPDATE SET ...
    WHERE excluded.updated_at > tickets.updated_at
       OR (excluded.updated_at = tickets.updated_at AND excluded.hash > tickets.hash);
  

  Transforms: id→'T-'+id, parent_id→'T-'+parent_id; status/priority/type enums copied verbatim (identical between schemas); decision_ref preserved (D-010); deps/labels straight; history ON CONFLICT(hash) DO NOTHING; milestones→labels (ticket_milestones → phase:4 label rows; no milestones table emitted); compute per-row hash + canonical_version.
• pql plan rebuild → verification gate: row-count parity (tickets 1013, deps 481, labels 48+, history 16), status distribution parity (backlog 193 / done 755 / cancelled 62 / in_progress 2 / review 1), and 5–10 spot-checks via pql ticket show T-N --with-context (a parent chain, a decision_ref ticket, the single comma-team ticket — pick its primary team).

Phase 3 — #N → T-N convention find-replace (safe, tested)

• Dedicated transform across markdown + skills/rules/docs (NOT git history): rewrite ticket refs #<digits> → T-<digits> ONLY where they denote tickets. Guard against false positives — markdown headings (# , ## ), anchors (#d-010-…), any non-ticket #. Match roughly (?<![\w#])#(\d+)\b in prose, frontmatter ticket:/ticket_refs:, and the Ticket: #N decision lines. Dry-run diff first → human review → apply.
• Update /pr-process to emit T-NNN; update rules/CLAUDE.md convention docs.

Phase 4 — Consumer cutover (single big-bang PR)

• /whats-next — pql ticket list --label phase:4 --unblocked (drops the manual dep-walk + the empty milestone_deps query); keep priority/fan-out ranking (fan-out client-side or per requirements); WIP via pql ticket list --status in_progress | count (advisory — pql doesn't enforce); pql ticket status … in_progress; Si-refinement → pql ticket refine list/next/write.
• /ticket — all 23 subcommands → pql ticket * / pql plan *; milestone subcommands → label ops; the sqlite-exec UPDATE description step → pql ticket append.
• /pr-process — step 8 emits T-NNN, calls pql ticket status T-NNN review.
• /pr-review, /git-commit — decision reads → pql decisions show/read.
• clerk (tooling/clerk-review, #965) — re-point D-record/ticket-drift checks to pql decisions show/refs + pql ticket show; do this together with the #965 re-enable (don't rewrite dead SQLite-clerk code).
• rules/ticket-cli.md, CLAUDE.md — rewrite the command surface to pql; document T-NNN, no-WIP-enforcement, no-state-machine, labels-as-milestones; drop the sqlite3-crash caveat.
• Makefile — decision targets → pql decisions …/pql plan status; remove db-backup/db-install.
• hooks — .config/hooks/pre-commit: git add .pql/changelog/ + pql decisions validate (+ Python check-dupes until pql warns); add .config/hooks/post-merge/post-checkout: pql plan import / pql plan rebuild + pql decisions sync.
• .claude/settings.json — drop SR_DB_PATH; rely on git-root vault discovery (or set PQL_VAULT/PQL_DB); update permission allowlist Bash(tooling/db/*) → Bash(pql *).

Phase 5 — docs/workshops/wiki fold-in

• Add decision_refs: frontmatter to docs/workshops/*/workshop-outcomes.md (workshop→decision provenance, currently prose-only) so pql backlinks surfaces the source workshop of a D-record.
• Adopt a D-NNN link convention (wikilink [[D-NNN]] or relative md link) where it unlocks backlinks/related; leave generated wiki READ-ONLY sections untouched (frontmatter only).
• Verify: pql backlinks governance/decisions/<domain>.md, pql related docs/briefings/<agent>.md, pql context <path> for Si-refinement bundles.

Phase 6 — Retire SQLite (post-merge follow-up PR)

• Archive settledreach.db + docs/backups/settledreach.db.backup; delete superseded tooling/db/{ticket,decision,decisions_sync.py,sqlite_connector.py,sqlite-query,sqlite-exec}.

Migration scripting

Script	Input	Output	Idempotent?
seed_pql_changelog.py (Phase 2)	settledreach.db (read-only via common.py)	.pql/changelog/{tickets,ticket_deps,ticket_labels,ticket_history}/<YYYY-MM>.sql	Yes — UPSERT + pql plan rebuild converges
convert_ticket_refs.py (Phase 3)	markdown + skills/rules/docs	in-place #N → T-N	Dry-run → review → apply; re-run is a no-op

Both are read-mostly and gated by review before any destructive step. Verification is row/status parity + spot-checks (Phase 2) and a reviewed diff + rg audit (Phase 3).

Benefits

• Text changelog kills binary-DB pain — no gitignored binary, no docs/backups/*.db.backup, no main-only make db-backup, no sqlite3-crash caveat. Merges become text-diffable, LWW, idempotent.
• Vault queries replace grep in Si refinement / lore-librarian / pr-review: pql search, pql query, pql context <path>, pql related <path>, pql backlinks over the indexed vault.
• pql plan whatsnext/review + pql ticket refine map onto existing skill steps — less bespoke logic.
• --unblocked built-in replaces the manual dependency walk; ticket show --with-context/--tree gives the implementer bundle /whats-next assembles by hand.
• Workshop→decision provenance via frontmatter + pql backlinks once outcomes are linked.

Verification

• Decisions: pql decisions list count == markdown record count; pql decisions validate clean; check-dupes clean; spot-check pql decisions show D-010 --with-tickets.
• Tickets: the Phase-2 gate; pql plan status matches; pql plan whatsnext returns the same next-batch /whats-next would.
• Find-replace: reviewed dry-run diff; post-run rg '#\d+' shows only intended residue.
• Hooks: a trivial ticket edit stages .pql/changelog/ in the commit; a simulated merge replays.
• Fold-in: pql backlinks on a decision surfaces its source workshop.
• End-to-end: a full /whats-next → /pr-process → /pr-review cycle on pql with no SQLite reads.

Critical files

• .gitignore; .pql/config.yaml (dqr_dir)
• decisions/*.md → governance/{decisions,questions,rejected}/; decisions/README.md
• tooling/db/common.py (seed-script read path); new tooling/seed_pql_changelog.py, tooling/convert_ticket_refs.py
• .claude/skills/{whats-next,ticket,pr-process,pr-review,git-commit}/SKILL.md
• .claude/agents/clerk* + tooling/clerk-review (with #965)
• .claude/rules/ticket-cli.md, CLAUDE.md, Makefile, .config/hooks/*, .claude/settings.json
• docs/workshops/*/workshop-outcomes.md (provenance frontmatter)

Gaps for the pql team

See pql-requirements.md.