settled-reach/docs/architecture/pql-migration.md

---
title: "pql Migration Plan — decisions + ticketing"
description: "Migrate the SQLite-backed ticket system and markdown decisions sync onto pql (markdown-vault indexer + planning tool)"
type: plan
status: completed
created: 2026-06-06
updated: 2026-06-06
decision_refs: []
tags: [migration, tooling, pql, decisions, ticketing]
---

> **COMPLETED & ARCHIVED (2026-06-06).** This plan shipped: Phases 0–5 in PR #154,
> the phases-as-hierarchy refactor + Phase-6 SQLite retirement in PR #155. It is kept
> as a historical record of *why and how* the planning system moved off the binary
> SQLite DB onto pql. **Current usage** is in `CLAUDE.md`, `.claude/rules/ticket-cli.md`,
> and `governance/README.md`; the one-shot transforms live in `tooling/pql-migrate/`.
> Open feedback for the pql team is in the root `pql-requirements.md`.

# pql Migration Plan — decisions + ticketing

> **Status:** approved, not yet executed. All work happens on the `pql-migration`
> branch; `main` is untouched until every verification in the merge gate passes.

## Context

Project planning data lives in two places today:

- **Tickets** — a binary SQLite DB at `<repo-parent>/settledreach.db` (gitignored),
  with a committed *binary* backup at `docs/backups/settledreach.db.backup` kept fresh
  by a main-only `make db-backup` ritual. Accessed via `tooling/db/ticket` (23
  subcommands) + `tooling/db/sqlite-query|exec`. 1013 tickets, 481 dependency edges,
  48 labels, 16 history rows, 2 milestones.
- **Decisions** — markdown is source-of-truth in `decisions/*.md` (358 D/Q/R records),
  *synced into the same SQLite* via `tooling/db/decisions_sync.py` (358 decisions, 1040
  cross-refs). ID claiming reserves the next id in the DB before the markdown edit.

Pain points: the binary DB is merge-conflict-prone; the backup ritual is fragile; the
`sqlite3` CLI crashes in this environment (`std::bad_alloc`); and docs/workshops/wiki
context is consumed by grep, not structured query.

**pql** (v1.6.2, `~/.local/bin/pql`; repo already `pql init`-ed — `.pql/` exists, vault
index built, `pql.db` empty) is purpose-built for this:

- **Decisions** — markdown-sourced, `pql decisions sync|validate|claim|list|show|read|refs|coverage`.
- **Tickets** — SQLite-native, versioned by a **git-tracked text changelog**
  (`.pql/changelog/<table>/<YYYY-MM>.sql`, last-writer-wins, idempotent replay).
  `pql ticket new|list|show|status|assign|team|block|unblock|append|label|board|setparent|refine`
  + `pql plan status|whatsnext|review|export|import|rebuild`.
- **Vault** — `pql query|search|backlinks|related|context|meta|tags|schema` over all
  ~4385 markdown files (frontmatter/tags/links/headings).

**Outcome:** decisions + tickets fully on pql; docs/workshops/wiki folded into the
queryable vault with provenance links; the old SQLite tooling retired; this doc +
`pql-requirements.md` authored.

## Findings that shaped the plan

- **Milestones are vestigial — do NOT migrate as an entity.** Live: 2 milestones (1 active
  "Phase 4" / 42 tickets, 1 completed / 0), `milestone_deps` **0 rows**, 42/1013 tickets
  linked, `cascade_phase` + cross-milestone ranking never exercised. → the one active
  milestone becomes a **label** (`phase:4`); `/whats-next` → `pql ticket list --label phase:4 --unblocked`.
- **`#N → T-N` is a clean bijection** — pql's ticket id is a TEXT column, so seed `T-440`
  for old `#440`. We **switch the convention to `T-NNN`** and run a safe find-replace of
  `#N → T-N` across markdown + skills (git history keeps `#N`, numerically equal to `T-N`).
- **pql's git hooks would be dead here** — repo uses `core.hooksPath=.config/hooks`; pql
  installs to `.git/hooks/`. Fold pql steps into `.config/hooks/*` manually; do NOT rely on `pql init`'s hooks.
- **`.gitignore` blanket-ignores `.pql/`** — must narrow to keep `.pql/index.db`/`.pql/pql.db`
  ignored but **track `.pql/changelog/`** (the durable artifact).
- **Duplicate D-035** (content.md + perception.md) and **R-011** (rejected.md + economics.md ×2)
  exist; `pql decisions sync` silently LWW-collapses them while our `check-dupes` errors. Fix in markdown first.
- **`pql plan import --legacy` is retired** — seed the changelog directly (format below).
- docs/ ~94–100% frontmattered; workshops/wiki ~89%; cross-refs are mostly **bare text**
  (searchable, but not `backlinks`-able). Workshop→decision provenance is prose-only today.

## Decisions locked

1. **Cutover:** big-bang after a verification gate (no dual-write window).
2. **Decisions layout:** restructure to the DQR governance tree (`governance/{decisions,questions,rejected}/`).
3. **docs/workshops/wiki fold-in:** in-scope for this migration (provenance frontmatter + link convention).
4. **Ticket refs:** switch to `T-NNN`, `T-N ≡ #N`, via a safe tested codebase find-replace.

## Execution safety: branch-only until fully verified

All work on the `pql-migration` branch. `settledreach.db`, its committed backup, and the
`tooling/db/*` CLIs stay intact and operational on `main` the whole time — **rollback =
don't merge.**

**Merge gate (ALL must pass on the branch before the PR merges):**
- Phase-2 ticket verification gate (row/status parity + spot-checks) green.
- `pql decisions validate` + `check-dupes` clean; decision count parity.
- Find-replace dry-run diff reviewed + applied; `rg '#\d+'` residue audited.
- Full cargo/test + lint suite green; hooks fire (changelog staged, replay works).
- A full `/whats-next → /pr-process → /pr-review` dry-run on pql with zero SQLite reads.
- Standard `/pr-review` (Hoshe + Tyre) APPROVED on the migration PR.

Phase 6 (retire SQLite + delete old tooling) runs **only after merge + a stable period**,
as a separate follow-up PR — never in the cutover merge, so rollback survives the first live days.

## Phases

### Phase 0 — Prep / unblock
- Resolve dup **D-035** (renumber one via a freshly-claimed id; fix inbound refs) and **R-011**
  (delete the economics.md copies; keep rejected.md canonical). Re-run `pql decisions sync` + `check-dupes` until clean.
- Narrow `.gitignore`: ignore `.pql/index.db` + `.pql/pql.db`, **track `.pql/changelog/`**.
- Set the hook fold-in strategy (`.config/hooks/*`, not pql's installer).

### Phase 1 — Decisions → DQR tree on pql
- Restructure `decisions/*.md` → `governance/{decisions,questions,rejected}/<domain>.md` (set
  `dqr_dir` in `.pql/config.yaml`). Split mixed D/Q/R domain files into the three trees; carry the
  `decisions/README.md` index forward; preserve the per-record format + amendment-in-place convention.
- Update literal `decisions/` path references (CLAUDE.md, rules, skills). `decision_refs` are
  ID-based, so they are unaffected by the path move.
- Swap workflow: `tooling/db/decision claim/sync` → `pql decisions claim/sync`; add `pql decisions validate`
  to pre-commit (the real replacement for the never-implemented `check-decision-ids`). Keep Python
  `check-dupes` wired until pql sync warns on dups (see `pql-requirements.md`).

### Phase 2 — Ticket data migration (the one-way move)
- Build a **read-only seed script** (Python; reuses `tooling/db/common.py` read path) exporting
  `settledreach.db` → `.pql/changelog/{tickets,ticket_deps,ticket_labels,ticket_history}/<YYYY-MM>.sql`
  in pql's proven UPSERT format:
  ```sql
  INSERT INTO tickets (id,type,parent_id,title,description,status,priority,assigned_to,team,
    decision_ref,created_at,updated_at,deleted_at,hash,canonical_version)
  VALUES ('T-440',...) ON CONFLICT(id) DO UPDATE SET ...
    WHERE excluded.updated_at > tickets.updated_at
       OR (excluded.updated_at = tickets.updated_at AND excluded.hash > tickets.hash);
  ```
  Transforms: `id→'T-'+id`, `parent_id→'T-'+parent_id`; status/priority/type enums copied verbatim
  (identical between schemas); `decision_ref` preserved (`D-010`); deps/labels straight;
  history `ON CONFLICT(hash) DO NOTHING`; **milestones→labels** (`ticket_milestones` → `phase:4`
  label rows; no milestones table emitted); compute per-row `hash` + `canonical_version`.
- `pql plan rebuild` → **verification gate:** row-count parity (tickets 1013, deps 481, labels 48+,
  history 16), status distribution parity (backlog 193 / done 755 / cancelled 62 / in_progress 2 /
  review 1), and 5–10 spot-checks via `pql ticket show T-N --with-context` (a parent chain, a
  decision_ref ticket, the single comma-team ticket — pick its primary team).

### Phase 3 — `#N → T-N` convention find-replace (safe, tested)
- Dedicated transform across **markdown + skills/rules/docs** (NOT git history): rewrite ticket
  refs `#<digits>` → `T-<digits>` ONLY where they denote tickets. Guard against false positives —
  markdown headings (`# `, `## `), anchors (`#d-010-…`), any non-ticket `#`. Match roughly
  `(?<![\w#])#(\d+)\b` in prose, frontmatter `ticket:`/`ticket_refs:`, and the `Ticket: #N` decision
  lines. **Dry-run diff first → human review → apply.**
- Update `/pr-process` to emit `T-NNN`; update rules/CLAUDE.md convention docs.

### Phase 4 — Consumer cutover (single big-bang PR)
- **/whats-next** — `pql ticket list --label phase:4 --unblocked` (drops the manual dep-walk + the
  empty milestone_deps query); keep priority/fan-out ranking (fan-out client-side or per requirements);
  WIP via `pql ticket list --status in_progress | count` (advisory — pql doesn't enforce);
  `pql ticket status … in_progress`; Si-refinement → `pql ticket refine list/next/write`.
- **/ticket** — all 23 subcommands → `pql ticket *` / `pql plan *`; milestone subcommands → label ops;
  the `sqlite-exec UPDATE description` step → `pql ticket append`.
- **/pr-process** — step 8 emits `T-NNN`, calls `pql ticket status T-NNN review`.
- **/pr-review, /git-commit** — decision reads → `pql decisions show/read`.
- **clerk (`tooling/clerk-review`, #965)** — re-point D-record/ticket-drift checks to
  `pql decisions show/refs` + `pql ticket show`; do this **together with the #965 re-enable** (don't
  rewrite dead SQLite-clerk code).
- **rules/ticket-cli.md, CLAUDE.md** — rewrite the command surface to pql; document `T-NNN`,
  no-WIP-enforcement, no-state-machine, labels-as-milestones; drop the `sqlite3`-crash caveat.
- **Makefile** — decision targets → `pql decisions …`/`pql plan status`; **remove `db-backup`/`db-install`**.
- **hooks** — `.config/hooks/pre-commit`: `git add .pql/changelog/` + `pql decisions validate`
  (+ Python check-dupes until pql warns); add `.config/hooks/post-merge`/`post-checkout`:
  `pql plan import` / `pql plan rebuild` + `pql decisions sync`.
- **.claude/settings.json** — drop `SR_DB_PATH`; rely on git-root vault discovery (or set
  `PQL_VAULT`/`PQL_DB`); update permission allowlist `Bash(tooling/db/*)` → `Bash(pql *)`.

### Phase 5 — docs/workshops/wiki fold-in
- Add `decision_refs:` frontmatter to `docs/workshops/*/workshop-outcomes.md` (workshop→decision
  **provenance**, currently prose-only) so `pql backlinks` surfaces the source workshop of a D-record.
- Adopt a D-NNN link convention (wikilink `[[D-NNN]]` or relative md link) where it unlocks
  `backlinks`/`related`; leave generated wiki READ-ONLY sections untouched (frontmatter only).
- Verify: `pql backlinks governance/decisions/<domain>.md`, `pql related docs/briefings/<agent>.md`,
  `pql context <path>` for Si-refinement bundles.

### Phase 6 — Retire SQLite (post-merge follow-up PR)
- Archive `settledreach.db` + `docs/backups/settledreach.db.backup`; delete superseded
  `tooling/db/{ticket,decision,decisions_sync.py,sqlite_connector.py,sqlite-query,sqlite-exec}`.

## Migration scripting

| Script | Input | Output | Idempotent? |
|--------|-------|--------|-------------|
| `seed_pql_changelog.py` (Phase 2) | `settledreach.db` (read-only via `common.py`) | `.pql/changelog/{tickets,ticket_deps,ticket_labels,ticket_history}/<YYYY-MM>.sql` | Yes — UPSERT + `pql plan rebuild` converges |
| `convert_ticket_refs.py` (Phase 3) | markdown + skills/rules/docs | in-place `#N → T-N` | Dry-run → review → apply; re-run is a no-op |

Both are read-mostly and gated by review before any destructive step. Verification is row/status
parity + spot-checks (Phase 2) and a reviewed diff + `rg` audit (Phase 3).

## Benefits

- **Text changelog kills binary-DB pain** — no gitignored binary, no `docs/backups/*.db.backup`, no
  main-only `make db-backup`, no `sqlite3`-crash caveat. Merges become text-diffable, LWW, idempotent.
- **Vault queries replace grep** in Si refinement / lore-librarian / pr-review:
  `pql search`, `pql query`, `pql context <path>`, `pql related <path>`, `pql backlinks` over the indexed vault.
- **`pql plan whatsnext/review` + `pql ticket refine`** map onto existing skill steps — less bespoke logic.
- **`--unblocked`** built-in replaces the manual dependency walk; **`ticket show --with-context/--tree`**
  gives the implementer bundle `/whats-next` assembles by hand.
- **Workshop→decision provenance** via frontmatter + `pql backlinks` once outcomes are linked.

## Verification

- **Decisions:** `pql decisions list` count == markdown record count; `pql decisions validate` clean;
  `check-dupes` clean; spot-check `pql decisions show D-010 --with-tickets`.
- **Tickets:** the Phase-2 gate; `pql plan status` matches; `pql plan whatsnext` returns the same
  next-batch `/whats-next` would.
- **Find-replace:** reviewed dry-run diff; post-run `rg '#\d+'` shows only intended residue.
- **Hooks:** a trivial ticket edit stages `.pql/changelog/` in the commit; a simulated merge replays.
- **Fold-in:** `pql backlinks` on a decision surfaces its source workshop.
- **End-to-end:** a full `/whats-next → /pr-process → /pr-review` cycle on pql with no SQLite reads.

## Critical files

- `.gitignore`; `.pql/config.yaml` (`dqr_dir`)
- `decisions/*.md` → `governance/{decisions,questions,rejected}/`; `decisions/README.md`
- `tooling/db/common.py` (seed-script read path); new `tooling/seed_pql_changelog.py`, `tooling/convert_ticket_refs.py`
- `.claude/skills/{whats-next,ticket,pr-process,pr-review,git-commit}/SKILL.md`
- `.claude/agents/clerk*` + `tooling/clerk-review` (with #965)
- `.claude/rules/ticket-cli.md`, `CLAUDE.md`, `Makefile`, `.config/hooks/*`, `.claude/settings.json`
- `docs/workshops/*/workshop-outcomes.md` (provenance frontmatter)

## Gaps for the pql team

See [`pql-requirements.md`](pql-requirements.md).