docs(meta): pql migration plan + requirements for the pql team

Deep analysis of migrating decisions + ticketing from the SQLite CLI to pql:
- pql-migration.md: phased branch-only plan (decisions→DQR tree, ticket changelog
  seed with T-N≡#N, #N→T-N find-replace, big-bang consumer cutover, docs/workshops/
  wiki fold-in, SQLite retirement) + verification gate + benefits.
- pql-requirements.md: 9 surfaced gaps for the pql team (critical: seeded ticket
  ids + counter-advance; high: dup-id detection, core.hooksPath awareness, changelog
  seed-format docs).

Key finding: milestones are vestigial (milestone_deps empty, 42/1013 tickets
linked) → mapped to a label, dropped as an entity.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-06 11:37:52 +02:00
co-authored by Claude Opus 4.8
parent f880635b24
commit 07e5ba005d
2 changed files with 386 additions and 0 deletions
+215
View File
@@ -0,0 +1,215 @@
---
title: "pql Migration Plan — decisions + ticketing"
description: "Migrate the SQLite-backed ticket system and markdown decisions sync onto pql (markdown-vault indexer + planning tool)"
type: plan
status: approved
created: 2026-06-06
updated: 2026-06-06
decision_refs: []
tags: [migration, tooling, pql, decisions, ticketing]
---
# pql Migration Plan — decisions + ticketing
> **Status:** approved, not yet executed. All work happens on the `pql-migration`
> branch; `main` is untouched until every verification in the merge gate passes.
## Context
Project planning data lives in two places today:
- **Tickets** — a binary SQLite DB at `<repo-parent>/settledreach.db` (gitignored),
with a committed *binary* backup at `docs/backups/settledreach.db.backup` kept fresh
by a main-only `make db-backup` ritual. Accessed via `tooling/db/ticket` (23
subcommands) + `tooling/db/sqlite-query|exec`. 1013 tickets, 481 dependency edges,
48 labels, 16 history rows, 2 milestones.
- **Decisions** — markdown is source-of-truth in `decisions/*.md` (358 D/Q/R records),
*synced into the same SQLite* via `tooling/db/decisions_sync.py` (358 decisions, 1040
cross-refs). ID claiming reserves the next id in the DB before the markdown edit.
Pain points: the binary DB is merge-conflict-prone; the backup ritual is fragile; the
`sqlite3` CLI crashes in this environment (`std::bad_alloc`); and docs/workshops/wiki
context is consumed by grep, not structured query.
**pql** (v1.6.2, `~/.local/bin/pql`; repo already `pql init`-ed — `.pql/` exists, vault
index built, `pql.db` empty) is purpose-built for this:
- **Decisions** — markdown-sourced, `pql decisions sync|validate|claim|list|show|read|refs|coverage`.
- **Tickets** — SQLite-native, versioned by a **git-tracked text changelog**
(`.pql/changelog/<table>/<YYYY-MM>.sql`, last-writer-wins, idempotent replay).
`pql ticket new|list|show|status|assign|team|block|unblock|append|label|board|setparent|refine`
+ `pql plan status|whatsnext|review|export|import|rebuild`.
- **Vault** — `pql query|search|backlinks|related|context|meta|tags|schema` over all
~4385 markdown files (frontmatter/tags/links/headings).
**Outcome:** decisions + tickets fully on pql; docs/workshops/wiki folded into the
queryable vault with provenance links; the old SQLite tooling retired; this doc +
`pql-requirements.md` authored.
## Findings that shaped the plan
- **Milestones are vestigial — do NOT migrate as an entity.** Live: 2 milestones (1 active
"Phase 4" / 42 tickets, 1 completed / 0), `milestone_deps` **0 rows**, 42/1013 tickets
linked, `cascade_phase` + cross-milestone ranking never exercised. → the one active
milestone becomes a **label** (`phase:4`); `/whats-next``pql ticket list --label phase:4 --unblocked`.
- **`#N → T-N` is a clean bijection** — pql's ticket id is a TEXT column, so seed `T-440`
for old `#440`. We **switch the convention to `T-NNN`** and run a safe find-replace of
`#N → T-N` across markdown + skills (git history keeps `#N`, numerically equal to `T-N`).
- **pql's git hooks would be dead here** — repo uses `core.hooksPath=.config/hooks`; pql
installs to `.git/hooks/`. Fold pql steps into `.config/hooks/*` manually; do NOT rely on `pql init`'s hooks.
- **`.gitignore` blanket-ignores `.pql/`** — must narrow to keep `.pql/index.db`/`.pql/pql.db`
ignored but **track `.pql/changelog/`** (the durable artifact).
- **Duplicate D-035** (content.md + perception.md) and **R-011** (rejected.md + economics.md ×2)
exist; `pql decisions sync` silently LWW-collapses them while our `check-dupes` errors. Fix in markdown first.
- **`pql plan import --legacy` is retired** — seed the changelog directly (format below).
- docs/ ~94100% frontmattered; workshops/wiki ~89%; cross-refs are mostly **bare text**
(searchable, but not `backlinks`-able). Workshop→decision provenance is prose-only today.
## Decisions locked
1. **Cutover:** big-bang after a verification gate (no dual-write window).
2. **Decisions layout:** restructure to the DQR governance tree (`governance/{decisions,questions,rejected}/`).
3. **docs/workshops/wiki fold-in:** in-scope for this migration (provenance frontmatter + link convention).
4. **Ticket refs:** switch to `T-NNN`, `T-N ≡ #N`, via a safe tested codebase find-replace.
## Execution safety: branch-only until fully verified
All work on the `pql-migration` branch. `settledreach.db`, its committed backup, and the
`tooling/db/*` CLIs stay intact and operational on `main` the whole time — **rollback =
don't merge.**
**Merge gate (ALL must pass on the branch before the PR merges):**
- Phase-2 ticket verification gate (row/status parity + spot-checks) green.
- `pql decisions validate` + `check-dupes` clean; decision count parity.
- Find-replace dry-run diff reviewed + applied; `rg '#\d+'` residue audited.
- Full cargo/test + lint suite green; hooks fire (changelog staged, replay works).
- A full `/whats-next → /pr-process → /pr-review` dry-run on pql with zero SQLite reads.
- Standard `/pr-review` (Hoshe + Tyre) APPROVED on the migration PR.
Phase 6 (retire SQLite + delete old tooling) runs **only after merge + a stable period**,
as a separate follow-up PR — never in the cutover merge, so rollback survives the first live days.
## Phases
### Phase 0 — Prep / unblock
- Resolve dup **D-035** (renumber one via a freshly-claimed id; fix inbound refs) and **R-011**
(delete the economics.md copies; keep rejected.md canonical). Re-run `pql decisions sync` + `check-dupes` until clean.
- Narrow `.gitignore`: ignore `.pql/index.db` + `.pql/pql.db`, **track `.pql/changelog/`**.
- Set the hook fold-in strategy (`.config/hooks/*`, not pql's installer).
### Phase 1 — Decisions → DQR tree on pql
- Restructure `decisions/*.md``governance/{decisions,questions,rejected}/<domain>.md` (set
`dqr_dir` in `.pql/config.yaml`). Split mixed D/Q/R domain files into the three trees; carry the
`decisions/README.md` index forward; preserve the per-record format + amendment-in-place convention.
- Update literal `decisions/` path references (CLAUDE.md, rules, skills). `decision_refs` are
ID-based, so they are unaffected by the path move.
- Swap workflow: `tooling/db/decision claim/sync``pql decisions claim/sync`; add `pql decisions validate`
to pre-commit (the real replacement for the never-implemented `check-decision-ids`). Keep Python
`check-dupes` wired until pql sync warns on dups (see `pql-requirements.md`).
### Phase 2 — Ticket data migration (the one-way move)
- Build a **read-only seed script** (Python; reuses `tooling/db/common.py` read path) exporting
`settledreach.db``.pql/changelog/{tickets,ticket_deps,ticket_labels,ticket_history}/<YYYY-MM>.sql`
in pql's proven UPSERT format:
```sql
INSERT INTO tickets (id,type,parent_id,title,description,status,priority,assigned_to,team,
decision_ref,created_at,updated_at,deleted_at,hash,canonical_version)
VALUES ('T-440',...) ON CONFLICT(id) DO UPDATE SET ...
WHERE excluded.updated_at > tickets.updated_at
OR (excluded.updated_at = tickets.updated_at AND excluded.hash > tickets.hash);
```
Transforms: `id→'T-'+id`, `parent_id→'T-'+parent_id`; status/priority/type enums copied verbatim
(identical between schemas); `decision_ref` preserved (`D-010`); deps/labels straight;
history `ON CONFLICT(hash) DO NOTHING`; **milestones→labels** (`ticket_milestones` → `phase:4`
label rows; no milestones table emitted); compute per-row `hash` + `canonical_version`.
- `pql plan rebuild` → **verification gate:** row-count parity (tickets 1013, deps 481, labels 48+,
history 16), status distribution parity (backlog 193 / done 755 / cancelled 62 / in_progress 2 /
review 1), and 510 spot-checks via `pql ticket show T-N --with-context` (a parent chain, a
decision_ref ticket, the single comma-team ticket — pick its primary team).
### Phase 3 — `#N → T-N` convention find-replace (safe, tested)
- Dedicated transform across **markdown + skills/rules/docs** (NOT git history): rewrite ticket
refs `#<digits>` → `T-<digits>` ONLY where they denote tickets. Guard against false positives —
markdown headings (`# `, `## `), anchors (`#d-010-…`), any non-ticket `#`. Match roughly
`(?<![\w#])#(\d+)\b` in prose, frontmatter `ticket:`/`ticket_refs:`, and the `Ticket: #N` decision
lines. **Dry-run diff first → human review → apply.**
- Update `/pr-process` to emit `T-NNN`; update rules/CLAUDE.md convention docs.
### Phase 4 — Consumer cutover (single big-bang PR)
- **/whats-next** — `pql ticket list --label phase:4 --unblocked` (drops the manual dep-walk + the
empty milestone_deps query); keep priority/fan-out ranking (fan-out client-side or per requirements);
WIP via `pql ticket list --status in_progress | count` (advisory — pql doesn't enforce);
`pql ticket status … in_progress`; Si-refinement → `pql ticket refine list/next/write`.
- **/ticket** — all 23 subcommands → `pql ticket *` / `pql plan *`; milestone subcommands → label ops;
the `sqlite-exec UPDATE description` step → `pql ticket append`.
- **/pr-process** — step 8 emits `T-NNN`, calls `pql ticket status T-NNN review`.
- **/pr-review, /git-commit** — decision reads → `pql decisions show/read`.
- **clerk (`tooling/clerk-review`, #965)** — re-point D-record/ticket-drift checks to
`pql decisions show/refs` + `pql ticket show`; do this **together with the #965 re-enable** (don't
rewrite dead SQLite-clerk code).
- **rules/ticket-cli.md, CLAUDE.md** — rewrite the command surface to pql; document `T-NNN`,
no-WIP-enforcement, no-state-machine, labels-as-milestones; drop the `sqlite3`-crash caveat.
- **Makefile** — decision targets → `pql decisions …`/`pql plan status`; **remove `db-backup`/`db-install`**.
- **hooks** — `.config/hooks/pre-commit`: `git add .pql/changelog/` + `pql decisions validate`
(+ Python check-dupes until pql warns); add `.config/hooks/post-merge`/`post-checkout`:
`pql plan import` / `pql plan rebuild` + `pql decisions sync`.
- **.claude/settings.json** — drop `SR_DB_PATH`; rely on git-root vault discovery (or set
`PQL_VAULT`/`PQL_DB`); update permission allowlist `Bash(tooling/db/*)` → `Bash(pql *)`.
### Phase 5 — docs/workshops/wiki fold-in
- Add `decision_refs:` frontmatter to `docs/workshops/*/workshop-outcomes.md` (workshop→decision
**provenance**, currently prose-only) so `pql backlinks` surfaces the source workshop of a D-record.
- Adopt a D-NNN link convention (wikilink `[[D-NNN]]` or relative md link) where it unlocks
`backlinks`/`related`; leave generated wiki READ-ONLY sections untouched (frontmatter only).
- Verify: `pql backlinks governance/decisions/<domain>.md`, `pql related docs/briefings/<agent>.md`,
`pql context <path>` for Si-refinement bundles.
### Phase 6 — Retire SQLite (post-merge follow-up PR)
- Archive `settledreach.db` + `docs/backups/settledreach.db.backup`; delete superseded
`tooling/db/{ticket,decision,decisions_sync.py,sqlite_connector.py,sqlite-query,sqlite-exec}`.
## Migration scripting
| Script | Input | Output | Idempotent? |
|--------|-------|--------|-------------|
| `seed_pql_changelog.py` (Phase 2) | `settledreach.db` (read-only via `common.py`) | `.pql/changelog/{tickets,ticket_deps,ticket_labels,ticket_history}/<YYYY-MM>.sql` | Yes — UPSERT + `pql plan rebuild` converges |
| `convert_ticket_refs.py` (Phase 3) | markdown + skills/rules/docs | in-place `#N → T-N` | Dry-run → review → apply; re-run is a no-op |
Both are read-mostly and gated by review before any destructive step. Verification is row/status
parity + spot-checks (Phase 2) and a reviewed diff + `rg` audit (Phase 3).
## Benefits
- **Text changelog kills binary-DB pain** — no gitignored binary, no `docs/backups/*.db.backup`, no
main-only `make db-backup`, no `sqlite3`-crash caveat. Merges become text-diffable, LWW, idempotent.
- **Vault queries replace grep** in Si refinement / lore-librarian / pr-review:
`pql search`, `pql query`, `pql context <path>`, `pql related <path>`, `pql backlinks` over the indexed vault.
- **`pql plan whatsnext/review` + `pql ticket refine`** map onto existing skill steps — less bespoke logic.
- **`--unblocked`** built-in replaces the manual dependency walk; **`ticket show --with-context/--tree`**
gives the implementer bundle `/whats-next` assembles by hand.
- **Workshop→decision provenance** via frontmatter + `pql backlinks` once outcomes are linked.
## Verification
- **Decisions:** `pql decisions list` count == markdown record count; `pql decisions validate` clean;
`check-dupes` clean; spot-check `pql decisions show D-010 --with-tickets`.
- **Tickets:** the Phase-2 gate; `pql plan status` matches; `pql plan whatsnext` returns the same
next-batch `/whats-next` would.
- **Find-replace:** reviewed dry-run diff; post-run `rg '#\d+'` shows only intended residue.
- **Hooks:** a trivial ticket edit stages `.pql/changelog/` in the commit; a simulated merge replays.
- **Fold-in:** `pql backlinks` on a decision surfaces its source workshop.
- **End-to-end:** a full `/whats-next → /pr-process → /pr-review` cycle on pql with no SQLite reads.
## Critical files
- `.gitignore`; `.pql/config.yaml` (`dqr_dir`)
- `decisions/*.md` → `governance/{decisions,questions,rejected}/`; `decisions/README.md`
- `tooling/db/common.py` (seed-script read path); new `tooling/seed_pql_changelog.py`, `tooling/convert_ticket_refs.py`
- `.claude/skills/{whats-next,ticket,pr-process,pr-review,git-commit}/SKILL.md`
- `.claude/agents/clerk*` + `tooling/clerk-review` (with #965)
- `.claude/rules/ticket-cli.md`, `CLAUDE.md`, `Makefile`, `.config/hooks/*`, `.claude/settings.json`
- `docs/workshops/*/workshop-outcomes.md` (provenance frontmatter)
## Gaps for the pql team
See [`pql-requirements.md`](pql-requirements.md).
+171
View File
@@ -0,0 +1,171 @@
---
title: "pql Requirements — gaps surfaced by the Settled Reach migration"
description: "Feature gaps and friction points found while migrating a 1013-ticket + 358-decision project onto pql, for the pql team"
type: requirements
status: draft
created: 2026-06-06
updated: 2026-06-06
tags: [pql, requirements, feedback]
---
# pql Requirements — gaps for the pql team
Compiled while planning a migration of **The Settled Reach** (a Godot+Rust game) off a
SQLite-backed ticket CLI + markdown-decisions-sync onto pql. Scale: **1013 tickets**, 481
dependency edges, 48 labels, 16 history rows; **358 decision records** (237 D / 109 Q / 12 R)
with 1040 cross-refs; a ~4385-file markdown vault (decisions, docs, 22 workshops, ~3500 wiki pages).
pql is a strong fit and most of our system maps cleanly. The items below are the gaps we hit.
Each lists **why we need it**, a **suggested shape**, and a **priority**. Where we have a viable
workaround, it's noted — but the workaround is friction we'd rather not ship.
---
## 1. Seeded / explicit ticket IDs + id-counter advance — **CRITICAL**
**Why.** Our 1013 tickets have integer ids referenced as `#440` in thousands of immutable git
commits, in decision "Ticket: #N" lines, and by our PR automation. We need the pql id to preserve
the number — `T-440 ≡ old #440` — so the mapping stays a trivial bijection. pql's `id` is a TEXT
column, so we can seed `'T-440'` directly into the changelog. **But `pql ticket new` auto-assigns
`T-N` sequentially starting at `T-1`** — after seeding `T-1…T-1013`, the next native create would
mint `T-1` and collide.
**Suggested shape.**
- `pql ticket new --id T-1014` (accept an explicit id), AND/OR
- on `pql plan rebuild`/import, **advance the internal id counter past the max existing id** so
native creates never recycle a seeded id.
**Workaround.** None safe — we'd have to hold all native ticket creation until this lands, or
hand-manage a sentinel. This is the one true blocker for a clean numbered migration.
---
## 2. Duplicate-ID detection in `pql decisions sync` / `validate` — **HIGH**
**Why.** Our markdown had a genuine integrity bug: `D-035` defined in two domain files, `R-011`
defined three times. **`pql decisions sync` silently last-writer-wins-collapsed them** (reported
`broken: 0`), so the duplication was invisible. Our legacy Python `check-dupes` errors on it. A
duplicate canonical-ID is almost always a data bug, not an intended merge.
**Suggested shape.** `pql decisions validate` (and `sync --strict`) should **warn or error on
duplicate D/Q/R ids across the vault**, reporting `file:line` for each occurrence.
**Workaround.** Keep our Python `check-dupes` wired into pre-commit alongside pql — extra tooling
we'd like to retire.
---
## 3. `core.hooksPath` awareness in `pql init` — **HIGH**
**Why.** Our repo sets `core.hooksPath = .config/hooks` (so hooks are version-controlled). `pql init`
installs its pre-commit/post-merge/post-checkout hooks into `.git/hooks/`, which git **ignores** when
`core.hooksPath` is set. pql's changelog-staging + replay hooks would be **silently dead** — the whole
durable-versioning model depends on them. This is a quiet, dangerous failure (data appears to work, but
`.pql/changelog/` is never staged).
**Suggested shape.** `pql init`/`pql doctor` should detect `core.hooksPath` and either install into the
configured dir or emit a loud warning with the manual fold-in snippet.
**Workaround.** Manually fold pql's hook logic into `.config/hooks/*` (we will), but `pql doctor`
should flag the mismatch so others don't get bitten.
---
## 4. Document the changelog seed format + `hash`/`canonical_version` algorithm — **HIGH**
**Why.** `pql plan import --legacy` is documented as retired; the real durable artifact is
`.pql/changelog/<table>/<YYYY-MM>.sql`. To migrate 1013 tickets we must emit those SQL rows directly.
We reverse-engineered the UPSERT shape, but the per-row `hash` and `canonical_version` columns +
the LWW guard (`WHERE excluded.updated_at > … OR (… AND excluded.hash > …)`) are opaque. If our seed
hashes don't match pql's scheme, later native edits still win by `updated_at` (so correctness holds),
but the LWW tie-break becomes unpredictable.
**Suggested shape.** Document (a) the canonical changelog row format per table, (b) the `hash`
input + algorithm, (c) how `canonical_version` increments — OR provide a supported one-shot
`pql plan seed <json|csv>` import that computes them.
**Workaround.** Emit our own stable per-row hash; rely on `updated_at` for correctness. Works, but
under-specified.
---
## 5. Bare-text reference detection / auto-link for `D-NNN` / `T-NNN` — **MEDIUM**
**Why.** Across ~4385 vault files (esp. ~3500 *generated* wiki pages), decision/ticket references are
mostly **bare text** (`D-117`, `#440`) rather than markdown/wikilinks. `pql search` finds them, but
`pql backlinks`/`related` only follow real links — so "what references D-117?" is incomplete. Rewriting
3500+ generated files to add links is impractical (they're regenerated from a pipeline).
**Suggested shape.** Optional **reference recognizers** (configurable regex → entity, e.g.
`D-\d+`→decision, `T-\d+`→ticket) so `backlinks`/`related` treat bare-text mentions as edges without
requiring link syntax in source.
**Workaround.** Adopt a link convention in *authored* files only; accept that generated files are
search-only.
---
## 6. WIP-limit enforcement (advisory) on `ticket status … in_progress` — **MEDIUM**
**Why.** Our kanban flow enforces a WIP limit (max 5 in_progress); our activation step relied on the
CLI warning when exceeded. pql has no forced state machine / WIP concept.
**Suggested shape.** An optional config (`wip_limit: 5`) that makes `pql ticket status <id> in_progress`
emit an advisory warning (not a hard block) when the in_progress count would exceed it.
**Workaround.** Compute `pql ticket list --status in_progress | count` in our skill and warn there.
---
## 7. Fan-out / downstream-unblock count on `ticket show` — **MEDIUM**
**Why.** Our batch-selection ranks ready tickets partly by **fan-out** — how many downstream tickets a
ticket unblocks. We currently compute it from the dependency graph. pql has `--unblocked` (great — it
replaced our manual blocker walk) but no "how many does this unblock" signal.
**Suggested shape.** A `blocks_count` / `unblocks_count` field on `pql ticket show`/`list`, or a
`pql ticket list --orders-by fanout`.
**Workaround.** Compute client-side from `ticket show --with-context` dep data.
---
## 8. Multi-team / array team field — **LOW**
**Why.** One ticket carries a comma-separated `team` (`client,server`). pql treats `team` as a single
string. (1 of 1013 — genuinely low.)
**Suggested shape.** Accept an array/comma team, or document the single-team constraint.
**Workaround.** Pick the primary team for the one affected ticket.
---
## 9. Bless a "labels-as-milestones" pattern (or an optional milestone entity) — **LOW / nice-to-have**
**Why.** Our schema has a milestones subsystem (milestones, ticket_milestones, milestone_deps,
cascade_phase) — but in practice it's **vestigial**: 2 milestones (1 active, 1 completed-with-0-tickets),
`milestone_deps` empty, 42/1013 tickets linked, cascade ordering never used. We're mapping the one
active milestone to a **label** (`phase:4`) and dropping the rest. This works today, so it is **not
blocking**.
**Suggested shape.** Either bless **labels-as-milestones** as the documented pattern (with helper
queries / rollups), or offer an optional lightweight milestone entity (M:N ticket links, status,
ordering, done/total rollups) for projects that actually use phase gating. We'd only need the latter
if we later adopt real cross-phase dependency gating.
---
## Summary
| # | Requirement | Priority | Have workaround? |
|---|-------------|----------|------------------|
| 1 | Seeded/explicit ticket id + counter-advance | **Critical** | No (real blocker) |
| 2 | Duplicate-ID detection in sync/validate | High | Yes (keep our checker) |
| 3 | `core.hooksPath` awareness | High | Yes (manual fold-in) |
| 4 | Documented changelog seed format + hash algo | High | Partial |
| 5 | Bare-text ref detection for backlinks | Medium | Partial (authored files only) |
| 6 | WIP-limit advisory | Medium | Yes (skill-side) |
| 7 | Fan-out / unblocks count | Medium | Yes (client-side) |
| 8 | Multi-team field | Low | Yes |
| 9 | Labels-as-milestones blessing / optional entity | Low | Yes (labels) |