diff --git a/pql-migration.md b/pql-migration.md new file mode 100644 index 000000000..667be61f6 --- /dev/null +++ b/pql-migration.md @@ -0,0 +1,215 @@ +--- +title: "pql Migration Plan — decisions + ticketing" +description: "Migrate the SQLite-backed ticket system and markdown decisions sync onto pql (markdown-vault indexer + planning tool)" +type: plan +status: approved +created: 2026-06-06 +updated: 2026-06-06 +decision_refs: [] +tags: [migration, tooling, pql, decisions, ticketing] +--- + +# pql Migration Plan — decisions + ticketing + +> **Status:** approved, not yet executed. All work happens on the `pql-migration` +> branch; `main` is untouched until every verification in the merge gate passes. + +## Context + +Project planning data lives in two places today: + +- **Tickets** — a binary SQLite DB at `/settledreach.db` (gitignored), + with a committed *binary* backup at `docs/backups/settledreach.db.backup` kept fresh + by a main-only `make db-backup` ritual. Accessed via `tooling/db/ticket` (23 + subcommands) + `tooling/db/sqlite-query|exec`. 1013 tickets, 481 dependency edges, + 48 labels, 16 history rows, 2 milestones. +- **Decisions** — markdown is source-of-truth in `decisions/*.md` (358 D/Q/R records), + *synced into the same SQLite* via `tooling/db/decisions_sync.py` (358 decisions, 1040 + cross-refs). ID claiming reserves the next id in the DB before the markdown edit. + +Pain points: the binary DB is merge-conflict-prone; the backup ritual is fragile; the +`sqlite3` CLI crashes in this environment (`std::bad_alloc`); and docs/workshops/wiki +context is consumed by grep, not structured query. + +**pql** (v1.6.2, `~/.local/bin/pql`; repo already `pql init`-ed — `.pql/` exists, vault +index built, `pql.db` empty) is purpose-built for this: + +- **Decisions** — markdown-sourced, `pql decisions sync|validate|claim|list|show|read|refs|coverage`. +- **Tickets** — SQLite-native, versioned by a **git-tracked text changelog** + (`.pql/changelog//.sql`, last-writer-wins, idempotent replay). + `pql ticket new|list|show|status|assign|team|block|unblock|append|label|board|setparent|refine` + + `pql plan status|whatsnext|review|export|import|rebuild`. +- **Vault** — `pql query|search|backlinks|related|context|meta|tags|schema` over all + ~4385 markdown files (frontmatter/tags/links/headings). + +**Outcome:** decisions + tickets fully on pql; docs/workshops/wiki folded into the +queryable vault with provenance links; the old SQLite tooling retired; this doc + +`pql-requirements.md` authored. + +## Findings that shaped the plan + +- **Milestones are vestigial — do NOT migrate as an entity.** Live: 2 milestones (1 active + "Phase 4" / 42 tickets, 1 completed / 0), `milestone_deps` **0 rows**, 42/1013 tickets + linked, `cascade_phase` + cross-milestone ranking never exercised. → the one active + milestone becomes a **label** (`phase:4`); `/whats-next` → `pql ticket list --label phase:4 --unblocked`. +- **`#N → T-N` is a clean bijection** — pql's ticket id is a TEXT column, so seed `T-440` + for old `#440`. We **switch the convention to `T-NNN`** and run a safe find-replace of + `#N → T-N` across markdown + skills (git history keeps `#N`, numerically equal to `T-N`). +- **pql's git hooks would be dead here** — repo uses `core.hooksPath=.config/hooks`; pql + installs to `.git/hooks/`. Fold pql steps into `.config/hooks/*` manually; do NOT rely on `pql init`'s hooks. +- **`.gitignore` blanket-ignores `.pql/`** — must narrow to keep `.pql/index.db`/`.pql/pql.db` + ignored but **track `.pql/changelog/`** (the durable artifact). +- **Duplicate D-035** (content.md + perception.md) and **R-011** (rejected.md + economics.md ×2) + exist; `pql decisions sync` silently LWW-collapses them while our `check-dupes` errors. Fix in markdown first. +- **`pql plan import --legacy` is retired** — seed the changelog directly (format below). +- docs/ ~94–100% frontmattered; workshops/wiki ~89%; cross-refs are mostly **bare text** + (searchable, but not `backlinks`-able). Workshop→decision provenance is prose-only today. + +## Decisions locked + +1. **Cutover:** big-bang after a verification gate (no dual-write window). +2. **Decisions layout:** restructure to the DQR governance tree (`governance/{decisions,questions,rejected}/`). +3. **docs/workshops/wiki fold-in:** in-scope for this migration (provenance frontmatter + link convention). +4. **Ticket refs:** switch to `T-NNN`, `T-N ≡ #N`, via a safe tested codebase find-replace. + +## Execution safety: branch-only until fully verified + +All work on the `pql-migration` branch. `settledreach.db`, its committed backup, and the +`tooling/db/*` CLIs stay intact and operational on `main` the whole time — **rollback = +don't merge.** + +**Merge gate (ALL must pass on the branch before the PR merges):** +- Phase-2 ticket verification gate (row/status parity + spot-checks) green. +- `pql decisions validate` + `check-dupes` clean; decision count parity. +- Find-replace dry-run diff reviewed + applied; `rg '#\d+'` residue audited. +- Full cargo/test + lint suite green; hooks fire (changelog staged, replay works). +- A full `/whats-next → /pr-process → /pr-review` dry-run on pql with zero SQLite reads. +- Standard `/pr-review` (Hoshe + Tyre) APPROVED on the migration PR. + +Phase 6 (retire SQLite + delete old tooling) runs **only after merge + a stable period**, +as a separate follow-up PR — never in the cutover merge, so rollback survives the first live days. + +## Phases + +### Phase 0 — Prep / unblock +- Resolve dup **D-035** (renumber one via a freshly-claimed id; fix inbound refs) and **R-011** + (delete the economics.md copies; keep rejected.md canonical). Re-run `pql decisions sync` + `check-dupes` until clean. +- Narrow `.gitignore`: ignore `.pql/index.db` + `.pql/pql.db`, **track `.pql/changelog/`**. +- Set the hook fold-in strategy (`.config/hooks/*`, not pql's installer). + +### Phase 1 — Decisions → DQR tree on pql +- Restructure `decisions/*.md` → `governance/{decisions,questions,rejected}/.md` (set + `dqr_dir` in `.pql/config.yaml`). Split mixed D/Q/R domain files into the three trees; carry the + `decisions/README.md` index forward; preserve the per-record format + amendment-in-place convention. +- Update literal `decisions/` path references (CLAUDE.md, rules, skills). `decision_refs` are + ID-based, so they are unaffected by the path move. +- Swap workflow: `tooling/db/decision claim/sync` → `pql decisions claim/sync`; add `pql decisions validate` + to pre-commit (the real replacement for the never-implemented `check-decision-ids`). Keep Python + `check-dupes` wired until pql sync warns on dups (see `pql-requirements.md`). + +### Phase 2 — Ticket data migration (the one-way move) +- Build a **read-only seed script** (Python; reuses `tooling/db/common.py` read path) exporting + `settledreach.db` → `.pql/changelog/{tickets,ticket_deps,ticket_labels,ticket_history}/.sql` + in pql's proven UPSERT format: + ```sql + INSERT INTO tickets (id,type,parent_id,title,description,status,priority,assigned_to,team, + decision_ref,created_at,updated_at,deleted_at,hash,canonical_version) + VALUES ('T-440',...) ON CONFLICT(id) DO UPDATE SET ... + WHERE excluded.updated_at > tickets.updated_at + OR (excluded.updated_at = tickets.updated_at AND excluded.hash > tickets.hash); + ``` + Transforms: `id→'T-'+id`, `parent_id→'T-'+parent_id`; status/priority/type enums copied verbatim + (identical between schemas); `decision_ref` preserved (`D-010`); deps/labels straight; + history `ON CONFLICT(hash) DO NOTHING`; **milestones→labels** (`ticket_milestones` → `phase:4` + label rows; no milestones table emitted); compute per-row `hash` + `canonical_version`. +- `pql plan rebuild` → **verification gate:** row-count parity (tickets 1013, deps 481, labels 48+, + history 16), status distribution parity (backlog 193 / done 755 / cancelled 62 / in_progress 2 / + review 1), and 5–10 spot-checks via `pql ticket show T-N --with-context` (a parent chain, a + decision_ref ticket, the single comma-team ticket — pick its primary team). + +### Phase 3 — `#N → T-N` convention find-replace (safe, tested) +- Dedicated transform across **markdown + skills/rules/docs** (NOT git history): rewrite ticket + refs `#` → `T-` ONLY where they denote tickets. Guard against false positives — + markdown headings (`# `, `## `), anchors (`#d-010-…`), any non-ticket `#`. Match roughly + `(?.md`, `pql related docs/briefings/.md`, + `pql context ` for Si-refinement bundles. + +### Phase 6 — Retire SQLite (post-merge follow-up PR) +- Archive `settledreach.db` + `docs/backups/settledreach.db.backup`; delete superseded + `tooling/db/{ticket,decision,decisions_sync.py,sqlite_connector.py,sqlite-query,sqlite-exec}`. + +## Migration scripting + +| Script | Input | Output | Idempotent? | +|--------|-------|--------|-------------| +| `seed_pql_changelog.py` (Phase 2) | `settledreach.db` (read-only via `common.py`) | `.pql/changelog/{tickets,ticket_deps,ticket_labels,ticket_history}/.sql` | Yes — UPSERT + `pql plan rebuild` converges | +| `convert_ticket_refs.py` (Phase 3) | markdown + skills/rules/docs | in-place `#N → T-N` | Dry-run → review → apply; re-run is a no-op | + +Both are read-mostly and gated by review before any destructive step. Verification is row/status +parity + spot-checks (Phase 2) and a reviewed diff + `rg` audit (Phase 3). + +## Benefits + +- **Text changelog kills binary-DB pain** — no gitignored binary, no `docs/backups/*.db.backup`, no + main-only `make db-backup`, no `sqlite3`-crash caveat. Merges become text-diffable, LWW, idempotent. +- **Vault queries replace grep** in Si refinement / lore-librarian / pr-review: + `pql search`, `pql query`, `pql context `, `pql related `, `pql backlinks` over the indexed vault. +- **`pql plan whatsnext/review` + `pql ticket refine`** map onto existing skill steps — less bespoke logic. +- **`--unblocked`** built-in replaces the manual dependency walk; **`ticket show --with-context/--tree`** + gives the implementer bundle `/whats-next` assembles by hand. +- **Workshop→decision provenance** via frontmatter + `pql backlinks` once outcomes are linked. + +## Verification + +- **Decisions:** `pql decisions list` count == markdown record count; `pql decisions validate` clean; + `check-dupes` clean; spot-check `pql decisions show D-010 --with-tickets`. +- **Tickets:** the Phase-2 gate; `pql plan status` matches; `pql plan whatsnext` returns the same + next-batch `/whats-next` would. +- **Find-replace:** reviewed dry-run diff; post-run `rg '#\d+'` shows only intended residue. +- **Hooks:** a trivial ticket edit stages `.pql/changelog/` in the commit; a simulated merge replays. +- **Fold-in:** `pql backlinks` on a decision surfaces its source workshop. +- **End-to-end:** a full `/whats-next → /pr-process → /pr-review` cycle on pql with no SQLite reads. + +## Critical files + +- `.gitignore`; `.pql/config.yaml` (`dqr_dir`) +- `decisions/*.md` → `governance/{decisions,questions,rejected}/`; `decisions/README.md` +- `tooling/db/common.py` (seed-script read path); new `tooling/seed_pql_changelog.py`, `tooling/convert_ticket_refs.py` +- `.claude/skills/{whats-next,ticket,pr-process,pr-review,git-commit}/SKILL.md` +- `.claude/agents/clerk*` + `tooling/clerk-review` (with #965) +- `.claude/rules/ticket-cli.md`, `CLAUDE.md`, `Makefile`, `.config/hooks/*`, `.claude/settings.json` +- `docs/workshops/*/workshop-outcomes.md` (provenance frontmatter) + +## Gaps for the pql team + +See [`pql-requirements.md`](pql-requirements.md). diff --git a/pql-requirements.md b/pql-requirements.md new file mode 100644 index 000000000..e24d7823f --- /dev/null +++ b/pql-requirements.md @@ -0,0 +1,171 @@ +--- +title: "pql Requirements — gaps surfaced by the Settled Reach migration" +description: "Feature gaps and friction points found while migrating a 1013-ticket + 358-decision project onto pql, for the pql team" +type: requirements +status: draft +created: 2026-06-06 +updated: 2026-06-06 +tags: [pql, requirements, feedback] +--- + +# pql Requirements — gaps for the pql team + +Compiled while planning a migration of **The Settled Reach** (a Godot+Rust game) off a +SQLite-backed ticket CLI + markdown-decisions-sync onto pql. Scale: **1013 tickets**, 481 +dependency edges, 48 labels, 16 history rows; **358 decision records** (237 D / 109 Q / 12 R) +with 1040 cross-refs; a ~4385-file markdown vault (decisions, docs, 22 workshops, ~3500 wiki pages). + +pql is a strong fit and most of our system maps cleanly. The items below are the gaps we hit. +Each lists **why we need it**, a **suggested shape**, and a **priority**. Where we have a viable +workaround, it's noted — but the workaround is friction we'd rather not ship. + +--- + +## 1. Seeded / explicit ticket IDs + id-counter advance — **CRITICAL** + +**Why.** Our 1013 tickets have integer ids referenced as `#440` in thousands of immutable git +commits, in decision "Ticket: #N" lines, and by our PR automation. We need the pql id to preserve +the number — `T-440 ≡ old #440` — so the mapping stays a trivial bijection. pql's `id` is a TEXT +column, so we can seed `'T-440'` directly into the changelog. **But `pql ticket new` auto-assigns +`T-N` sequentially starting at `T-1`** — after seeding `T-1…T-1013`, the next native create would +mint `T-1` and collide. + +**Suggested shape.** +- `pql ticket new --id T-1014` (accept an explicit id), AND/OR +- on `pql plan rebuild`/import, **advance the internal id counter past the max existing id** so + native creates never recycle a seeded id. + +**Workaround.** None safe — we'd have to hold all native ticket creation until this lands, or +hand-manage a sentinel. This is the one true blocker for a clean numbered migration. + +--- + +## 2. Duplicate-ID detection in `pql decisions sync` / `validate` — **HIGH** + +**Why.** Our markdown had a genuine integrity bug: `D-035` defined in two domain files, `R-011` +defined three times. **`pql decisions sync` silently last-writer-wins-collapsed them** (reported +`broken: 0`), so the duplication was invisible. Our legacy Python `check-dupes` errors on it. A +duplicate canonical-ID is almost always a data bug, not an intended merge. + +**Suggested shape.** `pql decisions validate` (and `sync --strict`) should **warn or error on +duplicate D/Q/R ids across the vault**, reporting `file:line` for each occurrence. + +**Workaround.** Keep our Python `check-dupes` wired into pre-commit alongside pql — extra tooling +we'd like to retire. + +--- + +## 3. `core.hooksPath` awareness in `pql init` — **HIGH** + +**Why.** Our repo sets `core.hooksPath = .config/hooks` (so hooks are version-controlled). `pql init` +installs its pre-commit/post-merge/post-checkout hooks into `.git/hooks/`, which git **ignores** when +`core.hooksPath` is set. pql's changelog-staging + replay hooks would be **silently dead** — the whole +durable-versioning model depends on them. This is a quiet, dangerous failure (data appears to work, but +`.pql/changelog/` is never staged). + +**Suggested shape.** `pql init`/`pql doctor` should detect `core.hooksPath` and either install into the +configured dir or emit a loud warning with the manual fold-in snippet. + +**Workaround.** Manually fold pql's hook logic into `.config/hooks/*` (we will), but `pql doctor` +should flag the mismatch so others don't get bitten. + +--- + +## 4. Document the changelog seed format + `hash`/`canonical_version` algorithm — **HIGH** + +**Why.** `pql plan import --legacy` is documented as retired; the real durable artifact is +`.pql/changelog/
/.sql`. To migrate 1013 tickets we must emit those SQL rows directly. +We reverse-engineered the UPSERT shape, but the per-row `hash` and `canonical_version` columns + +the LWW guard (`WHERE excluded.updated_at > … OR (… AND excluded.hash > …)`) are opaque. If our seed +hashes don't match pql's scheme, later native edits still win by `updated_at` (so correctness holds), +but the LWW tie-break becomes unpredictable. + +**Suggested shape.** Document (a) the canonical changelog row format per table, (b) the `hash` +input + algorithm, (c) how `canonical_version` increments — OR provide a supported one-shot +`pql plan seed ` import that computes them. + +**Workaround.** Emit our own stable per-row hash; rely on `updated_at` for correctness. Works, but +under-specified. + +--- + +## 5. Bare-text reference detection / auto-link for `D-NNN` / `T-NNN` — **MEDIUM** + +**Why.** Across ~4385 vault files (esp. ~3500 *generated* wiki pages), decision/ticket references are +mostly **bare text** (`D-117`, `#440`) rather than markdown/wikilinks. `pql search` finds them, but +`pql backlinks`/`related` only follow real links — so "what references D-117?" is incomplete. Rewriting +3500+ generated files to add links is impractical (they're regenerated from a pipeline). + +**Suggested shape.** Optional **reference recognizers** (configurable regex → entity, e.g. +`D-\d+`→decision, `T-\d+`→ticket) so `backlinks`/`related` treat bare-text mentions as edges without +requiring link syntax in source. + +**Workaround.** Adopt a link convention in *authored* files only; accept that generated files are +search-only. + +--- + +## 6. WIP-limit enforcement (advisory) on `ticket status … in_progress` — **MEDIUM** + +**Why.** Our kanban flow enforces a WIP limit (max 5 in_progress); our activation step relied on the +CLI warning when exceeded. pql has no forced state machine / WIP concept. + +**Suggested shape.** An optional config (`wip_limit: 5`) that makes `pql ticket status in_progress` +emit an advisory warning (not a hard block) when the in_progress count would exceed it. + +**Workaround.** Compute `pql ticket list --status in_progress | count` in our skill and warn there. + +--- + +## 7. Fan-out / downstream-unblock count on `ticket show` — **MEDIUM** + +**Why.** Our batch-selection ranks ready tickets partly by **fan-out** — how many downstream tickets a +ticket unblocks. We currently compute it from the dependency graph. pql has `--unblocked` (great — it +replaced our manual blocker walk) but no "how many does this unblock" signal. + +**Suggested shape.** A `blocks_count` / `unblocks_count` field on `pql ticket show`/`list`, or a +`pql ticket list --orders-by fanout`. + +**Workaround.** Compute client-side from `ticket show --with-context` dep data. + +--- + +## 8. Multi-team / array team field — **LOW** + +**Why.** One ticket carries a comma-separated `team` (`client,server`). pql treats `team` as a single +string. (1 of 1013 — genuinely low.) + +**Suggested shape.** Accept an array/comma team, or document the single-team constraint. + +**Workaround.** Pick the primary team for the one affected ticket. + +--- + +## 9. Bless a "labels-as-milestones" pattern (or an optional milestone entity) — **LOW / nice-to-have** + +**Why.** Our schema has a milestones subsystem (milestones, ticket_milestones, milestone_deps, +cascade_phase) — but in practice it's **vestigial**: 2 milestones (1 active, 1 completed-with-0-tickets), +`milestone_deps` empty, 42/1013 tickets linked, cascade ordering never used. We're mapping the one +active milestone to a **label** (`phase:4`) and dropping the rest. This works today, so it is **not +blocking**. + +**Suggested shape.** Either bless **labels-as-milestones** as the documented pattern (with helper +queries / rollups), or offer an optional lightweight milestone entity (M:N ticket links, status, +ordering, done/total rollups) for projects that actually use phase gating. We'd only need the latter +if we later adopt real cross-phase dependency gating. + +--- + +## Summary + +| # | Requirement | Priority | Have workaround? | +|---|-------------|----------|------------------| +| 1 | Seeded/explicit ticket id + counter-advance | **Critical** | No (real blocker) | +| 2 | Duplicate-ID detection in sync/validate | High | Yes (keep our checker) | +| 3 | `core.hooksPath` awareness | High | Yes (manual fold-in) | +| 4 | Documented changelog seed format + hash algo | High | Partial | +| 5 | Bare-text ref detection for backlinks | Medium | Partial (authored files only) | +| 6 | WIP-limit advisory | Medium | Yes (skill-side) | +| 7 | Fan-out / unblocks count | Medium | Yes (client-side) | +| 8 | Multi-team field | Low | Yes | +| 9 | Labels-as-milestones blessing / optional entity | Low | Yes (labels) |