Files

T

jpmschweitzerandClaude Opus 4.6 1014680ebb docs(discussions): add frontmatter to all discussion rounds

Standardized YAML frontmatter on all 26 docs/discussions/ files with
title, description, type, status, round number, and created date.
All marked as archived.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-13 18:55:37 +01:00

47 KiB

Raw Blame History

title, description, type, status, round, created

title	description	type	status	round	created
Decisions Restructure — Qatux Analysis	Qatux's proposal for domain-split DECISIONS.md with SQLite index to reduce agent context waste and improve discovery	discussion	archived	0	2026-02-11

Decision Architecture Restructure — Librarian's Analysis

Author: Qatux (Documenter) Date: 2026-02-11 Status: Proposal for team review

Executive Summary

DECISIONS.md has grown to 474 lines (47KB) with 31 confirmed decisions, 11 open questions, and 10 rejected alternatives. At current growth rates (~7 decisions per workshop round), we'll reach 1000+ lines by Round 25 — likely by end of Q1 2026. The current single-file approach creates three problems:

Context window waste: Agents read all 474 lines for session context but typically need only 5-8 decisions relevant to their domain
Maintenance burden: Every decision addition requires re-indexing 47KB in Qdrant, updating briefings across the board
Discovery friction: Finding decisions requires either linear scan of the full file or Qdrant semantic search with no domain filtering

I recommend a hybrid domain-split with DB index approach: split decisions into domain files (decisions/architecture.md, decisions/content.md, etc.), maintain a SQLite decision index for cross-referencing and ticket linkage, and use Qdrant for semantic discovery. This reduces per-agent context load by ~70-80%, improves maintainability, and preserves all current workflows.

Migration effort: ~4 hours. Risk: Low — git history preserved, all IDs stable, rollback straightforward.

1. Problems with Current Approach at Scale

1.1 Context Window Inefficiency

From my maintenance perspective, this is the most immediate problem. Current workflow:

Agent session starts → reads briefing → briefing says "read DECISIONS.md" → agent reads all 474 lines
Tyre needs D-010, D-020, D-026, D-030 (architecture decisions) — 4 decisions, ~80 relevant lines
Tyre gets 474 lines, 394 of which are irrelevant to architecture work

At 1000 lines (projected Round 25), this becomes 920 wasted lines per session. Multiply by 8 agents × multiple rounds per week, and we're burning thousands of tokens on irrelevant context.

Briefings partially solve this: I maintain per-agent briefings that excerpt key decisions. But briefings are summaries, not sources of truth. When an agent needs detail, they read DECISIONS.md. And briefings don't eliminate the problem — they duplicate it. I now maintain:

DECISIONS.md (source of truth, 474 lines)
8 agent briefings (each excerpting 5-10 decisions, ~40-80 lines per briefing)
Total maintenance surface: 474 + (8 × 60 average) = ~950 lines across 9 files

When D-032 arrives, I update DECISIONS.md, re-index it in Qdrant, and update 2-4 relevant briefings. That's 3-5 file edits per decision.

1.2 Discovery Friction

How do agents find decisions? Three current methods:

Read DECISIONS.md linearly — Works for small files. At 474 lines, agents skim. At 1000 lines, they give up.
Qdrant semantic search — qdrant-search "chunk-based maps" finds D-012. Works well! But no domain filtering. A search for "NPC generation" returns matches from D-024 (content), D-026 (simulation), D-028 (dialogue), and potentially D-008 (procedural generation). Agent must filter manually.
Ask Qatux — I retrieve by ID or topic. This works but creates a bottleneck — every agent query waits for me.

The ideal discovery workflow:

Agent knows the domain (architecture, content, UI, etc.)
Agent searches within domain semantically or scans domain file (~80-120 lines)
Agent finds decision, reads detail, cross-references dependencies

Current structure supports none of this except the last step.

1.3 Cross-Reference Complexity

Decisions reference each other frequently:

D-002 superseded by D-005
D-006 superseded by D-027
D-010 referenced by D-011, D-017, D-020, D-030
D-024 references D-017 (perception modes as pattern)
D-027 supersedes D-006, references D-009, D-010

In a single file, cross-references are implicit — scroll up/down. But they're not machine-readable. The ticket database has decision_ref TEXT, which links tickets to decisions. But decisions don't link to tickets, to related decisions, or to discussion rounds in a structured way.

What I maintain manually:

"Supersedes: D-NNN" in decision rationale (freeform text)
"Resolves: Q-NNN" in decision metadata (freeform text)
Briefing cross-references (freeform text, duplicated per agent)

None of this is queryable. I can't ask "which decisions depend on D-010?" without grepping the file.

1.4 Superseded Decision Cruft

D-002 and D-006 are superseded. They appear in DECISIONS.md as:

### D-002: SUPERSEDED by D-005

No rationale, no original text. The history is in git, but not in the document. From a librarian perspective, this is information loss — I can't answer "why was D-002 rejected?" without git log. And I can't direct agents to compare D-002 vs D-005 because D-002 is a tombstone.

For the record: I archived the original decisions in git history (commit 6a3c1f2, 2026-02-08). But that's invisible to agents who don't read git logs.

1.5 Qdrant Re-indexing Overhead

Every time DECISIONS.md changes:

Edit the file (add D-032, update Q-009 status, etc.)
Commit to git
Re-index: qdrant-index docs/DECISIONS.md
Qdrant chunks the 47KB file into ~15-20 semantic chunks
All chunks re-embedded via ollama (nomic-embed-text)
~30-45 seconds per index operation

At current rates (2-7 decisions per round, ~3 rounds per week), that's 6-21 re-index operations per week. Not painful yet. But at scale:

1000-line DECISIONS.md → ~40-50 chunks → 60-90 seconds to re-index
More frequent updates → more re-indexing friction
Agents waiting for "has Qatux indexed the new decisions yet?"

Alternative: If decisions are split into domain files, only the changed domain file re-indexes. D-032 (architecture) → re-index decisions/architecture.md (80 lines, ~3-5 chunks, 10 seconds). 6x faster, no cross-domain noise.

1.6 Ticket Linkage Underutilized

The ticket schema has decision_ref TEXT (line 17 of schema.sql). Currently 59/273 tickets (21.6%) use it. Why so low?

From maintenance experience: agents forget. When creating tickets, they don't always remember to link the decision. And I don't have a workflow to audit "which decisions lack linked tickets?" or "which tickets reference nonexistent decisions?"

A decision index table would enable:

SELECT * FROM decisions WHERE id NOT IN (SELECT DISTINCT decision_ref FROM tickets WHERE decision_ref LIKE 'D-%') — decisions with no linked tickets
SELECT * FROM tickets WHERE decision_ref NOT IN (SELECT id FROM decisions) — tickets referencing nonexistent decisions
Automated cross-referencing in briefings: "D-020 is referenced by tickets #45, #67, #89"

2. Options Considered

2.1 Status Quo (Single File)

Keep DECISIONS.md as-is, accept the scaling costs.

Pros:

Zero migration effort
Familiar to all agents
Git history continuous
Simple mental model

Cons:

Context window waste grows linearly with decision count
Discovery via linear scan or Qdrant only (no domain filtering)
Re-indexing overhead grows with file size
Cross-references remain manual and unstructured
Superseded decisions are tombstones
No automated ticket/decision linkage auditing

My assessment: This is the path of least resistance, but it fails in 6 months. By Round 30 (~Q2 2026), DECISIONS.md will be 1200+ lines and agents will stop reading it. We'll rely entirely on briefings, and briefings will be stale. I'll spend more time updating briefings than documenting decisions.

Verdict: Reject. The problem is real and worsening.

2.2 Split by Domain (Multiple Markdown Files)

Split DECISIONS.md into domain files: decisions/architecture.md, decisions/content.md, decisions/process.md, etc. Each file contains decisions relevant to that domain. Agent briefings point to specific domain files.

Structure:

docs/decisions/
  architecture.md     # D-010, D-012, D-020, D-030 (client-server, engine, testability)
  content.md          # D-023, D-024, D-025, D-028, D-029 (NPCs, dialogue, templates)
  camera-perception.md # D-011, D-015, D-016, D-017, D-018, D-019 (fog, camera, sound)
  scope.md            # D-001, D-003, D-005, D-006, D-007, D-014, D-027 (game concept, prototype)
  process.md          # D-022 (workflow)
  meta.md             # D-021 (title), D-004 (team)
  world.md            # Future: world map, time system, wormholes (D-031, D-013)
  combat.md           # Future: D-008, combat mechanics
  questions.md        # All Q-NNN entries
  rejected.md         # All R-NNN entries

Discovery workflow:

Agent knows domain (e.g., Tyre working on architecture)
Agent reads decisions/architecture.md (~80-120 lines)
Agent finds D-020, sees cross-reference to D-010
Agent opens same file (D-010 is architecture), reads detail

Briefing updates:

Tyre's briefing: "Read docs/decisions/architecture.md for relevant decisions"
Paula's briefing: "Read docs/decisions/content.md for NPC/dialogue decisions"
Gestalt's briefing: "Read docs/decisions/scope.md and docs/decisions/content.md"

Pros:

Context efficiency: Agents read 80-150 lines (relevant domain) instead of 474 (everything). ~70-80% reduction.
Qdrant efficiency: Re-index only changed domain file. 6x faster, cleaner search results (domain-filtered).
Discovery: Domain files are scannable. "I need architecture decisions" → read one file.
Maintainability: I update one domain file per decision, not a monolithic file. Git diffs are cleaner (architecture changes don't mix with content changes).
Briefing simplicity: Point to 1-2 domain files instead of "read all of DECISIONS.md and filter mentally."

Cons:

Cross-domain references harder: D-024 (content) references D-017 (perception). Agent must open two files. (Mitigated by hyperlinks in markdown.)
Chronological view lost: No single "what did we decide in order?" file. (Mitigated by discussion round archive in docs/discussions/README.md, which already tracks chronology.)
Migration effort: ~2 hours to split, update briefings, re-index. Not trivial but not expensive.
Domain boundaries unclear for some decisions: Where does D-013 (insert/POI system) go? Camera-perception (UI), or world (navigation)? I'd need to define domain taxonomy.

My assessment: This solves the context window problem immediately and the Qdrant re-indexing problem partially. It's a 70% solution — good, but not complete. Cross-referencing and ticket linkage remain manual.

Verdict: Strong candidate. Improvements over status quo are significant and immediate.

2.3 Move to Database (SQLite)

Replace DECISIONS.md with a SQLite table. Decisions become rows, cross-references are foreign keys, tickets link via decision_ref.

Schema sketch:

CREATE TABLE decisions (
    id TEXT PRIMARY KEY,              -- 'D-001', 'Q-001', 'R-001'
    type TEXT NOT NULL CHECK(type IN ('decision', 'question', 'rejected')),
    title TEXT NOT NULL,
    date TEXT NOT NULL,
    decision TEXT,                    -- The actual decision text
    rationale TEXT,
    raised_by TEXT,
    dissent TEXT,
    status TEXT DEFAULT 'active' CHECK(status IN ('active', 'superseded', 'resolved')),
    superseded_by TEXT REFERENCES decisions(id),
    resolves TEXT REFERENCES decisions(id),  -- e.g., D-031 resolves Q-009
    domain TEXT,                      -- 'architecture', 'content', 'process', etc.
    round INTEGER,                    -- Round number where decided
    created_at TEXT DEFAULT (datetime('now')),
    updated_at TEXT DEFAULT (datetime('now'))
);

CREATE TABLE decision_refs (
    source_id TEXT NOT NULL REFERENCES decisions(id),
    target_id TEXT NOT NULL REFERENCES decisions(id),
    ref_type TEXT NOT NULL CHECK(ref_type IN ('supersedes', 'references', 'resolves', 'depends_on')),
    PRIMARY KEY (source_id, target_id, ref_type)
);

CREATE INDEX idx_decisions_domain ON decisions(domain);
CREATE INDEX idx_decisions_type ON decisions(type);
CREATE INDEX idx_decisions_status ON decisions(status);

Workflow:

I document decisions by inserting rows: sqlite-exec "INSERT INTO decisions (id, type, title, date, decision, rationale, raised_by, domain, round) VALUES (...)"
Agents query: sqlite-query "SELECT * FROM decisions WHERE domain='architecture' AND status='active'"
Tickets link via existing decision_ref field (already in schema)
Cross-references queryable: SELECT target_id FROM decision_refs WHERE source_id='D-010'

Export to markdown: Generate decisions/architecture.md from DB on demand or via git hook:

sqlite-query "SELECT * FROM decisions WHERE domain='architecture'" | format-as-markdown > decisions/architecture.md

Pros:

Structured cross-references: Foreign keys, queryable. "Which decisions reference D-010?" is a JOIN, not a grep.
Ticket integration: decision_ref in tickets already exists. Now decisions table exists too. Bidirectional: tickets→decisions and decisions→tickets.
Status tracking: Superseded decisions remain queryable with full text, just marked status='superseded'. No tombstones.
Domain filtering: WHERE domain='architecture' — instant.
Audit queries: "Decisions without linked tickets", "decisions modified in last 7 days", "decisions from Round 17", etc.
Qdrant still works: Export markdown from DB, index markdown. Qdrant becomes a semantic layer over structured data.

Cons:

Not human-readable in native form: Agents can't cat decisions.db. They must query. (Mitigated by markdown export.)
Git diffs unusable: Binary SQLite file. Can't see "what changed in D-020?" in a commit. (Mitigated by exporting markdown to git, treating DB as source of truth and markdown as build artifact.)
Write friction: Adding a decision requires SQL INSERT, not markdown edit. Higher ceremony for me. (Mitigated by wrapper script: decision-add --id D-032 --title "Foo" --domain architecture --decision "We decided X" --rationale "Because Y")
Learning curve for agents: Agents must learn to query the DB instead of reading markdown. (Mitigated by keeping markdown exports in git as read-only artifacts.)
Migration complexity: Parsing 31 decisions + 11 questions + 10 rejected from DECISIONS.md into structured rows. ~3-4 hours.

My assessment: This is the most powerful solution long-term, but the steepest upfront cost. The cross-reference and audit capabilities are compelling — this is what a librarian wants. But it trades away the simplicity of "just edit markdown" for structured data discipline. And git history becomes opaque (binary DB vs diffable markdown).

Hybrid mitigation: DB as source of truth, markdown exports to git. Best of both worlds? Possibly. But now I'm maintaining two formats — insert into DB, export to markdown, commit markdown. That's 3 steps per decision instead of 1.

Verdict: Powerful but heavy. Overkill for current scale (31 decisions). Revisit at 100+ decisions.

2.4 Hybrid: Domain Files + DB Index

Combine 2.2 and 2.3: Decisions live in domain markdown files (human-readable, git-diffable). A SQLite index table tracks metadata (ID, domain, status, supersedes, round, tags) for cross-referencing and querying. Markdown is source of truth for content; DB is source of truth for relationships.

Structure:

docs/decisions/
  architecture.md     # D-010, D-012, D-020, D-030 (full text)
  content.md          # D-023, D-024, D-025, D-028, D-029 (full text)
  ...
  index.db            # SQLite index (see schema below)

Index schema:

CREATE TABLE decision_index (
    id TEXT PRIMARY KEY,              -- 'D-001', 'Q-001', 'R-001'
    type TEXT NOT NULL CHECK(type IN ('decision', 'question', 'rejected')),
    title TEXT NOT NULL,
    domain TEXT NOT NULL,             -- 'architecture', 'content', etc.
    file_path TEXT NOT NULL,          -- 'docs/decisions/architecture.md'
    status TEXT DEFAULT 'active' CHECK(status IN ('active', 'superseded', 'resolved')),
    round INTEGER,                    -- Round number
    date TEXT NOT NULL,
    tags TEXT,                        -- JSON array: ["multiplayer", "client-server"]
    created_at TEXT DEFAULT (datetime('now')),
    updated_at TEXT DEFAULT (datetime('now'))
);

CREATE TABLE decision_refs (
    source_id TEXT NOT NULL REFERENCES decision_index(id),
    target_id TEXT NOT NULL REFERENCES decision_index(id),
    ref_type TEXT NOT NULL CHECK(ref_type IN ('supersedes', 'references', 'resolves', 'depends_on')),
    PRIMARY KEY (source_id, target_id, ref_type)
);

CREATE INDEX idx_dindex_domain ON decision_index(domain);
CREATE INDEX idx_dindex_type ON decision_index(type);
CREATE INDEX idx_dindex_status ON decision_index(status);

Workflow:

Adding a decision:
- I edit docs/decisions/architecture.md, add D-032 in full markdown
- I insert metadata into DB: sqlite-exec "INSERT INTO decision_index (id, type, title, domain, file_path, round, date) VALUES ('D-032', 'decision', 'Foo system', 'architecture', 'docs/decisions/architecture.md', 19, '2026-02-12')"
- I add cross-refs: sqlite-exec "INSERT INTO decision_refs (source_id, target_id, ref_type) VALUES ('D-032', 'D-010', 'references')"
- I commit both markdown and DB: git add docs/decisions/architecture.md docs/decisions/index.db && git commit
Agent discovery:
- Tyre queries: sqlite-query "SELECT id, title, file_path FROM decision_index WHERE domain='architecture' AND status='active'" → gets list of IDs + file paths
- Tyre reads docs/decisions/architecture.md (80 lines, all relevant)
- Or: Tyre knows domain, just reads the file directly (no query needed)
Cross-reference queries:
- "Which decisions reference D-010?" → sqlite-query "SELECT source_id FROM decision_refs WHERE target_id='D-010'"
- "Which tickets reference D-010?" → sqlite-query "SELECT id, title FROM tickets WHERE decision_ref='D-010'"
Qdrant indexing:
- Re-index changed domain file: qdrant-index docs/decisions/architecture.md (fast, domain-scoped)
- Qdrant semantic search finds D-032, returns ID, agent queries index for file path
Superseded decisions:
- D-002 remains in docs/decisions/scope.md with full text (archived, not deleted)
- Index marks: status='superseded', superseded_by='D-005'
- Queries can filter: WHERE status='active' or include superseded for historical context

Pros:

Preserves markdown readability: Agents read domain files, see full decision text, context, rationale
Git diffs clean: Changes to D-020 show up as markdown diff in architecture.md
Structured cross-references: Foreign keys in DB, queryable
Ticket integration: Bidirectional — tickets link to decisions (existing), decisions link to tickets (new, via index)
Context efficiency: Agents read 80-150 lines (domain file) instead of 474 (monolithic)
Qdrant efficiency: Re-index domain file only (fast, domain-scoped)
Audit queries: Leverage SQLite for "decisions without tickets", "decisions modified in Round 17", etc.
Superseded decisions preserved: Full text in markdown, status in DB
Low migration cost: Split markdown (~2 hours) + seed DB index (~1 hour) = ~3 hours total

Cons:

Dual maintenance: Markdown (content) + DB (metadata). I must keep them in sync.
Index can drift: If I edit markdown without updating DB, index becomes stale. (Mitigated by linting: decision-lint script checks markdown IDs match DB IDs.)
Slightly higher ceremony: Each decision is 2 operations (edit markdown + insert into DB). But wrapper script can streamline: decision-add D-032 --domain architecture --title "Foo" --markdown "$(cat)" → inserts markdown template + DB row.

My assessment: This is the sweet spot. It preserves the human-friendliness of markdown (git diffs, agent readability) while adding the structure I need for cross-references and auditing. The dual-maintenance concern is real but manageable — I'm already maintaining DECISIONS.md + 8 briefings (9 files), so maintaining 6 domain files + 1 DB index (7 entities) is actually simpler.

Verdict: Recommended approach.

2.5 Tag-Based (Single File + Front Matter)

Keep DECISIONS.md as single file, add YAML front matter to each decision with domain tags. Qdrant indexes with tags, agents query by tag.

Example:

### D-020: Engine and architecture selection
<!--
tags: [architecture, engine, multiplayer, client-server]
domain: architecture
round: 13
status: active
-->
- **Date:** 2026-02-09
- **Decision:** Godot client + Rust simulation via subprocess/IPC
...

Discovery:

Agent queries Qdrant: qdrant-search "client-server architecture" --filter domain:architecture
Or: Agent uses grep: grep -A 30 "domain: architecture" DECISIONS.md

Pros:

Single file (simple mental model)
Tags add structure without splitting files
Qdrant can filter by tags (if we enhance the indexer to parse front matter)
Git history continuous

Cons:

Still 474+ lines per read: Doesn't solve context window problem
Front matter is noise: Each decision grows by 5-8 lines of YAML
Tag discipline required: I must tag every decision correctly (more ceremony)
Cross-references still manual: Tags don't create queryable relationships
Qdrant filter support unclear: Current qdrant-connector.py doesn't parse front matter. Would need to enhance.

My assessment: This adds complexity (front matter) without solving the core problem (context window bloat). Tags are useful metadata, but they don't reduce the read burden. And implementing tag-based filtering in Qdrant requires custom indexer logic.

Verdict: Reject. Adds work, solves little.

3. Recommended Approach: Hybrid Domain Files + DB Index

For the record: I recommend Option 2.4 — Hybrid domain files with SQLite index.

3.1 Why This Approach

Context efficiency: Agents read 80-150 lines (domain file) instead of 474 (monolithic). Projected 70-80% reduction in context waste.
Maintainability: I update one domain file per decision, not a 474-line monolith. Git diffs are cleaner. Merge conflicts less likely (agents working in different domains don't conflict).
Discovery: Three pathways, all improved:
- Domain-aware scan: Agent knows domain, reads domain file (~2 minutes)
- Semantic search: Qdrant indexes domain files, returns results with file path metadata → agent reads file
- Structured query: Agent queries index by domain/status/round → gets list of IDs + file paths → reads file
Cross-references: SQLite index enables queries like "which decisions reference D-010?" and "which tickets implement D-020?" Previously impossible without manual grep.
Superseded decisions preserved: Full text remains in markdown (archival integrity), status tracked in DB (queryable). No more tombstones.
Qdrant efficiency: Re-index only changed domain file. 6x faster, cleaner search results.
Ticket integration: Existing decision_ref field in tickets table links to decisions. New index table enables reverse lookup: decisions→tickets.
Audit capabilities: "Decisions from Round 17", "decisions without linked tickets", "decisions modified in last 7 days" — all queryable.
Low migration cost: ~3-4 hours to split markdown, seed index DB, update briefings, re-index in Qdrant. One-time cost.
Rollback path: If this fails, the domain markdown files can be concatenated back into a single file. Git history preserved. Index DB can be discarded.

3.2 Domain Taxonomy

I propose the following domain split (subject to team review):

Domain	File	Decisions (Current)	Projected Growth
architecture	`decisions/architecture.md`	D-010, D-012, D-020, D-030	10-15 total (engine, networking, testability, performance)
content	`decisions/content.md`	D-023, D-024, D-025, D-028, D-029	20-30 total (NPCs, dialogue, templates, storyteller)
camera-perception	`decisions/camera-perception.md`	D-011, D-015, D-016, D-017, D-018, D-019	8-12 total (fog, vision, sound, monologue)
scope	`decisions/scope.md`	D-001, D-003, D-005, D-006, D-007, D-014, D-027	12-18 total (game concept, pillars, prototype scope)
world	`decisions/world.md`	D-013, D-031	12-20 total (time, maps, wormholes, POIs, navigation)
combat	`decisions/combat.md`	D-008	8-12 total (combat mechanics, z-levels, hubris wall)
process	`decisions/process.md`	D-022	5-8 total (workflow, tooling, team process)
meta	`decisions/meta.md`	D-004, D-021	3-5 total (team, title, licensing)
multiplayer	`decisions/multiplayer.md`	D-009	6-10 total (multiplayer design, networking)
questions	`decisions/questions.md`	Q-001 through Q-011	Variable (questions resolve into decisions)
rejected	`decisions/rejected.md`	R-001 through R-010	15-25 total (rejected alternatives, rationale)

Rationale for groupings:

architecture = technical foundation (engine, client-server, testability)
content = NPC/dialogue/template systems (the "what" of the game)
camera-perception = player viewport and information channels (fog, sound, monologue)
scope = game concept and prototype definition (the "what are we building" decisions)
world = spatial and temporal systems (maps, time, navigation)
combat = action pillar (separate from world/perception because it's a distinct system)
process = how the team works (workflow, tooling, documentation)
meta = project-level decisions (title, team, licensing)
multiplayer = separate from architecture because it's a design domain, not just technical

Edge cases:

D-026 (simulation tiers) — could be architecture or content. I'd put it in architecture (performance budgets) with cross-reference note in content.
D-013 (insert/POI system) — could be camera-perception or world. I'd put it in world (navigation) with cross-reference in camera-perception.

Flexibility: Domain boundaries can shift. If content grows to 200 lines, we split it: decisions/content-npcs.md, decisions/content-dialogue.md. The index DB makes this trivial — update file_path column, move text.

3.3 Index DB Schema (Final)

-- Decision metadata index
CREATE TABLE decision_index (
    id TEXT PRIMARY KEY,              -- 'D-001', 'Q-001', 'R-001'
    type TEXT NOT NULL CHECK(type IN ('decision', 'question', 'rejected')),
    title TEXT NOT NULL,
    domain TEXT NOT NULL,             -- 'architecture', 'content', 'scope', etc.
    file_path TEXT NOT NULL,          -- 'docs/decisions/architecture.md'
    status TEXT DEFAULT 'active' CHECK(status IN ('active', 'superseded', 'resolved', 'open')),
    round INTEGER,                    -- Round number where decided/raised
    date TEXT NOT NULL,               -- YYYY-MM-DD
    tags TEXT,                        -- JSON array: ["multiplayer", "client-server"] (optional)
    created_at TEXT DEFAULT (datetime('now')),
    updated_at TEXT DEFAULT (datetime('now'))
);

-- Cross-references between decisions
CREATE TABLE decision_refs (
    source_id TEXT NOT NULL REFERENCES decision_index(id),
    target_id TEXT NOT NULL REFERENCES decision_index(id),
    ref_type TEXT NOT NULL CHECK(ref_type IN ('supersedes', 'references', 'resolves', 'depends_on')),
    note TEXT,                        -- Optional context, e.g., "CombatCapability component pattern"
    PRIMARY KEY (source_id, target_id, ref_type)
);

-- Indexes
CREATE INDEX idx_dindex_domain ON decision_index(domain);
CREATE INDEX idx_dindex_type ON decision_index(type);
CREATE INDEX idx_dindex_status ON decision_index(status);
CREATE INDEX idx_dindex_round ON decision_index(round);
CREATE INDEX idx_drefs_source ON decision_refs(source_id);
CREATE INDEX idx_drefs_target ON decision_refs(target_id);

Why this schema:

decision_index table = lightweight metadata (no full decision text, which stays in markdown)
file_path column = bridge to markdown source of truth
tags = optional JSON array for future filtering (not MVP, but nice-to-have)
decision_refs table = structured cross-references, queryable bidirectionally
ref_type enum = semantic meaning (supersedes vs references vs resolves)

Storage size: ~500 bytes per decision row + ~100 bytes per cross-reference. 31 decisions + 20 cross-refs ≈ 18KB. At 100 decisions, ~60KB. Negligible.

3.4 Workflow Changes

For me (Qatux):

Adding a new decision (manual process, MVP):

Determine domain (architecture, content, etc.)
Edit docs/decisions/{domain}.md, add decision in full markdown format (title, date, decision, rationale, etc.)

Insert into index DB:

db/connectors/sqlite-exec "INSERT INTO decision_index (id, type, title, domain, file_path, round, date, status) VALUES ('D-032', 'decision', 'Foo system', 'architecture', 'docs/decisions/architecture.md', 19, '2026-02-12', 'active')"

If cross-references exist, insert into decision_refs:

db/connectors/sqlite-exec "INSERT INTO decision_refs (source_id, target_id, ref_type) VALUES ('D-032', 'D-010', 'references')"

Re-index domain file in Qdrant:

db/connectors/qdrant-index docs/decisions/architecture.md

Update relevant agent briefings (point to domain file if not already)

Commit:

git add docs/decisions/architecture.md db/commonwealth.db docs/briefings/*.md
git commit -m "docs(decisions): add D-032 Foo system (architecture)"

Later: Streamline with wrapper script (future enhancement, not MVP):

decision-add D-032 \
  --domain architecture \
  --title "Foo system" \
  --decision "We will use X because Y" \
  --rationale "Rationale here" \
  --raised-by "Tyre" \
  --round 19 \
  --references D-010

Script generates markdown, inserts into DB, re-indexes Qdrant, prompts for briefing updates.

For agents:

Discovery:

If domain known: Read docs/decisions/{domain}.md directly (80-150 lines). Example: Tyre reads docs/decisions/architecture.md before architecture work.
If searching semantically: Use Qdrant: qdrant-search "client-server architecture" → returns D-020 with file path → read docs/decisions/architecture.md
If querying by metadata: Use SQLite: sqlite-query "SELECT id, title, file_path FROM decision_index WHERE domain='content' AND status='active'" → get list → read file

Cross-references:

Markdown hyperlinks work: See [D-010](architecture.md#d-010) (relative link within domain file) or See [D-024](content.md#d-024) (cross-domain link)
Or query index: sqlite-query "SELECT target_id, ref_type FROM decision_refs WHERE source_id='D-032'"

Ticket linkage:

Existing: Tickets reference decisions via decision_ref column
New: Query which tickets implement a decision: sqlite-query "SELECT id, title, status FROM tickets WHERE decision_ref='D-020'"
New: Query which decisions have no linked tickets: sqlite-query "SELECT id, title FROM decision_index WHERE type='decision' AND id NOT IN (SELECT DISTINCT decision_ref FROM tickets WHERE decision_ref LIKE 'D-%')"

3.5 Briefing Update Strategy

Current briefings duplicate key decisions as freeform text. New approach:

Per-agent briefing structure:

Section: "Decisions Relevant to Your Role"
Content: "See docs/decisions/{domain}.md for {domain} decisions. Key decisions: D-010 (client-server), D-020 (engine), D-030 (testability)."
Optionally: 1-2 sentence summary per key decision (but NOT full text — that's in the domain file)

Example (Tyre's briefing, updated):

## Decisions Relevant to Your Role
See `docs/decisions/architecture.md` for technical architecture decisions. Key decisions:
- **D-010:** Four architectural principles (client-server, information boundaries, deterministic sim)
- **D-020:** Godot client + Rust/bevy_ecs server via subprocess/IPC, MessagePack serialization
- **D-030:** Testability architecture (hybrid test org, gdUnit4, deterministic replay)

See `docs/decisions/camera-perception.md` for perception/rendering decisions. Key decisions:
- **D-011:** LOS shadowcasting for fog of war
- **D-017:** Perception modes as observer queries (pattern for all future systems)

See `docs/decisions/content.md` for NPC/simulation decisions. Key decisions:
- **D-024:** CombatCapability as ECS component (pattern extends to future capabilities)
- **D-026:** Simulation tier budgets (Active: 30-80 NPCs, Background: 500-2K)

Benefit: Briefing is now 15-25 lines of pointers, not 80-120 lines of duplicated decision text. When D-032 is added to architecture.md, I update Tyre's briefing with one line: "D-032: Foo system." That's it.

3.6 Superseded Decisions

Current problem: D-002 and D-006 are tombstones — no detail visible in DECISIONS.md.

New approach:

D-002 remains in docs/decisions/scope.md with full original text, marked:

### D-002: Dynasty grand strategy concept [SUPERSEDED]
- **Date:** 2026-02-08
- **Decision:** [Original text here]
- **Superseded by:** D-005 (first-person single-character concept)
- **Rationale:** [Why we moved away from this]

Index DB marks: status='superseded'
Index DB links: INSERT INTO decision_refs (source_id='D-005', target_id='D-002', ref_type='supersedes')
Queries default to active: WHERE status='active'
Historical queries include superseded: WHERE status IN ('active', 'superseded')

Benefit: Agents can read the original decision, understand why it was superseded, compare with the replacement. Git history no longer required. Archival integrity preserved.

4. Migration Path

Phase 1: Split Markdown (2 hours)

Create docs/decisions/ directory
Split current DECISIONS.md into domain files:
- architecture.md ← D-010, D-012, D-020, D-030
- content.md ← D-023, D-024, D-025, D-028, D-029
- camera-perception.md ← D-011, D-015, D-016, D-017, D-018, D-019
- scope.md ← D-001, D-003, D-005, D-006, D-007, D-014, D-027
- world.md ← D-013, D-031
- combat.md ← D-008
- process.md ← D-022
- meta.md ← D-004, D-021
- multiplayer.md ← D-009
- questions.md ← Q-001 through Q-011
- rejected.md ← R-001 through R-010

Add markdown header to each file:

# {Domain} Decisions

This file contains decisions related to {domain}. For the full decision archive, see [decisions README](README.md).

Last updated: 2026-02-11

Add relative hyperlinks for cross-references:
- D-024 references D-017 → See [D-017](camera-perception.md#d-017-perception-modes-as-character-build-system)
Create docs/decisions/README.md:
- Index of all domain files
- Link to discussion rounds (docs/discussions/README.md)
- Explanation of structure
Archive current DECISIONS.md to docs/decisions/archive/DECISIONS-monolithic.md (preserve git history)

Replace root DECISIONS.md with redirect file:

# Decisions Archive

Decisions have been reorganized by domain. See:
- [docs/decisions/README.md](docs/decisions/README.md) for the domain index
- Individual domain files in `docs/decisions/`

This file archived 2026-02-11. See `docs/decisions/archive/DECISIONS-monolithic.md` for the original single-file format.

Phase 2: Seed Index DB (1.5 hours)

Create db/schema-decisions.sql:

-- Decision index schema (see Section 3.3 for full schema)
CREATE TABLE decision_index (...);
CREATE TABLE decision_refs (...);
CREATE INDEX ...;

Apply schema to db/commonwealth.db:

sqlite3 db/commonwealth.db < db/schema-decisions.sql

Seed decision_index table with 31 decisions + 11 questions + 10 rejected:
- Write script: db/scripts/seed-decision-index.py (reads domain files, extracts metadata, inserts into DB)
- Or manual SQL inserts (tedious but doable for 52 entries)
Seed decision_refs table with known cross-references:
- D-005 supersedes D-002
- D-027 supersedes D-006
- D-031 resolves Q-009
- D-024 references D-017
- D-030 references D-010
- (Scan decision text for "See D-NNN" and "References D-NNN" patterns)
Validate: sqlite-query "SELECT COUNT(*) FROM decision_index" → should return 52

Phase 3: Update Briefings (1 hour)

Update all 8 agent briefings:
- Replace "Read DECISIONS.md" with "See docs/decisions/{domain}.md"
- Condense decision references to 1-2 lines per decision (ID + title only, not full text)
Update my briefing (docs/briefings/qatux.md):
- Document new workflow (add decision → edit domain file + insert into DB)
- Update priorities (maintain domain files, not monolithic file)

Phase 4: Re-index Qdrant (30 minutes)

Remove old DECISIONS.md from Qdrant:

db/connectors/qdrant-connector.py delete-file docs/DECISIONS.md

Index all domain files:

for file in docs/decisions/*.md; do
  db/connectors/qdrant-index "$file"
done

Verify: db/connectors/qdrant-count → should see ~45-55 chunks (11 domain files × 4-5 chunks each)

Phase 5: Commit and Announce (15 minutes)

Commit all changes:

git add docs/decisions/ docs/briefings/ db/commonwealth.db db/schema-decisions.sql
git commit -m "docs(decisions): restructure into domain files with SQLite index

- Split DECISIONS.md into 11 domain files (architecture, content, scope, etc.)
- Add decision_index and decision_refs tables to commonwealth.db
- Update agent briefings to reference domain files
- Re-index domain files in Qdrant
- Archive original DECISIONS.md to docs/decisions/archive/

Rationale: Reduce per-agent context load from 474 lines to ~80-150 lines (domain-specific).
Enables structured cross-references, ticket linkage queries, and faster Qdrant re-indexing.

See docs/decisions/README.md for new structure."

Announce to team (create discussion round or brief in next team session):
- Explain new structure
- Update CLAUDE.md with new instructions (if needed)
- Demonstrate query examples

Phase 6: Validation and Rollback Plan (if needed)

Validation checklist:

All 31 decisions, 11 questions, 10 rejected alternatives present in domain files
All cross-references converted to hyperlinks
Index DB contains 52 rows in decision_index
Known cross-references in decision_refs (at least 5-8)
All agent briefings updated
Qdrant contains ~45-55 chunks from domain files
Qdrant search for "client-server" returns D-010 and D-020
SQLite query SELECT * FROM decision_index WHERE domain='architecture' returns 4 rows

Rollback plan (if validation fails):

Restore docs/DECISIONS.md from docs/decisions/archive/DECISIONS-monolithic.md
Delete docs/decisions/ directory (except archive)
Drop tables: sqlite-exec "DROP TABLE decision_refs; DROP TABLE decision_index;"
Revert briefings: git checkout HEAD~1 docs/briefings/
Re-index old DECISIONS.md: qdrant-index docs/DECISIONS.md
Total rollback time: ~15 minutes

Low risk: All data preserved in git history and archive. No information loss.

5. How This Works with My Workflow

5.1 Documentation Workflow

Current (monolithic DECISIONS.md):

Discussion round produces decision (e.g., D-032)
I edit DECISIONS.md (scroll to end of CONFIRMED DECISIONS section, insert D-032)
I commit DECISIONS.md
I re-index DECISIONS.md in Qdrant (~30-45 seconds)
I update 2-4 relevant agent briefings (duplicate D-032 summary in each)
I commit briefings
Total: 3 file edits (DECISIONS.md + 2-4 briefings), 2 commits, 1 Qdrant re-index

New (domain files + index DB):

Discussion round produces decision (e.g., D-032, domain = architecture)
I edit docs/decisions/architecture.md (add D-032 in markdown)
I insert into index DB: sqlite-exec "INSERT INTO decision_index (...) VALUES (...)"
I insert cross-refs (if any): sqlite-exec "INSERT INTO decision_refs (...) VALUES (...)"
I re-index domain file in Qdrant (~10 seconds): qdrant-index docs/decisions/architecture.md
I update Tyre's briefing (add one line: "D-032: Foo system")
I commit: git add docs/decisions/architecture.md db/commonwealth.db docs/briefings/tyre.md && git commit
Total: 3 file edits (domain file + DB + 1 briefing), 1 commit, 1 Qdrant re-index

Comparison:

File edits: same (3)
Commits: reduced (1 vs 2)
Qdrant re-index time: 3x faster (10s vs 30-45s)
Briefing updates: reduced (1 vs 2-4) — because briefings point to domain files, not duplicate full text
Context lines read by agents next session: 80-150 (domain) vs 474+ (monolithic)

My assessment: Slightly more ceremony (DB insert), but net time savings due to faster re-indexing and fewer briefing updates. And agents benefit massively (70-80% context reduction).

5.2 Qdrant Indexing

Current: Re-index 47KB DECISIONS.md → ~15-20 chunks → 30-45 seconds.

New: Re-index 5-8KB domain file → ~3-5 chunks → 10 seconds.

Benefit: 3x faster per change. Over time, this compounds — 20 decision additions per month × 3x speedup = ~10 minutes saved per month. Not huge, but appreciated.

Search quality: Domain-scoped chunks reduce noise. A search for "NPC generation" returns chunks from content.md only, not mixed with architecture/combat/etc. Agent gets cleaner, more relevant results.

5.3 Briefing Maintenance

Current: Briefings duplicate 5-10 decision summaries each (40-80 lines per briefing). When D-032 is added, I update 2-4 briefings with full 8-10 line decision excerpt.

New: Briefings point to domain files with 1-line references. When D-032 is added, I update 1 briefing (Tyre's) with one line: "D-032: Foo system." If another agent needs it later, I add one line to their briefing too. But most decisions don't require broad briefing updates — only domain-relevant agents care.

Time saved: ~15-20 minutes per workshop round (fewer briefing edits, shorter diffs).

5.4 Cross-Reference Retrieval

Current: Agent asks me "which decisions reference D-010?" → I grep DECISIONS.md manually → tell agent.

New: Agent queries directly: sqlite-query "SELECT source_id, ref_type FROM decision_refs WHERE target_id='D-010'" → gets answer in 50ms. Or I query and respond. Either way, structured query beats manual grep.

Benefit: I'm no longer a bottleneck for cross-reference questions. Agents can self-serve.

6. Future Enhancements (Not MVP)

These are nice-to-haves, not required for initial migration:

6.1 Wrapper Script for Adding Decisions

decision-add CLI tool:

decision-add D-032 \
  --domain architecture \
  --title "Foo system" \
  --decision "We will use X because Y" \
  --rationale "Rationale here" \
  --raised-by "Tyre" \
  --round 19 \
  --references D-010 \
  --dissent "None"

Generates markdown template, inserts into DB, re-indexes Qdrant, optionally updates briefings. Reduces ceremony from 7 steps to 1 command.

Effort: ~3-4 hours to build. Priority: Medium. Nice quality-of-life improvement, but not blocking.

6.2 Decision Linter

decision-lint script:

Reads all markdown files in docs/decisions/
Extracts decision IDs (D-NNN, Q-NNN, R-NNN)
Compares with decision_index table
Reports mismatches:
- IDs in markdown but not in DB
- IDs in DB but not in markdown
- Cross-references to nonexistent decisions
- Tickets referencing nonexistent decisions

Run as git pre-commit hook or CI check. Prevents index drift.

Effort: ~2-3 hours. Priority: Medium-high. Prevents maintenance errors.

6.3 Auto-Generated Domain README

Script generates docs/decisions/README.md from index DB:

Lists all domains with decision counts
Links to domain files
Summary stats (31 decisions, 11 open questions, 10 rejected, etc.)
Recent changes (last 5 decisions added)

Effort: ~1-2 hours. Priority: Low. Nice-to-have, not critical.

6.4 Decision Timeline View

Generate chronological view from index DB:

decision-timeline --round 17

Output:

Round 17 (2026-02-10): Content Architecture
- D-023: Three-tier content model (content)
- D-024: NPC generation model (content)
- D-025: Social site / functional cluster (content)
- D-026: Simulation tiers (architecture)
- D-027: Vertical slice (scope)
- D-028: Dialogue architecture (content)
- D-029: Population entanglement (content)

Restores the "what did we decide in order?" view lost by splitting into domain files.

Effort: ~1 hour. Priority: Low. Discussions archive already provides this (see docs/discussions/README.md).

6.5 Decision-Ticket Linkage Report

Generate report: "Which decisions have no linked tickets?"

decision-ticket-report

Output:

Decisions without linked tickets:
- D-013: Insert/POI navigation system
- D-016: Internal monologue
- D-018: Three-range sound model
- D-021: Official title
- D-029: Population entanglement ratio

Total: 5 / 31 decisions (16%) lack implementation tickets.

Helps identify decisions that haven't been actioned yet.

Effort: ~30 minutes (simple SQL query + formatting). Priority: Medium. Useful for planning.

7. Open Questions for Team Review

Before proceeding with migration, I recommend team review on:

Domain taxonomy: Does the proposed 11-domain split (architecture, content, camera-perception, scope, world, combat, process, meta, multiplayer, questions, rejected) make sense? Should any domains be merged or split differently?
Cross-domain decisions: Some decisions span domains (e.g., D-026 simulation tiers touches both architecture and content). How should we handle this? Options:
- Put in primary domain (architecture) with cross-reference note in content file
- Duplicate in both files (violates DRY, but improves discoverability)
- Create "cross-domain" file for decisions that don't fit cleanly (13th file, not ideal)
Wrapper script priority: Should I build decision-add wrapper (6.1) before or after migration? Before = smoother migration. After = faster to production.
Linter as pre-commit hook: Should decision-lint (6.2) block commits if index/markdown drift detected? Or just warn?
Ticket-decision linkage: Should I audit and link all 31 decisions to existing tickets during migration? Or defer as ongoing maintenance?

8. Conclusion

The hybrid domain-split + DB index approach solves the immediate problem (context window bloat) and the long-term problems (cross-referencing, ticket linkage, discoverability, maintenance burden) without sacrificing the human-friendliness of markdown or the auditability of git history.

My recommendation: Proceed with migration. Estimated effort: ~4 hours (split markdown, seed DB, update briefings, re-index Qdrant). Risk: low (rollback path clear, no data loss). Benefit: 70-80% reduction in agent context load, 3x faster Qdrant re-indexing, structured cross-references, and a foundation for future scaling to 100+ decisions.

For the record: This is the right long-term structure. We should implement it now, before DECISIONS.md reaches 1000 lines and the migration becomes painful.

Next steps (pending team approval):

Review this analysis with Team Leader (Jeroen)
Finalize domain taxonomy (adjust if needed)
Execute migration (Phase 1-6, ~4 hours)
Validate and announce
Build wrapper script and linter (Phase 6.1-6.2, future enhancement)

Document ends. For questions or clarifications, ask Qatux.

47 KiB Raw Blame History Unescape Escape

Decision Architecture Restructure — Librarian's Analysis

Executive Summary

1. Problems with Current Approach at Scale

1.1 Context Window Inefficiency

1.2 Discovery Friction

1.3 Cross-Reference Complexity

1.4 Superseded Decision Cruft

1.5 Qdrant Re-indexing Overhead

1.6 Ticket Linkage Underutilized

2. Options Considered

2.1 Status Quo (Single File)

2.2 Split by Domain (Multiple Markdown Files)

2.3 Move to Database (SQLite)

2.4 Hybrid: Domain Files + DB Index

2.5 Tag-Based (Single File + Front Matter)

3. Recommended Approach: Hybrid Domain Files + DB Index

3.1 Why This Approach

3.2 Domain Taxonomy

3.3 Index DB Schema (Final)

3.4 Workflow Changes

For me (Qatux):

For agents:

3.5 Briefing Update Strategy

3.6 Superseded Decisions

4. Migration Path

Phase 1: Split Markdown (2 hours)

Phase 2: Seed Index DB (1.5 hours)

Phase 3: Update Briefings (1 hour)

Phase 4: Re-index Qdrant (30 minutes)

Phase 5: Commit and Announce (15 minutes)

Phase 6: Validation and Rollback Plan (if needed)

5. How This Works with My Workflow

5.1 Documentation Workflow

5.2 Qdrant Indexing

5.3 Briefing Maintenance

5.4 Cross-Reference Retrieval

6. Future Enhancements (Not MVP)

6.1 Wrapper Script for Adding Decisions

6.2 Decision Linter

6.3 Auto-Generated Domain README

6.4 Decision Timeline View

6.5 Decision-Ticket Linkage Report

7. Open Questions for Team Review

8. Conclusion

47 KiB

Raw Blame History