# Testing Protocol — Lords of Ash

> *How to test each milestone, what tools to use, and how Claude Code can help.*

## Overview

Each milestone has testing checkpoints defined in its milestone doc
(`docs/milestones/M<N>-*.md`). This document provides the shared testing
infrastructure: debug decisions, log analysis, helper scripts, and a
workflow that lets Claude Code do most of the heavy lifting.

The core idea: **temporary debug decisions** gated behind a `debug_mode` game
rule let us force specific game states for testing without relying on console
commands. Combined with log analysis scripts and structured test runs, Claude
Code can validate milestone completion semi-autonomously.

---

## File Locations

### CK3 Paths (this system)

| What | Path |
|------|------|
| **Game install** | `~/.local/share/Steam/steamapps/common/Crusader Kings III/` |
| **Vanilla game data** | `~/.local/share/Steam/steamapps/common/Crusader Kings III/game/` |
| **User data** | `~/.local/share/Paradox Interactive/Crusader Kings III/` |
| **Logs directory** | `~/.local/share/Paradox Interactive/Crusader Kings III/logs/` |
| **Mod descriptor (game)** | `~/.local/share/Paradox Interactive/Crusader Kings III/mod/lords-of-ash.mod` |
| **DLC/mod load config** | `~/.local/share/Paradox Interactive/Crusader Kings III/dlc_load.json` |
| **Mod project root** | `/var/home/jeroenschweitzer/Projects/lords-of-ash/` |
| **Mod game files** | `/var/home/jeroenschweitzer/Projects/lords-of-ash/lords_of_ash/` |

### Key Log Files

| File | Purpose | Typical size |
|------|---------|--------------|
| `error.log` | All errors and warnings. **Primary debug file.** | 5-25 MB |
| `game.log` | General game flow and loading stages | 5-10 MB |
| `setup.log` | Initialization, mod loading order | ~30 KB |
| `debug.log` | Verbose debug output | 5-10 MB |
| `text.log` | Missing localization keys | 0-1 MB |
| `database_conflicts.log` | Key conflicts between mod and vanilla | 0-100 KB |

### Shortcuts

```bash
# Alias for the logs directory
export CK3_LOGS="$HOME/.local/share/Paradox Interactive/Crusader Kings III/logs"
export CK3_GAME="$HOME/.local/share/Steam/steamapps/common/Crusader Kings III/game"
export CK3_MOD="/var/home/jeroenschweitzer/Projects/lords-of-ash/lords_of_ash"
```

---

## Log Analysis Commands

### Quick health check after a test run

```bash
# Clear old logs before a test run (do this BEFORE launching CK3)
: > "$CK3_LOGS/error.log"
: > "$CK3_LOGS/text.log"

# After the test run, check for mod-specific errors
grep -c 'mistborn\|lords.of.ash\|scadrial\|00_mistborn\|00_scadrial' "$CK3_LOGS/error.log"

# Show the first 50 mod-related errors (first errors are most important)
grep -n 'mistborn\|lords.of.ash\|scadrial\|00_mistborn\|00_scadrial' "$CK3_LOGS/error.log" | head -50

# Check for crash-causing errors
grep -i 'crash\|fatal\|assert\|CTD' "$CK3_LOGS/error.log"

# Check for encoding issues (causes cascading parse failures)
grep -i 'utf8-bom\|encoding' "$CK3_LOGS/error.log"

# Check for missing localization keys
grep -c 'Missing localization' "$CK3_LOGS/text.log"
grep 'mistborn\|scadrial' "$CK3_LOGS/text.log" | head -20

# Check mod loading order in setup.log
grep -i 'lords.of.ash\|mistborn' "$CK3_LOGS/setup.log"
```

### Tail logs in real-time during a test run

```bash
# Watch error.log live while playing (run in a separate terminal)
tail -f "$CK3_LOGS/error.log" | grep --line-buffered 'mistborn\|scadrial\|00_mistborn'

# Watch for any new errors (not mod-filtered)
tail -f "$CK3_LOGS/error.log" | grep --line-buffered -i 'error\|warning'

# Watch game.log for event firing
tail -f "$CK3_LOGS/game.log" | grep --line-buffered 'mistborn'
```

### Categorize errors

```bash
# Count errors by type
grep 'mistborn\|scadrial' "$CK3_LOGS/error.log" | \
  sed 's/.*\(Missing\|Invalid\|Unexpected\|Duplicate\|Unknown\).*/\1/' | \
  sort | uniq -c | sort -rn

# Find which files have the most errors
grep 'mistborn\|scadrial' "$CK3_LOGS/error.log" | \
  grep -oP '[^\s]*\.txt' | sort | uniq -c | sort -rn | head -20

# Check for specific error patterns
grep 'Missing closing bracket' "$CK3_LOGS/error.log" | head -10
grep 'Invalid scope' "$CK3_LOGS/error.log" | head -10
grep 'not found' "$CK3_LOGS/error.log" | grep 'mistborn\|scadrial' | head -10
```

---

## Debug Game Rule

All debug decisions are gated behind a game rule that is **off by default**.
This rule should be created as part of M1 (Foundation) alongside the
Narrative/Sandbox toggle.

### Game Rule Definition

File: `lords_of_ash/common/game_rules/00_mistborn_game_rules.txt`

```
debug_mode = {
    default = debug_off
    debug_off = {
        # Normal gameplay — no debug decisions visible
    }
    debug_on = {
        # Debug decisions appear in the decision list
        # Console-like power for testing without opening console
    }
}
```

### Debug Decision Template

All debug decisions follow this pattern:

```
debug_<action>_decision = {
    is_shown = {
        has_game_rule = debug_on
        is_ai = no
    }
    # No cost, no cooldown, immediate effect
    effect = {
        # ... the debug action
    }
}
```

File: `lords_of_ash/common/decisions/99_debug_decisions.txt`

The `99_` prefix ensures debug decisions load last and sort to the bottom.
This file is **removed or emptied before Steam release** (M8 task).

---

## Debug Decisions Per Milestone

### M1 — Foundation

No debug decisions needed. M1 validates that the mod loads.

**Test method:** Boot the game, check `error.log`, verify no crashes.

---

### M2 — The Metallic Arts

| Decision | Effect | Tests |
|----------|--------|-------|
| `debug_grant_allomantic_potential` | Adds `trait_allomantic_potential` to player | Snapping prerequisites |
| `debug_trigger_snapping` | Forces snapping event chain on player | Snapping event flow |
| `debug_grant_mistborn` | Adds `trait_mistborn` + all Misting traits | Trait interactions, compounding |
| `debug_grant_feruchemist` | Adds `trait_feruchemist` | Compounding checks |
| `debug_grant_hemalurgist` | Adds `trait_hemalurgist` + spike traits | Hemalurgy events |
| `debug_set_metal_supply` | Cycles metal supply: abundant → low → depleted | Metal scarcity events |
| `debug_set_allomantic_strength` | Cycles: weak → normal → strong → savant | Strength tier effects |
| `debug_spawn_misting_courtier` | Spawns a courtier with random Misting trait | Court composition testing |

---

### M3 — The Final Empire

| Decision | Effect | Tests |
|----------|--------|-------|
| `debug_set_attention_0` | Sets Lord Ruler attention to 0 | Attention floor behavior |
| `debug_set_attention_50` | Sets Lord Ruler attention to 50 | Mid-tier attention events |
| `debug_set_attention_100` | Sets Lord Ruler attention to 100 | Max attention consequences |
| `debug_spawn_inquisitor` | Spawns a Steel Inquisitor in player's court | Inquisitor interactions |
| `debug_join_keepers` | Instantly grants `keeper_identity` flag | Keeper network events |
| `debug_grant_kandra_contract` | Spawns a kandra in player's court | Kandra mechanics |
| `debug_trigger_education` | Starts education for a child immediately | Education track testing |
| `debug_unlock_legacy_tier` | Grants 1000 renown + unlocks next legacy | Legacy perk validation |
| `debug_reveal_all_secrets` | Exposes all secrets in player's court | Secret/hook system |
| `debug_spawn_artifact` | Spawns a menu of artifact types to choose from | Artifact system |

---

### M4 — Scadrial on the Map

| Decision | Effect | Tests |
|----------|--------|-------|
| `debug_declare_house_war` | Instantly starts House War CB against random neighbor | CB mechanics |
| `debug_start_scheme` | Opens a scheme against random courtier | Scheme flow |
| `debug_set_caste_law` | Cycles: strict → relaxed → abolished | Caste law effects |
| `debug_create_half_skaa` | Creates a half-skaa child with hidden heritage | Caste events |
| `debug_set_government` | Cycles: imperial → theocratic → elective | Government transitions |
| `debug_create_faction` | Spawns a rebellion faction at 80% power | Faction ultimatum |
| `debug_change_contract` | Sets all vassal contracts to max obligation | Contract effects |
| `debug_add_secret` | Menu: choose a secret type to add to player | Secret integration |
| `debug_set_court_type` | Cycles court types: keep → canton → hideout | Court type behavior |
| `debug_unlock_all_innovations` | Unlocks all innovations (both eras) | Innovation effects |
| `debug_set_era_liberation` | Sets `lord_ruler_fallen` flag | Post-rebellion systems |
| `debug_assign_lifestyle` | Menu: choose a lifestyle + focus | Lifestyle/perk testing |
| `debug_host_activity` | Menu: choose an activity to host immediately | Activity flow |

---

### M5 — The Crew

| Decision | Effect | Tests |
|----------|--------|-------|
| `debug_toggle_game_mode` | Switches Narrative ↔ Sandbox mode mid-game | Game rule gating |
| `debug_fill_crew` | Spawns all 8 crew members if missing | Crew completeness |
| `debug_kill_kelsier` | Kills Kelsier, triggers succession | Vin succession mechanic |
| `debug_assign_crew_roles` | Resets all crew roles to defaults | Band council |
| `debug_vacate_crew_role` | Removes a crew member from their role | Vacancy penalties |
| `debug_simulate_no_dlc` | Disables RtP features, tests fallback | Non-DLC path |

---

### M6 — The Plan

| Decision | Effect | Tests |
|----------|--------|-------|
| `debug_complete_crew_assembly` | Sets all crew assembly flags as done | Skip to later arcs |
| `debug_advance_skaa_army` | Sets skaa army to max strength | Army arc testing |
| `debug_trigger_infiltration` | Starts noble infiltration arc immediately | Infiltration events |
| `debug_start_rebellion` | Forces rebellion arc to fire | Rebellion mechanics |
| `debug_trigger_martyrdom` | Kills Kelsier in rebellion context | Martyrdom event chain |
| `debug_skip_to_arc` | Menu: jump to any story arc | Non-linear testing |

---

### M7 — The Deepness

| Decision | Effect | Tests |
|----------|--------|-------|
| `debug_trigger_raid` | Starts a raid on nearest noble house | Raid event chain |
| `debug_discover_eleventh_metal` | Grants eleventh metal insight immediately | Eleventh metal prereq |
| `debug_trigger_duel` | Forces a Mistborn duel with random opponent | Duel mechanics |
| `debug_trigger_ball_assassination` | Forces assassination variant at next ball | Ball assassination |
| `debug_set_atium_supply` | Cycles atium economy states | Atium system |
| `debug_start_endgame` | Sets all endgame prerequisites, fires assault | Fall of Kredik Shaw |
| `debug_spawn_artifact_rare` | Menu: choose rare/lore artifact to spawn | Rare artifact testing |
| `debug_host_mistborn_activity` | Menu: choose Mistborn activity to host | Activity testing |
| `debug_trigger_council_event` | Menu: choose a council interaction event | Council events |

---

### M8 — The Art of Metals

No unique debug decisions needed. M8 is graphics, map, and polish. Testing
uses the existing debug decisions from M2-M7.

**Focus:** Error.log is completely clean. Visual review of icons, portraits,
map rendering. Performance profiling.

---

## Testing Workflow

### Per-Milestone Test Protocol

When a milestone is feature-complete, run this workflow:

#### 1. Pre-test setup

```bash
# Clear logs
: > "$CK3_LOGS/error.log"
: > "$CK3_LOGS/text.log"
: > "$CK3_LOGS/game.log"

# Verify mod is in active playset
cat "$HOME/.local/share/Paradox Interactive/Crusader Kings III/dlc_load.json"

# Verify mod descriptor path is correct
cat "$HOME/.local/share/Paradox Interactive/Crusader Kings III/mod/lords-of-ash.mod"
```

#### 2. Boot test

Launch CK3 with the mod enabled. Do NOT start a game yet.

```bash
# After CK3 loads to main menu, check for load errors:
grep -c 'mistborn\|scadrial' "$CK3_LOGS/error.log"
# Target: 0 mod-related errors

# Check setup.log for mod loading confirmation
grep 'lords.of.ash' "$CK3_LOGS/setup.log"
```

#### 3. Start a test game

- Select "End of an Empire" bookmark
- Enable `Debug Mode` game rule (set to `debug_on`)
- Start as the character specified in the milestone's testing checkpoint

#### 4. Execute testing checkpoints

Follow the testing checkpoints in the milestone doc. Use debug decisions to
force conditions that are hard to trigger naturally.

For each checkpoint:
1. Use the debug decision to set up the required state
2. Verify the expected outcome
3. Check `error.log` for new errors after each test
4. Note any issues in a test report

#### 5. Extended playthrough

After all checkpoints pass:

```bash
# Clear logs again for the extended run
: > "$CK3_LOGS/error.log"
```

- Play 20 in-game years (or as specified in the milestone doc)
- Observe for: vanilla bleed-through, event spam, broken mechanics,
  performance degradation

```bash
# After the playthrough, full log analysis:
echo "=== Error count ==="
wc -l "$CK3_LOGS/error.log"

echo "=== Mod-specific errors ==="
grep -c 'mistborn\|scadrial' "$CK3_LOGS/error.log"

echo "=== Error categories ==="
grep 'mistborn\|scadrial' "$CK3_LOGS/error.log" | \
  grep -oP '(Missing|Invalid|Unexpected|Duplicate|Unknown)\w*' | \
  sort | uniq -c | sort -rn

echo "=== Missing localization ==="
grep -c 'mistborn\|scadrial' "$CK3_LOGS/text.log"

echo "=== Files with errors ==="
grep 'mistborn\|scadrial' "$CK3_LOGS/error.log" | \
  grep -oP '[^\s/]*\.txt' | sort | uniq -c | sort -rn | head -20
```

#### 6. Generate test report

Claude Code can generate a structured test report by:
1. Reading `error.log` and filtering for mod errors
2. Categorizing errors by type and file
3. Cross-referencing against the milestone's acceptance criteria
4. Producing a pass/fail checklist

---

## Claude Code Testing Capabilities

### What Claude Code can do autonomously

- **Static analysis:** Read mod files and check for syntax issues, missing
  references, unmatched brackets, missing localization keys
- **Log analysis:** Read and categorize error.log after a test run
- **Cross-reference validation:** Check that every trait referenced in events
  exists in trait files, every localization key used has a definition, every
  modifier referenced is defined
- **Encoding checks:** Verify all `.txt` and `.yml` files have UTF-8 BOM
- **Vanilla comparison:** Compare mod file structure against vanilla to find
  missing required fields

### What requires human interaction

- **Launching CK3:** Claude Code cannot start the game
- **Gameplay testing:** Playing the game, clicking through events, visual
  inspection
- **Screenshot review:** Checking that UI elements display correctly
- **Subjective quality:** "Does this feel like Scadrial?"

### Pre-launch validation script

Claude Code should run this before every test launch to catch issues early:

```bash
#!/bin/bash
# pre-test-validate.sh — Run before launching CK3
# Usage: bash docs/scripts/pre-test-validate.sh

MOD_DIR="/var/home/jeroenschweitzer/Projects/lords-of-ash/lords_of_ash"
ERRORS=0

echo "=== Lords of Ash Pre-Test Validation ==="
echo ""

# 1. Check encoding on all .txt and .yml files
echo "--- Encoding check ---"
find "$MOD_DIR" -name '*.txt' -o -name '*.yml' | while read f; do
    if ! file "$f" | grep -q 'BOM'; then
        echo "MISSING BOM: $f"
        ERRORS=$((ERRORS + 1))
    fi
done

# 2. Check for unmatched braces in .txt files
echo "--- Brace matching ---"
find "$MOD_DIR" -name '*.txt' | while read f; do
    OPEN=$(grep -o '{' "$f" | wc -l)
    CLOSE=$(grep -o '}' "$f" | wc -l)
    if [ "$OPEN" -ne "$CLOSE" ]; then
        echo "BRACE MISMATCH: $f (open=$OPEN, close=$CLOSE)"
    fi
done

# 3. Check for @# comment prefix (CK3 uses # not @#)
echo "--- Invalid comment prefix ---"
grep -rn '@#' "$MOD_DIR" --include='*.txt' | head -10

# 4. Check mod descriptor
echo "--- Mod descriptor ---"
if [ -f "$HOME/.local/share/Paradox Interactive/Crusader Kings III/mod/lords-of-ash.mod" ]; then
    echo "OK: Mod descriptor exists"
    grep 'path=' "$HOME/.local/share/Paradox Interactive/Crusader Kings III/mod/lords-of-ash.mod"
else
    echo "MISSING: Mod descriptor not found in game mod directory"
fi

# 5. Check that debug decisions file exists (if past M1)
if [ -f "$MOD_DIR/common/decisions/99_debug_decisions.txt" ]; then
    echo "OK: Debug decisions file exists"
else
    echo "NOTE: No debug decisions file yet (expected before M2 testing)"
fi

echo ""
echo "=== Validation complete ==="
```

### Post-run analysis script

```bash
#!/bin/bash
# post-test-analyze.sh — Run after a CK3 test session
# Usage: bash docs/scripts/post-test-analyze.sh

CK3_LOGS="$HOME/.local/share/Paradox Interactive/Crusader Kings III/logs"

echo "=== Lords of Ash Post-Test Analysis ==="
echo "Log directory: $CK3_LOGS"
echo "Analysis time: $(date)"
echo ""

# 1. Error.log summary
echo "--- error.log ---"
TOTAL=$(wc -l < "$CK3_LOGS/error.log" 2>/dev/null || echo 0)
MOD=$(grep -c 'mistborn\|scadrial\|lords.of.ash' "$CK3_LOGS/error.log" 2>/dev/null || echo 0)
echo "Total errors: $TOTAL"
echo "Mod-related: $MOD"

if [ "$MOD" -gt 0 ]; then
    echo ""
    echo "Error categories:"
    grep 'mistborn\|scadrial' "$CK3_LOGS/error.log" | \
      grep -oP '(Missing|Invalid|Unexpected|Duplicate|Unknown)\S*' | \
      sort | uniq -c | sort -rn

    echo ""
    echo "Top 10 files with errors:"
    grep 'mistborn\|scadrial' "$CK3_LOGS/error.log" | \
      grep -oP '\S*\.txt' | sort | uniq -c | sort -rn | head -10

    echo ""
    echo "First 10 mod errors:"
    grep -n 'mistborn\|scadrial' "$CK3_LOGS/error.log" | head -10
fi

# 2. Missing localization
echo ""
echo "--- text.log (missing localization) ---"
LOC=$(grep -c 'mistborn\|scadrial' "$CK3_LOGS/text.log" 2>/dev/null || echo 0)
echo "Missing mod localization keys: $LOC"
if [ "$LOC" -gt 0 ]; then
    echo "Keys:"
    grep 'mistborn\|scadrial' "$CK3_LOGS/text.log" | head -20
fi

# 3. Database conflicts
echo ""
echo "--- database_conflicts.log ---"
CONF=$(wc -l < "$CK3_LOGS/database_conflicts.log" 2>/dev/null || echo 0)
echo "Database conflicts: $CONF"

# 4. Crash indicators
echo ""
echo "--- Crash indicators ---"
CRASHES=$(grep -ci 'crash\|fatal\|assert\|CTD' "$CK3_LOGS/error.log" 2>/dev/null || echo 0)
echo "Crash-related entries: $CRASHES"
if [ "$CRASHES" -gt 0 ]; then
    grep -i 'crash\|fatal\|assert\|CTD' "$CK3_LOGS/error.log" | head -5
fi

echo ""
echo "=== Analysis complete ==="
if [ "$MOD" -eq 0 ] && [ "$LOC" -eq 0 ] && [ "$CRASHES" -eq 0 ]; then
    echo "RESULT: CLEAN — no mod errors, no missing loc, no crashes"
else
    echo "RESULT: ISSUES FOUND — see details above"
fi
```

---

## Static Validation Checks (Claude Code)

These checks can be run without launching CK3. Claude Code should run them
as part of milestone review:

### 1. Localization completeness

For every trait, event, decision, modifier, and title defined in mod files,
verify a matching localization key exists in `localization/english/`.

```
Pattern: trait_<name> → needs trait_<name>:0 and trait_<name>_desc:0
Pattern: <event_namespace>.<id>.t → event title
Pattern: <event_namespace>.<id>.desc → event description
Pattern: <decision_name> → needs <decision_name>:0 and <decision_name>_desc:0
```

### 2. Reference integrity

- Every `has_trait = trait_X` in triggers/effects → `trait_X` exists in
  `common/traits/`
- Every `has_modifier = mod_X` → `mod_X` exists in `common/modifiers/`
- Every `trigger_event = { id = X }` → event `X` exists in `events/`
- Every `faith = faith:X` → faith `X` exists in `common/religion/`
- Every `culture = culture:X` → culture `X` exists in `common/culture/`

### 3. Encoding audit

Every `.txt` and `.yml` file in `lords_of_ash/` must be UTF-8 with BOM.

### 4. Comment syntax

CK3 uses `#` for comments. The `@` symbol is for scripted variables only.
No `@#` patterns should exist in any `.txt` file.

---

## Release Checklist (M8 — Before Steam)

Before publishing to Steam Workshop:

- [ ] Remove or gate `99_debug_decisions.txt` (hide behind a game rule that
      is not visible to players, or delete the file entirely)
- [ ] Remove `debug_mode` game rule from game setup screen (or rename to
      something hidden)
- [ ] Run full error.log clean pass (zero mod errors)
- [ ] Run `text.log` clean pass (zero missing localization)
- [ ] Verify `lords-of-ash.mod` has correct `supported_version` for current
      CK3 patch
- [ ] Test without any other mods enabled
- [ ] Test with Roads to Power and Royal Court DLC both enabled and disabled
- [ ] Verify non-DLC fallbacks work (M5)
- [ ] Full Narrative Mode playthrough: bookmark → endgame
- [ ] Full Sandbox Mode playthrough: 30 years as a noble house
- [ ] Performance: game speed acceptable at 5-speed for 50 years