lords-of-ash/docs/TESTING.md

Testing Protocol — Lords of Ash

How to test each milestone, what tools to use, and how Claude Code can help.

Overview

Each milestone has testing checkpoints defined in its milestone doc (docs/milestones/M<N>-*.md). This document provides the shared testing infrastructure: debug decisions, log analysis, helper scripts, and a workflow that lets Claude Code do most of the heavy lifting.

The core idea: temporary debug decisions gated behind a debug_mode game rule let us force specific game states for testing without relying on console commands. Combined with log analysis scripts and structured test runs, Claude Code can validate milestone completion semi-autonomously.

---

File Locations

CK3 Paths (this system)

What	Path
Game install	~/.local/share/Steam/steamapps/common/Crusader Kings III/
Vanilla game data	~/.local/share/Steam/steamapps/common/Crusader Kings III/game/
User data	~/.local/share/Paradox Interactive/Crusader Kings III/
Logs directory	~/.local/share/Paradox Interactive/Crusader Kings III/logs/
Mod descriptor (game)	~/.local/share/Paradox Interactive/Crusader Kings III/mod/lords-of-ash.mod
DLC/mod load config	~/.local/share/Paradox Interactive/Crusader Kings III/dlc_load.json
Mod project root	/var/home/jeroenschweitzer/Projects/lords-of-ash/
Mod game files	/var/home/jeroenschweitzer/Projects/lords-of-ash/lords_of_ash/

Key Log Files

File	Purpose	Typical size
error.log	All errors and warnings. Primary debug file.	5-25 MB
game.log	General game flow and loading stages	5-10 MB
setup.log	Initialization, mod loading order	~30 KB
debug.log	Verbose debug output	5-10 MB
text.log	Missing localization keys	0-1 MB
database_conflicts.log	Key conflicts between mod and vanilla	0-100 KB

Shortcuts

# Alias for the logs directory
export CK3_LOGS="$HOME/.local/share/Paradox Interactive/Crusader Kings III/logs"
export CK3_GAME="$HOME/.local/share/Steam/steamapps/common/Crusader Kings III/game"
export CK3_MOD="/var/home/jeroenschweitzer/Projects/lords-of-ash/lords_of_ash"

---

Log Analysis Commands

Quick health check after a test run

# Clear old logs before a test run (do this BEFORE launching CK3)
: > "$CK3_LOGS/error.log"
: > "$CK3_LOGS/text.log"

# After the test run, check for mod-specific errors
grep -c 'mistborn\|lords.of.ash\|scadrial\|00_mistborn\|00_scadrial' "$CK3_LOGS/error.log"

# Show the first 50 mod-related errors (first errors are most important)
grep -n 'mistborn\|lords.of.ash\|scadrial\|00_mistborn\|00_scadrial' "$CK3_LOGS/error.log" | head -50

# Check for crash-causing errors
grep -i 'crash\|fatal\|assert\|CTD' "$CK3_LOGS/error.log"

# Check for encoding issues (causes cascading parse failures)
grep -i 'utf8-bom\|encoding' "$CK3_LOGS/error.log"

# Check for missing localization keys
grep -c 'Missing localization' "$CK3_LOGS/text.log"
grep 'mistborn\|scadrial' "$CK3_LOGS/text.log" | head -20

# Check mod loading order in setup.log
grep -i 'lords.of.ash\|mistborn' "$CK3_LOGS/setup.log"

Tail logs in real-time during a test run

# Watch error.log live while playing (run in a separate terminal)
tail -f "$CK3_LOGS/error.log" | grep --line-buffered 'mistborn\|scadrial\|00_mistborn'

# Watch for any new errors (not mod-filtered)
tail -f "$CK3_LOGS/error.log" | grep --line-buffered -i 'error\|warning'

# Watch game.log for event firing
tail -f "$CK3_LOGS/game.log" | grep --line-buffered 'mistborn'

Categorize errors

# Count errors by type
grep 'mistborn\|scadrial' "$CK3_LOGS/error.log" | \
  sed 's/.*\(Missing\|Invalid\|Unexpected\|Duplicate\|Unknown\).*/\1/' | \
  sort | uniq -c | sort -rn

# Find which files have the most errors
grep 'mistborn\|scadrial' "$CK3_LOGS/error.log" | \
  grep -oP '[^\s]*\.txt' | sort | uniq -c | sort -rn | head -20

# Check for specific error patterns
grep 'Missing closing bracket' "$CK3_LOGS/error.log" | head -10
grep 'Invalid scope' "$CK3_LOGS/error.log" | head -10
grep 'not found' "$CK3_LOGS/error.log" | grep 'mistborn\|scadrial' | head -10

---

Debug Game Rule

All debug decisions are gated behind a game rule that is off by default. This rule should be created as part of M1 (Foundation) alongside the Narrative/Sandbox toggle.

Game Rule Definition

File: lords_of_ash/common/game_rules/00_mistborn_game_rules.txt

debug_mode = {
    default = debug_off
    debug_off = {
        # Normal gameplay — no debug decisions visible
    }
    debug_on = {
        # Debug decisions appear in the decision list
        # Console-like power for testing without opening console
    }
}

Debug Decision Template

All debug decisions follow this pattern:

debug_<action>_decision = {
    is_shown = {
        has_game_rule = debug_on
        is_ai = no
    }
    # No cost, no cooldown, immediate effect
    effect = {
        # ... the debug action
    }
}

File: lords_of_ash/common/decisions/99_debug_decisions.txt

The 99_ prefix ensures debug decisions load last and sort to the bottom. This file is removed or emptied before Steam release (M8 task).

---

Debug Decisions Per Milestone

M1 — Foundation

No debug decisions needed. M1 validates that the mod loads.

Test method: Boot the game, check error.log, verify no crashes.

---

M2 — The Metallic Arts

Decision	Effect	Tests
debug_grant_allomantic_potential	Adds trait_allomantic_potential to player	Snapping prerequisites
debug_trigger_snapping	Forces snapping event chain on player	Snapping event flow
debug_grant_mistborn	Adds trait_mistborn + all Misting traits	Trait interactions, compounding
debug_grant_feruchemist	Adds trait_feruchemist	Compounding checks
debug_grant_hemalurgist	Adds trait_hemalurgist + spike traits	Hemalurgy events
debug_set_metal_supply	Cycles metal supply: abundant → low → depleted	Metal scarcity events
debug_set_allomantic_strength	Cycles: weak → normal → strong → savant	Strength tier effects
debug_spawn_misting_courtier	Spawns a courtier with random Misting trait	Court composition testing

---

M3 — The Final Empire

Decision	Effect	Tests
debug_set_attention_0	Sets Lord Ruler attention to 0	Attention floor behavior
debug_set_attention_50	Sets Lord Ruler attention to 50	Mid-tier attention events
debug_set_attention_100	Sets Lord Ruler attention to 100	Max attention consequences
debug_spawn_inquisitor	Spawns a Steel Inquisitor in player's court	Inquisitor interactions
debug_join_keepers	Instantly grants keeper_identity flag	Keeper network events
debug_grant_kandra_contract	Spawns a kandra in player's court	Kandra mechanics
debug_trigger_education	Starts education for a child immediately	Education track testing
debug_unlock_legacy_tier	Grants 1000 renown + unlocks next legacy	Legacy perk validation
debug_reveal_all_secrets	Exposes all secrets in player's court	Secret/hook system
debug_spawn_artifact	Spawns a menu of artifact types to choose from	Artifact system

---

M4 — Scadrial on the Map

Decision	Effect	Tests
debug_declare_house_war	Instantly starts House War CB against random neighbor	CB mechanics
debug_start_scheme	Opens a scheme against random courtier	Scheme flow
debug_set_caste_law	Cycles: strict → relaxed → abolished	Caste law effects
debug_create_half_skaa	Creates a half-skaa child with hidden heritage	Caste events
debug_set_government	Cycles: imperial → theocratic → elective	Government transitions
debug_create_faction	Spawns a rebellion faction at 80% power	Faction ultimatum
debug_change_contract	Sets all vassal contracts to max obligation	Contract effects
debug_add_secret	Menu: choose a secret type to add to player	Secret integration
debug_set_court_type	Cycles court types: keep → canton → hideout	Court type behavior
debug_unlock_all_innovations	Unlocks all innovations (both eras)	Innovation effects
debug_set_era_liberation	Sets lord_ruler_fallen flag	Post-rebellion systems
debug_assign_lifestyle	Menu: choose a lifestyle + focus	Lifestyle/perk testing
debug_host_activity	Menu: choose an activity to host immediately	Activity flow

---

M5 — The Crew

Decision	Effect	Tests
debug_toggle_game_mode	Switches Narrative ↔ Sandbox mode mid-game	Game rule gating
debug_fill_crew	Spawns all 8 crew members if missing	Crew completeness
debug_kill_kelsier	Kills Kelsier, triggers succession	Vin succession mechanic
debug_assign_crew_roles	Resets all crew roles to defaults	Band council
debug_vacate_crew_role	Removes a crew member from their role	Vacancy penalties
debug_simulate_no_dlc	Disables RtP features, tests fallback	Non-DLC path

---

M6 — The Plan

Decision	Effect	Tests
debug_complete_crew_assembly	Sets all crew assembly flags as done	Skip to later arcs
debug_advance_skaa_army	Sets skaa army to max strength	Army arc testing
debug_trigger_infiltration	Starts noble infiltration arc immediately	Infiltration events
debug_start_rebellion	Forces rebellion arc to fire	Rebellion mechanics
debug_trigger_martyrdom	Kills Kelsier in rebellion context	Martyrdom event chain
debug_skip_to_arc	Menu: jump to any story arc	Non-linear testing

---

M7 — The Deepness

Decision	Effect	Tests
debug_trigger_raid	Starts a raid on nearest noble house	Raid event chain
debug_discover_eleventh_metal	Grants eleventh metal insight immediately	Eleventh metal prereq
debug_trigger_duel	Forces a Mistborn duel with random opponent	Duel mechanics
debug_trigger_ball_assassination	Forces assassination variant at next ball	Ball assassination
debug_set_atium_supply	Cycles atium economy states	Atium system
debug_start_endgame	Sets all endgame prerequisites, fires assault	Fall of Kredik Shaw
debug_spawn_artifact_rare	Menu: choose rare/lore artifact to spawn	Rare artifact testing
debug_host_mistborn_activity	Menu: choose Mistborn activity to host	Activity testing
debug_trigger_council_event	Menu: choose a council interaction event	Council events

---

M8 — The Art of Metals

No unique debug decisions needed. M8 is graphics, map, and polish. Testing uses the existing debug decisions from M2-M7.

Focus: Error.log is completely clean. Visual review of icons, portraits, map rendering. Performance profiling.

---

Testing Workflow

Per-Milestone Test Protocol

When a milestone is feature-complete, run this workflow:

1. Pre-test setup

# Clear logs
: > "$CK3_LOGS/error.log"
: > "$CK3_LOGS/text.log"
: > "$CK3_LOGS/game.log"

# Verify mod is in active playset
cat "$HOME/.local/share/Paradox Interactive/Crusader Kings III/dlc_load.json"

# Verify mod descriptor path is correct
cat "$HOME/.local/share/Paradox Interactive/Crusader Kings III/mod/lords-of-ash.mod"

2. Boot test

Launch CK3 with the mod enabled. Do NOT start a game yet.

# After CK3 loads to main menu, check for load errors:
grep -c 'mistborn\|scadrial' "$CK3_LOGS/error.log"
# Target: 0 mod-related errors

# Check setup.log for mod loading confirmation
grep 'lords.of.ash' "$CK3_LOGS/setup.log"

3. Start a test game

• Select "End of an Empire" bookmark
• Enable Debug Mode game rule (set to debug_on)
• Start as the character specified in the milestone's testing checkpoint

4. Execute testing checkpoints

Follow the Player Test Script in the milestone doc (section below). Each milestone has a numbered list of specific in-game actions the player must perform. Claude Code cannot launch CK3 or interact with the UI — these steps require a human.

For each checkpoint:

1. Perform the specific action described in the test script
2. Verify the expected visual/gameplay outcome
3. Note any issues (wrong text, missing icons, broken behavior)
4. After all actions, quit to desktop so Claude Code can analyze logs

5. Extended playthrough

After all checkpoints pass:

# Clear logs again for the extended run
: > "$CK3_LOGS/error.log"

• Play the number of years specified in the milestone doc (default: 5 years)
• Set game speed to 4 or 5
• Observe for: vanilla bleed-through, event spam, broken mechanics, performance degradation
• Note anything that looks wrong, even if it doesn't crash

# After the playthrough, full log analysis:
echo "=== Error count ==="
wc -l "$CK3_LOGS/error.log"

echo "=== Mod-specific errors ==="
grep -c 'mistborn\|scadrial' "$CK3_LOGS/error.log"

echo "=== Error categories ==="
grep 'mistborn\|scadrial' "$CK3_LOGS/error.log" | \
  grep -oP '(Missing|Invalid|Unexpected|Duplicate|Unknown)\w*' | \
  sort | uniq -c | sort -rn

echo "=== Missing localization ==="
grep -c 'mistborn\|scadrial' "$CK3_LOGS/text.log"

echo "=== Files with errors ==="
grep 'mistborn\|scadrial' "$CK3_LOGS/error.log" | \
  grep -oP '[^\s/]*\.txt' | sort | uniq -c | sort -rn | head -20

6. Generate test report

Claude Code can generate a structured test report by:

1. Reading error.log and filtering for mod errors
2. Categorizing errors by type and file
3. Cross-referencing against the milestone's acceptance criteria
4. Producing a pass/fail checklist

---

Claude Code Testing Capabilities

What Claude Code can do autonomously

• Static analysis: Read mod files and check for syntax issues, missing references, unmatched brackets, missing localization keys
• Log analysis: Read and categorize error.log after a test run
• Cross-reference validation: Check that every trait referenced in events exists in trait files, every localization key used has a definition, every modifier referenced is defined
• Encoding checks: Verify all .txt and .yml files have UTF-8 BOM
• Vanilla comparison: Compare mod file structure against vanilla to find missing required fields

What requires human interaction

• Launching CK3: Claude Code cannot start the game
• Gameplay testing: Playing the game, clicking through events, visual inspection
• Screenshot review: Checking that UI elements display correctly
• Subjective quality: "Does this feel like Scadrial?"

Pre-launch validation script

Claude Code should run this before every test launch to catch issues early:

#!/bin/bash
# pre-test-validate.sh — Run before launching CK3
# Usage: bash docs/scripts/pre-test-validate.sh

MOD_DIR="/var/home/jeroenschweitzer/Projects/lords-of-ash/lords_of_ash"
ERRORS=0

echo "=== Lords of Ash Pre-Test Validation ==="
echo ""

# 1. Check encoding on all .txt and .yml files
echo "--- Encoding check ---"
find "$MOD_DIR" -name '*.txt' -o -name '*.yml' | while read f; do
    if ! file "$f" | grep -q 'BOM'; then
        echo "MISSING BOM: $f"
        ERRORS=$((ERRORS + 1))
    fi
done

# 2. Check for unmatched braces in .txt files
echo "--- Brace matching ---"
find "$MOD_DIR" -name '*.txt' | while read f; do
    OPEN=$(grep -o '{' "$f" | wc -l)
    CLOSE=$(grep -o '}' "$f" | wc -l)
    if [ "$OPEN" -ne "$CLOSE" ]; then
        echo "BRACE MISMATCH: $f (open=$OPEN, close=$CLOSE)"
    fi
done

# 3. Check for @# comment prefix (CK3 uses # not @#)
echo "--- Invalid comment prefix ---"
grep -rn '@#' "$MOD_DIR" --include='*.txt' | head -10

# 4. Check mod descriptor
echo "--- Mod descriptor ---"
if [ -f "$HOME/.local/share/Paradox Interactive/Crusader Kings III/mod/lords-of-ash.mod" ]; then
    echo "OK: Mod descriptor exists"
    grep 'path=' "$HOME/.local/share/Paradox Interactive/Crusader Kings III/mod/lords-of-ash.mod"
else
    echo "MISSING: Mod descriptor not found in game mod directory"
fi

# 5. Check that debug decisions file exists (if past M1)
if [ -f "$MOD_DIR/common/decisions/99_debug_decisions.txt" ]; then
    echo "OK: Debug decisions file exists"
else
    echo "NOTE: No debug decisions file yet (expected before M2 testing)"
fi

echo ""
echo "=== Validation complete ==="

Post-run analysis script

#!/bin/bash
# post-test-analyze.sh — Run after a CK3 test session
# Usage: bash docs/scripts/post-test-analyze.sh

CK3_LOGS="$HOME/.local/share/Paradox Interactive/Crusader Kings III/logs"

echo "=== Lords of Ash Post-Test Analysis ==="
echo "Log directory: $CK3_LOGS"
echo "Analysis time: $(date)"
echo ""

# 1. Error.log summary
echo "--- error.log ---"
TOTAL=$(wc -l < "$CK3_LOGS/error.log" 2>/dev/null || echo 0)
MOD=$(grep -c 'mistborn\|scadrial\|lords.of.ash' "$CK3_LOGS/error.log" 2>/dev/null || echo 0)
echo "Total errors: $TOTAL"
echo "Mod-related: $MOD"

if [ "$MOD" -gt 0 ]; then
    echo ""
    echo "Error categories:"
    grep 'mistborn\|scadrial' "$CK3_LOGS/error.log" | \
      grep -oP '(Missing|Invalid|Unexpected|Duplicate|Unknown)\S*' | \
      sort | uniq -c | sort -rn

    echo ""
    echo "Top 10 files with errors:"
    grep 'mistborn\|scadrial' "$CK3_LOGS/error.log" | \
      grep -oP '\S*\.txt' | sort | uniq -c | sort -rn | head -10

    echo ""
    echo "First 10 mod errors:"
    grep -n 'mistborn\|scadrial' "$CK3_LOGS/error.log" | head -10
fi

# 2. Missing localization
echo ""
echo "--- text.log (missing localization) ---"
LOC=$(grep -c 'mistborn\|scadrial' "$CK3_LOGS/text.log" 2>/dev/null || echo 0)
echo "Missing mod localization keys: $LOC"
if [ "$LOC" -gt 0 ]; then
    echo "Keys:"
    grep 'mistborn\|scadrial' "$CK3_LOGS/text.log" | head -20
fi

# 3. Database conflicts
echo ""
echo "--- database_conflicts.log ---"
CONF=$(wc -l < "$CK3_LOGS/database_conflicts.log" 2>/dev/null || echo 0)
echo "Database conflicts: $CONF"

# 4. Crash indicators
echo ""
echo "--- Crash indicators ---"
CRASHES=$(grep -ci 'crash\|fatal\|assert\|CTD' "$CK3_LOGS/error.log" 2>/dev/null || echo 0)
echo "Crash-related entries: $CRASHES"
if [ "$CRASHES" -gt 0 ]; then
    grep -i 'crash\|fatal\|assert\|CTD' "$CK3_LOGS/error.log" | head -5
fi

echo ""
echo "=== Analysis complete ==="
if [ "$MOD" -eq 0 ] && [ "$LOC" -eq 0 ] && [ "$CRASHES" -eq 0 ]; then
    echo "RESULT: CLEAN — no mod errors, no missing loc, no crashes"
else
    echo "RESULT: ISSUES FOUND — see details above"
fi

---

Static Validation Checks (Claude Code)

These checks can be run without launching CK3. Claude Code should run them as part of milestone review:

1. Localization completeness

For every trait, event, decision, modifier, and title defined in mod files, verify a matching localization key exists in localization/english/.

Pattern: trait_<name> → needs trait_<name>:0 and trait_<name>_desc:0
Pattern: <event_namespace>.<id>.t → event title
Pattern: <event_namespace>.<id>.desc → event description
Pattern: <decision_name> → needs <decision_name>:0 and <decision_name>_desc:0

2. Reference integrity

• Every has_trait = trait_X in triggers/effects → trait_X exists in common/traits/
• Every has_modifier = mod_X → mod_X exists in common/modifiers/
• Every trigger_event = { id = X } → event X exists in events/
• Every faith = faith:X → faith X exists in common/religion/
• Every culture = culture:X → culture X exists in common/culture/

3. Encoding audit

Every .txt and .yml file in lords_of_ash/ must be UTF-8 with BOM.

4. Comment syntax

CK3 uses # for comments. The @ symbol is for scripted variables only. No @# patterns should exist in any .txt file.

---

Release Checklist (M8 — Before Steam)

Before publishing to Steam Workshop:

• Remove or gate 99_debug_decisions.txt (hide behind a game rule that is not visible to players, or delete the file entirely)
• Remove debug_mode game rule from game setup screen (or rename to something hidden)
• Run full error.log clean pass (zero mod errors)
• Run text.log clean pass (zero missing localization)
• Verify lords-of-ash.mod has correct supported_version for current CK3 patch
• Test without any other mods enabled
• Test with Roads to Power and Royal Court DLC both enabled and disabled
• Verify non-DLC fallbacks work (M5)
• Full Narrative Mode playthrough: bookmark → endgame
• Full Sandbox Mode playthrough: 30 years as a noble house
• Performance: game speed acceptable at 5-speed for 50 years