settled-reach/docs/workshops/test-architecture/justine-round3.md

title, description, type, status, workshop, agent, round, created

title	description	type	status	workshop	agent	round	created
Justine — Round 3: Final Tooling Specifications	Justine's implementation-ready specifications for content validation, perf baselines, and golden file diffing	workshop	archived	test-architecture	justine	3	2026-02-17

Justine — Round 3: Final Tooling Specifications

Workshop: QA Strategy & Test Architecture Track: 5 (Content Scaling & CI Pipeline) Round: 3 (Prioritization — build-ready specs) Date: 2026-02-17

These specifications are implementation-ready. A developer should be able to build each target by reading this document alone.

---

1. make pre-pr — Final Specification

Reconciliation

Hoshe proposed 6 steps; I proposed 5 in Round 2. The difference: Hoshe lists check-fact-ids as a separate step; I proposed absorbing it into validate-content. Since absorption hasn't been implemented yet, keep them separate for now. When Phase 2 of the content validator ships, check-fact-ids becomes redundant and the step count drops to 5.

Tyre argued fixture staleness should be BLOCKER, not WARNING. Accepted. Stale fixtures mean client tests run against outdated protocol data — every passing client test becomes a false positive. BLOCKER is correct.

Makefile Additions

# --- Pre-PR verification ---
# Run before pushing any PR. Chains all checks in dependency order.
# Fails fast on first error. Total: ~90-180s on incremental build.

.PHONY: pre-pr pre-pr-server pre-pr-client pre-pr-content fixtures-check

pre-pr: lint build test validate-content check-fact-ids fixtures-check
	@echo ""
	@echo "=== PRE-PR: ALL CHECKS PASSED ==="
	@echo "Safe to create PR."

# Branch-specific variants
pre-pr-server: lint-server build-server test-server fixtures-check
	@echo "Server pre-PR checks PASSED."

pre-pr-client: lint-client build-client test-client
	@echo "Client pre-PR checks PASSED."

pre-pr-content: validate-content check-fact-ids
	@echo "Content pre-PR checks PASSED."

# Fixture staleness check — BLOCKER (Tyre R2: stale fixtures = false positive client tests)
fixtures-check: fixtures
	@if git diff --quiet client/tests/fixtures/; then \
		echo "Fixtures: up to date"; \
	else \
		echo ""; \
		echo "FIXTURES STALE — protocol changed but fixtures not committed."; \
		echo "The following fixture files differ from the committed version:"; \
		git diff --stat client/tests/fixtures/; \
		echo ""; \
		echo "To fix: stage and commit the updated fixtures:"; \
		echo "  git add client/tests/fixtures/"; \
		echo "  git commit -m \"chore(fixtures): regenerate for protocol changes\""; \
		exit 1; \
	fi

Execution Chain

make pre-pr
 │
 ├── 1. lint          lint-server (clippy + fmt) + lint-client (GDScript check)
 │                    Duration: ~15s
 │                    Catches: clippy warnings, fmt violations, GDScript errors
 │                    Failure: exits immediately, no point building broken code
 │
 ├── 2. build         build-server (cargo build) + build-client (godot --headless --quit)
 │                    Duration: ~30-90s (incremental), ~5-7min (clean)
 │                    Catches: compilation errors both sides
 │
 ├── 3. test          test-server (cargo nextest) + test-client (gdUnit4 headless)
 │                    Duration: ~15-30s
 │                    Catches: unit + integration test failures
 │
 ├── 4. validate-content    YAML schema validation (existing Python script)
 │                    Duration: ~2-5s
 │                    Catches: malformed YAML, schema violations
 │                    Future: cross-reference validation (Phase 2)
 │
 ├── 5. check-fact-ids      Fact ID resolution against knowledge catalogs
 │                    Duration: ~2s
 │                    Catches: dangling fact_id references
 │                    Future: absorbed into validate-content Phase 2
 │
 └── 6. fixtures-check      Regenerate fixtures + git diff --exit-code
                      Duration: ~10-15s
                      Catches: stale protocol fixtures (BLOCKER)
                      Requires: server build (already done in step 2)

Failure Output Examples

Lint failure (step 1):

cd server && cargo clippy -- -D warnings
error: unused variable `x`
  --> src/simulation/movement.rs:42:9
make: *** [lint-server] Error 1

Fixture staleness (step 6):

FIXTURES STALE — protocol changed but fixtures not committed.
The following fixture files differ from the committed version:
 client/tests/fixtures/msgpack/snapshot_one_npc.msgpack | Bin 45 -> 52 bytes
 client/tests/fixtures/msgpack/snapshot_v2_full.msgpack  | Bin 89 -> 96 bytes

To fix: stage and commit the updated fixtures:
  git add client/tests/fixtures/
  git commit -m "chore(fixtures): regenerate for protocol changes"
make: *** [fixtures-check] Error 1

Success:

--- lint: done ---
--- build: done ---
--- test: done ---
Validated 47 files, 3 skipped, 0 errors
check-fact-ids: OK — 42 references validated against 42 canonical facts
Fixtures: up to date

=== PRE-PR: ALL CHECKS PASSED ===
Safe to create PR.

Duration Budget

Step	Incremental	Clean Cache	Notes
lint	~15s	~15s	No build dependency
build	~30s	~5-7min	Rust incremental build is fast
test	~15-30s	~15-30s	Tests compile quickly once build exists
validate-content	~2-5s	~2-5s	Python, no build dependency
check-fact-ids	~2s	~2s	Bash grep, no build dependency
fixtures-check	~10-15s	~10-15s	Runs gen_fixtures (server build already cached)
Total	~90s	~8min	Fast enough for every PR

What pre-pr Does NOT Include

• Layer 3 subprocess tests (slow, nightly-tier — make test-layer3)
• Performance benchmarks (machine-dependent — make perf-baseline)
• Gauntlet golden file tests (depends on Gauntlet implementation — make golden-diff)
• Content RON conversion (make content-ron — deferred per R2-OQ-03, not yet needed for validation)

For deeper verification:

# Optional: run everything including slow tests
pre-pr-deep: pre-pr test-layer3 perf-baseline golden-diff
	@echo "=== DEEP VERIFICATION: ALL CHECKS PASSED ==="

---

2. make perf-baseline — Final Specification

Dependency

Blocked on Gauntlet implementation. The perf benchmark needs build_gauntlet_app(seed) to construct the test world. Until the Gauntlet ships, the shadowcast benchmark (shadowcast_bench.rs) is the only available benchmark. The tooling is designed to accommodate both.

Baseline File: tests/perf/baseline.json

Checked into the repo. Updated explicitly by the developer.

{
  "_meta": {
    "format_version": 1,
    "updated_at": "2026-02-17T14:30:00Z",
    "updated_by": "developer-name",
    "commit": "abc123f",
    "machine": "workstation-01",
    "rust_version": "1.82.0",
    "build_profile": "release"
  },
  "benchmarks": {
    "gauntlet_100_ticks": {
      "median_ms": 142.3,
      "min_ms": 138.1,
      "max_ms": 156.7,
      "runs": 5,
      "description": "Gauntlet map, seed 42, 100 simulation ticks"
    },
    "shadowcast_150x150_30pct": {
      "median_ms": 12.4,
      "min_ms": 11.8,
      "max_ms": 14.2,
      "runs": 5,
      "description": "Shadowcast, 150x150 map, 30% walls, 1000 iterations"
    }
  }
}

Rust Benchmark Test: server/tests/gauntlet_perf.rs

//! Performance benchmarks for the Gauntlet.
//! Run with: cargo test --release --test gauntlet_perf -- --ignored --nocapture
//! Outputs structured PERF_RESULT lines for tooling/perf-compare to parse.

use std::time::Instant;

#[test]
#[ignore]
fn gauntlet_100_ticks_5_runs() {
    let mut times_ms: Vec<f64> = Vec::new();

    for run in 0..5 {
        let mut app = build_gauntlet_app(42); // seed 42, deterministic

        let start = Instant::now();
        for _ in 0..100 {
            app.update();
        }
        let elapsed = start.elapsed().as_secs_f64() * 1000.0;
        times_ms.push(elapsed);
        eprintln!("PERF_RUN: gauntlet_100_ticks run={} ms={:.2}", run, elapsed);
    }

    times_ms.sort_by(|a, b| a.partial_cmp(b).unwrap());
    let median = times_ms[2]; // middle of 5
    let min = times_ms[0];
    let max = times_ms[4];

    eprintln!("PERF_RESULT: gauntlet_100_ticks median={:.2} min={:.2} max={:.2} runs=5",
        median, min, max);
}

Output format (machine-parseable):

PERF_RUN: gauntlet_100_ticks run=0 ms=141.23
PERF_RUN: gauntlet_100_ticks run=1 ms=138.12
PERF_RUN: gauntlet_100_ticks run=2 ms=142.87
PERF_RUN: gauntlet_100_ticks run=3 ms=156.71
PERF_RUN: gauntlet_100_ticks run=4 ms=139.44
PERF_RESULT: gauntlet_100_ticks median=141.23 min=138.12 max=156.71 runs=5

Comparison Script: tooling/perf-compare

#!/usr/bin/env bash
# Compare performance results against committed baseline.
# Usage: tooling/perf-compare <results_file> [baseline_file]
#   results_file: raw output from cargo test (contains PERF_RESULT lines)
#   baseline_file: defaults to tests/perf/baseline.json
#
# Exit codes: 0 = all pass, 1 = warning(s), 2 = failure(s)
set -euo pipefail

REPO_ROOT="$(git rev-parse --show-toplevel)"
RESULTS="${1:?Usage: perf-compare <results_file> [baseline_file]}"
BASELINE="${2:-$REPO_ROOT/tests/perf/baseline.json}"

WARN_THRESHOLD=15   # % regression = warning
FAIL_THRESHOLD=30   # % regression = failure

if [ ! -f "$BASELINE" ]; then
    echo "No baseline at $BASELINE — run 'make perf-baseline-update' to create."
    exit 0
fi

# Parse PERF_RESULT lines from test output
# Format: PERF_RESULT: <name> median=<float> min=<float> max=<float> runs=<int>
EXIT_CODE=0

echo ""
echo "=== Performance Comparison ==="
echo "Baseline: $(python3 -c "import json; d=json.load(open('$BASELINE')); print(d['_meta']['commit'], '('+d['_meta']['updated_at'][:10]+')', 'on', d['_meta']['machine'])")"
echo ""
printf "%-30s %12s %12s %10s %8s\n" "Benchmark" "Baseline" "Current" "Delta" "Status"
printf "%s\n" "$(printf '%.0s─' {1..76})"

while IFS= read -r line; do
    NAME=$(echo "$line" | sed 's/.*PERF_RESULT: //' | awk '{print $1}')
    CURRENT=$(echo "$line" | grep -oP 'median=\K[0-9.]+')

    # Look up baseline
    BASELINE_VAL=$(python3 -c "
import json, sys
d = json.load(open('$BASELINE'))
b = d.get('benchmarks', {}).get('$NAME', {})
print(b.get('median_ms', 'N/A'))
" 2>/dev/null)

    if [ "$BASELINE_VAL" = "N/A" ]; then
        printf "%-30s %12s %10.1fms %10s %8s\n" "$NAME" "N/A" "$CURRENT" "—" "NEW"
        continue
    fi

    DELTA=$(python3 -c "print(f'{(($CURRENT - $BASELINE_VAL) / $BASELINE_VAL) * 100:.1f}')")
    DELTA_ABS=$(python3 -c "print(abs(($CURRENT - $BASELINE_VAL) / $BASELINE_VAL) * 100)")

    if python3 -c "exit(0 if $CURRENT < $BASELINE_VAL else 1)" 2>/dev/null; then
        STATUS="PASS"
    elif python3 -c "exit(0 if $DELTA_ABS < $WARN_THRESHOLD else 1)" 2>/dev/null; then
        STATUS="PASS"
    elif python3 -c "exit(0 if $DELTA_ABS < $FAIL_THRESHOLD else 1)" 2>/dev/null; then
        STATUS="WARN"
        [ "$EXIT_CODE" -lt 1 ] && EXIT_CODE=1
    else
        STATUS="FAIL"
        EXIT_CODE=2
    fi

    printf "%-30s %10.1fms %10.1fms %+9.1f%% %8s\n" \
        "$NAME" "$BASELINE_VAL" "$CURRENT" "$DELTA" "$STATUS"

done < <(grep "^PERF_RESULT:" "$RESULTS")

echo ""
case $EXIT_CODE in
    0) echo "Overall: PASS" ;;
    1) echo "Overall: WARNING — investigate regressions above 15%" ;;
    2) echo "Overall: FAIL — regression(s) exceed 30% threshold" ;;
esac

exit $EXIT_CODE

Makefile Targets

# --- Performance benchmarks (local, release builds only) ---

.PHONY: perf-baseline perf-baseline-update

perf-baseline:
	@echo "Building server (release)..."
	@cd server && cargo build --release 2>&1 | tail -1
	@echo "Running benchmarks (5 runs each)..."
	@cd server && cargo test --release --test gauntlet_perf -- --ignored --nocapture \
		2>&1 | tee ../.cache/perf_raw.txt
	@tooling/perf-compare .cache/perf_raw.txt

perf-baseline-update:
	@echo "Building server (release)..."
	@cd server && cargo build --release 2>&1 | tail -1
	@echo "Running benchmarks (5 runs each)..."
	@cd server && cargo test --release --test gauntlet_perf -- --ignored --nocapture \
		2>&1 | tee ../.cache/perf_raw.txt
	@tooling/perf-update .cache/perf_raw.txt tests/perf/baseline.json
	@echo ""
	@echo "Baseline updated. Review and commit tests/perf/baseline.json"
	@git diff --stat tests/perf/baseline.json

Baseline Update Script: tooling/perf-update

#!/usr/bin/env bash
# Update the performance baseline file from benchmark results.
# Usage: tooling/perf-update <results_file> <baseline_file>
set -euo pipefail

RESULTS="${1:?Usage: perf-update <results_file> <baseline_file>}"
BASELINE="${2:?Usage: perf-update <results_file> <baseline_file>}"

python3 -c "
import json, sys, os, subprocess, datetime

results_file = '$RESULTS'
baseline_file = '$BASELINE'

# Parse PERF_RESULT lines
benchmarks = {}
with open(results_file) as f:
    for line in f:
        if 'PERF_RESULT:' not in line:
            continue
        parts = line.strip().split()
        name = parts[1]
        vals = {}
        for p in parts[2:]:
            k, v = p.split('=')
            vals[k] = float(v) if '.' in v else int(v)
        benchmarks[name] = {
            'median_ms': vals['median'],
            'min_ms': vals['min'],
            'max_ms': vals['max'],
            'runs': vals['runs'],
        }

# Get metadata
commit = subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD']).decode().strip()
rust_ver = subprocess.check_output(['rustc', '--version']).decode().strip().split()[1]
machine = os.uname().nodename

baseline = {
    '_meta': {
        'format_version': 1,
        'updated_at': datetime.datetime.now(datetime.timezone.utc).isoformat(),
        'updated_by': os.environ.get('USER', 'unknown'),
        'commit': commit,
        'machine': machine,
        'rust_version': rust_ver,
        'build_profile': 'release',
    },
    'benchmarks': benchmarks,
}

with open(baseline_file, 'w') as f:
    json.dump(baseline, f, indent=2)
    f.write('\n')

print(f'Wrote {len(benchmarks)} benchmark(s) to {baseline_file}')
"

Human-Readable Output

Normal run:

=== Performance Comparison ===
Baseline: abc123f (2026-02-15) on workstation-01

Benchmark                        Baseline      Current      Delta   Status
────────────────────────────────────────────────────────────────────────────
gauntlet_100_ticks                 142.3ms      148.7ms     +4.5%     PASS
shadowcast_150x150_30pct            12.4ms       12.1ms     -2.4%     PASS

Overall: PASS

Regression detected:

gauntlet_100_ticks                 142.3ms      203.1ms    +42.7%     FAIL
  Exceeds 30% threshold. Likely regression.

Overall: FAIL — regression(s) exceed 30% threshold

Thresholds

Delta	Status	Action
<15% or faster	PASS	Normal variance
15-30% slower	WARN	Investigate. May be noise or real regression
>30% slower	FAIL	Almost certainly a regression. Profile before merging

Machine Variance Protection

• Machine tag in baseline: Baseline records which machine it was set on. If the machine differs, the comparison script prints a warning: "WARNING: baseline was measured on workstation-01, current machine is laptop-02. Results may not be comparable."
• Release builds only: Debug builds have 10-20x performance variance. The Makefile targets use --release.
• Median of 5: Absorbs outliers from background processes, thermal throttling.

---

3. Golden File Workflow — Final Specification

Overview

Golden files are canonical ObserverSnapshot outputs from the Gauntlet at fixed ticks with a fixed seed. They're checked into the repo as sorted JSON. When server behavior changes, the golden files change — and the diff shows exactly what changed.

Dependency

Blocked on Gauntlet implementation. Requires build_gauntlet_app(seed) from Dudley/Tyre and serde_json derives on ObserverSnapshot.

File Layout

tests/
  golden/
    gauntlet_tick_0.json       # ObserverSnapshot at tick 0, seed 42
    gauntlet_tick_10.json      # ObserverSnapshot at tick 10, seed 42
    gauntlet_tick_100.json     # ObserverSnapshot at tick 100, seed 42

JSON Format

Sorted keys, pretty-printed, deterministic. Generated with serde_json::to_string_pretty() and sorted keys (#[serde(sort_maps)] or post-processing):

{
  "current_monologue": null,
  "entities": [
    {
      "entity_id": 1,
      "kind": "Player",
      "observation": "Visible",
      "relationship": "Unknown",
      "visibility": "Forward",
      "x": 16.5,
      "y": 16.5,
      "z": 0
    }
  ],
  "game_time": {
    "day": 0,
    "day_phase": "Morning",
    "tick_rate": "Full",
    "time_of_day": 0
  },
  "nearby_interactions": [],
  "pending_recognitions": [],
  "player_facing": "North",
  "player_inventory": [],
  "player_stance": "Walk",
  "tick": 0,
  "version": 7,
  "visible_tiles": []
}

Why JSON, not MessagePack:

• Human-readable in git diff — PR reviewers see exactly what changed
• Sorted keys = deterministic output regardless of Rust struct field order
• serde_json is already a dev-dependency in the server crate
• Standard format — no custom tooling for basic inspection

Golden File Generator: server/tests/gauntlet_golden_gen.rs

//! Regenerate Gauntlet golden files.
//! Run with: cargo test --test gauntlet_golden_gen -- --ignored --nocapture

use settled_reach_server::bridge::types::ObserverSnapshot;
use std::fs;
use std::path::Path;

const GOLDEN_DIR: &str = "../tests/golden";
const SEED: u64 = 42;
const TICKS: &[u64] = &[0, 10, 100];

fn write_golden(name: &str, snapshot: &ObserverSnapshot) {
    let dir = Path::new(GOLDEN_DIR);
    fs::create_dir_all(dir).expect("create golden dir");
    let path = dir.join(format!("{}.json", name));
    let json = serde_json::to_string_pretty(snapshot).expect("serialize to JSON");
    fs::write(&path, &json).expect("write golden file");
    eprintln!("Wrote {} ({} bytes)", path.display(), json.len());
}

#[test]
#[ignore]
fn regenerate_golden_files() {
    for &tick in TICKS {
        let mut app = build_gauntlet_app(SEED);
        for _ in 0..tick {
            app.update();
        }
        let snapshot = extract_observer_snapshot(&app);
        write_golden(&format!("gauntlet_tick_{}", tick), &snapshot);
    }
}

Golden File Comparator: server/tests/gauntlet_golden.rs

This is the test that runs during make golden-diff. It loads the checked-in golden file, runs the Gauntlet fresh, and compares field-by-field.

//! Compare current Gauntlet output against committed golden files.
//! Run with: cargo test --test gauntlet_golden -- --nocapture
//! On mismatch: prints structured diff and fails.

use settled_reach_server::bridge::types::*;
use std::path::Path;

const GOLDEN_DIR: &str = "../tests/golden";
const SEED: u64 = 42;

fn load_golden(name: &str) -> ObserverSnapshot {
    let path = Path::new(GOLDEN_DIR).join(format!("{}.json", name));
    let content = std::fs::read_to_string(&path)
        .unwrap_or_else(|_| panic!("Golden file not found: {}. Run 'make golden-update'.", path.display()));
    serde_json::from_str(&content)
        .unwrap_or_else(|e| panic!("Failed to parse {}: {}", path.display(), e))
}

fn diff_snapshots(golden: &ObserverSnapshot, actual: &ObserverSnapshot) -> Vec<String> {
    let mut diffs = Vec::new();

    // Scalar fields
    if golden.version != actual.version {
        diffs.push(format!("  version: {} -> {}", golden.version, actual.version));
    }
    if golden.tick != actual.tick {
        diffs.push(format!("  tick: {} -> {}", golden.tick, actual.tick));
    }
    if golden.player_facing != actual.player_facing {
        diffs.push(format!("  player_facing: {:?} -> {:?}", golden.player_facing, actual.player_facing));
    }
    if golden.player_stance != actual.player_stance {
        diffs.push(format!("  player_stance: {:?} -> {:?}", golden.player_stance, actual.player_stance));
    }

    // Game time
    if golden.game_time != actual.game_time {
        diffs.push(format!("  game_time: {:?} -> {:?}", golden.game_time, actual.game_time));
    }

    // Entity count
    if golden.entities.len() != actual.entities.len() {
        diffs.push(format!("  entities.count: {} -> {}", golden.entities.len(), actual.entities.len()));
    }

    // Per-entity comparison (both sorted by entity_id per determinism fixes)
    let max_len = golden.entities.len().max(actual.entities.len());
    for i in 0..max_len {
        match (golden.entities.get(i), actual.entities.get(i)) {
            (Some(g), Some(a)) => {
                let label = format!("entities[{}](id:{})", i, g.entity_id);
                if g.entity_id != a.entity_id {
                    diffs.push(format!("  {}.entity_id: {} -> {}", label, g.entity_id, a.entity_id));
                }
                if (g.x - a.x).abs() > 0.01 || (g.y - a.y).abs() > 0.01 {
                    diffs.push(format!("  {}: ({},{}) -> ({},{})  <- POSITION", label, g.x, g.y, a.x, a.y));
                }
                if g.z != a.z {
                    diffs.push(format!("  {}.z: {} -> {}", label, g.z, a.z));
                }
                if g.kind != a.kind {
                    diffs.push(format!("  {}.kind: {:?} -> {:?}", label, g.kind, a.kind));
                }
                if g.visibility != a.visibility {
                    diffs.push(format!("  {}.visibility: {:?} -> {:?}  <- VISIBILITY", label, g.visibility, a.visibility));
                }
                if g.relationship != a.relationship {
                    diffs.push(format!("  {}.relationship: {:?} -> {:?}", label, g.relationship, a.relationship));
                }
                if g.observation != a.observation {
                    diffs.push(format!("  {}.observation: {:?} -> {:?}", label, g.observation, a.observation));
                }
            }
            (Some(g), None) => {
                diffs.push(format!("  REMOVED: entities[{}] {{ id:{}, kind:{:?}, pos:({},{}) }}", i, g.entity_id, g.kind, g.x, g.y));
            }
            (None, Some(a)) => {
                diffs.push(format!("  ADDED: entities[{}] {{ id:{}, kind:{:?}, pos:({},{}) }}", i, a.entity_id, a.kind, a.x, a.y));
            }
            (None, None) => {}
        }
    }

    // Visible tiles (summary only — per-tile diff is too noisy)
    if golden.visible_tiles.len() != actual.visible_tiles.len() {
        diffs.push(format!("  visible_tiles.count: {} -> {}", golden.visible_tiles.len(), actual.visible_tiles.len()));
    }

    // Interactions
    if golden.nearby_interactions.len() != actual.nearby_interactions.len() {
        diffs.push(format!("  nearby_interactions.count: {} -> {}", golden.nearby_interactions.len(), actual.nearby_interactions.len()));
    }

    // Inventory
    if golden.player_inventory != actual.player_inventory {
        diffs.push(format!("  player_inventory: {:?} -> {:?}", golden.player_inventory, actual.player_inventory));
    }

    // Monologue
    if golden.current_monologue != actual.current_monologue {
        diffs.push(format!("  current_monologue: {:?} -> {:?}", golden.current_monologue, actual.current_monologue));
    }

    diffs
}

#[test]
fn gauntlet_tick_0_matches_golden() {
    compare_golden("gauntlet_tick_0", 0);
}

#[test]
fn gauntlet_tick_10_matches_golden() {
    compare_golden("gauntlet_tick_10", 10);
}

#[test]
fn gauntlet_tick_100_matches_golden() {
    compare_golden("gauntlet_tick_100", 100);
}

fn compare_golden(name: &str, ticks: u64) {
    let golden = load_golden(name);
    let mut app = build_gauntlet_app(SEED);
    for _ in 0..ticks {
        app.update();
    }
    let actual = extract_observer_snapshot(&app);

    let diffs = diff_snapshots(&golden, &actual);
    if !diffs.is_empty() {
        eprintln!();
        eprintln!("=== Golden File Mismatch: {} ===", name);
        eprintln!("Seed: {}, Ticks: {}", SEED, ticks);
        eprintln!();
        eprintln!("CHANGED FIELDS ({}):", diffs.len());
        for d in &diffs {
            eprintln!("{}", d);
        }
        eprintln!();
        eprintln!("To update golden files: make golden-update");
        eprintln!("Then review: git diff tests/golden/");
        panic!("{} field(s) differ from golden file", diffs.len());
    }
}

Makefile Targets

# --- Golden file operations ---

.PHONY: golden-diff golden-update

# Compare current Gauntlet output against committed golden files.
# Prints structured diff on mismatch.
golden-diff:
	cd server && cargo test --test gauntlet_golden -- --nocapture

# Regenerate golden files from current server behavior.
# Review the diff before committing.
golden-update:
	cd server && cargo test --test gauntlet_golden_gen -- --ignored --nocapture
	@echo ""
	@echo "Golden files regenerated. Review changes:"
	@git diff --stat tests/golden/
	@echo ""
	@echo "If changes are expected, commit:"
	@echo "  git add tests/golden/"
	@echo "  git commit -m \"chore(golden): update for <reason>\""

Developer Workflow

1. Developer changes server logic
2. make golden-diff — shows structured diff if anything changed
3. Developer reviews the diff: "Yes, I moved NPC guard-1 by one tile, this is expected"
4. make golden-update — regenerates golden files
5. git diff tests/golden/ — final review of JSON changes
6. Commit the updated golden files alongside the code change

---

4. CI Pipeline Design — Deferred but Documented

When the lead greenlights CI, this is the ready-to-implement specification. Based on my Round 1 proposal, Hoshe's 3-tier model, and Tyre's adjustments (15min PR budget, fixture staleness BLOCKER, content scaling in nightly).

Workflow File: .gitea/workflows/ci.yaml

name: CI
on:
  push:
    branches: ['*']
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 3 * * *'  # Nightly at 03:00 UTC

jobs:
  # ──────────────────────────────────────────────
  # TIER 1: Commit checks (every push, ~2 min)
  # ──────────────────────────────────────────────
  commit-checks:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4
      - name: Lint server
        run: make lint-server
      - name: Lint client
        run: make lint-client
      - name: Validate content (schema)
        run: make validate-content
      - name: Check fact IDs
        run: make check-fact-ids

  # ──────────────────────────────────────────────
  # TIER 2: PR checks (merge gate, <15 min)
  # ──────────────────────────────────────────────
  server-build-test:
    if: github.event_name == 'pull_request'
    needs: commit-checks
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4
      - name: Cache cargo
        uses: actions/cache@v4
        with:
          path: |
            ~/.cargo/registry
            ~/.cargo/git
            server/target
          key: cargo-${{ hashFiles('server/Cargo.lock') }}
      - name: Build server
        run: make build-server
      - name: Test server
        run: make test-server
      - name: Generate fixtures
        run: make fixtures
      - name: Check fixture staleness (BLOCKER)
        run: |
          if ! git diff --quiet client/tests/fixtures/; then
            echo "::error::Fixtures are stale. Run 'make fixtures' and commit."
            git diff --stat client/tests/fixtures/
            exit 1
          fi
      - name: Upload fixtures
        uses: actions/upload-artifact@v4
        with:
          name: msgpack-fixtures
          path: client/tests/fixtures/msgpack/

  client-build-test:
    if: github.event_name == 'pull_request'
    needs: [commit-checks, server-build-test]
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4
      - name: Download fixtures
        uses: actions/download-artifact@v4
        with:
          name: msgpack-fixtures
          path: client/tests/fixtures/msgpack/
      - name: Build client
        run: make build-client
      - name: Test client
        run: make test-client

  # ──────────────────────────────────────────────
  # TIER 3: Nightly (deep validation, <30 min)
  # ──────────────────────────────────────────────
  nightly:
    if: github.event_name == 'schedule'
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4
      - name: Full build
        run: make build
      - name: All tests
        run: make test
      - name: Layer 3 subprocess test
        run: cd server && cargo test --test layer3 -- --ignored --nocapture
      - name: Golden file check
        run: |
          make golden-diff || {
            echo "::warning::Golden file mismatch detected"
            cd server && cargo test --test gauntlet_golden -- --nocapture 2>&1 || true
          }
      - name: Performance benchmark
        run: |
          make perf-baseline 2>&1 | tee .cache/perf_output.txt
          # Annotate but don't fail
          grep "FAIL\|WARN" .cache/perf_output.txt && \
            echo "::warning::Performance regression detected" || true
      - name: Content scaling stress test
        run: cd server && cargo test --test content_scaling -- --ignored --nocapture

Runner Requirements

Dependency	How to Provide
Rust stable + clippy + rustfmt	Pre-installed on self-hosted runner via rustup
cargo-nextest	Pre-installed: cargo install cargo-nextest --locked
Godot 4.6 headless	Pre-installed to /usr/local/bin/godot4 on runner
Python 3 + jsonschema + pyyaml	Pre-installed: pip install jsonschema pyyaml

Self-hosted runner is required for:

• Pre-installed Godot (no download step)
• Stable performance baselines (no co-tenancy variance)
• Access to internal network (Gitea at git.schweitz.internal)

Merge-Blocking Policy (Gitea Branch Protection)

Job	Required for Merge?	Rationale
commit-checks	Yes	Lint + content validation are fast gates
server-build-test	Yes	Server tests + fixture staleness are correctness gates
client-build-test	Yes	Client tests verify the rendering contract
nightly	No	Deep tests are informational, not blocking

Gitea branch protection settings for main:

• Required status checks: commit-checks, server-build-test, client-build-test
• Require 1 approval
• Dismiss stale approvals on new pushes

Estimated Wiring Effort

~1 day. The Makefile targets already exist. The workflow file is the only new artifact. Self-hosted runner setup is separate infrastructure work (~0.5 day).

---

5. Fixture Staleness Check — Final Specification

Status: BLOCKER

Confirmed BLOCKER, not WARNING. Tyre's Round 2 argument is definitive: stale fixtures mean client tests run against outdated protocol data. Every passing client test is a false positive. This is exactly the class of bug (wire format mismatch, like Bug #4) that the entire serialization testing track exists to prevent.

How It Works

make fixtures-check
  │
  ├── 1. Run make fixtures
  │     → cd server && cargo test --test gen_fixtures -- --ignored
  │     → Writes .msgpack files to client/tests/fixtures/msgpack/
  │
  └── 2. Check for uncommitted changes
        → git diff --quiet client/tests/fixtures/
        → Exit 0: fixtures match committed versions (PASS)
        → Exit 1: fixtures differ from committed versions (FAIL)

Where It Runs

Context	How	Blocking?
make pre-pr	Step 6 of 6	Yes — pre-pr fails
make pre-pr-server	Final step	Yes — server changes affect fixtures
CI PR tier (future)	server-build-test job	Yes — merge blocked
CI nightly (future)	Not separately — covered by PR tier	N/A
make pre-pr-client	Not included	No — client doesn't generate fixtures
make pre-pr-content	Not included	No — content changes don't affect fixtures

Makefile Target (repeated from Section 1 for standalone reference)

fixtures-check: fixtures
	@if git diff --quiet client/tests/fixtures/; then \
		echo "Fixtures: up to date"; \
	else \
		echo ""; \
		echo "FIXTURES STALE — protocol changed but fixtures not committed."; \
		echo "The following fixture files differ from the committed version:"; \
		git diff --stat client/tests/fixtures/; \
		echo ""; \
		echo "To fix: stage and commit the updated fixtures:"; \
		echo "  git add client/tests/fixtures/"; \
		echo "  git commit -m \"chore(fixtures): regenerate for protocol changes\""; \
		exit 1; \
	fi

Edge Cases

1. New fixture files (untracked): git diff --quiet does NOT detect untracked files. If gen_fixtures.rs adds a new fixture, it won't be flagged by git diff. Mitigation: use git diff --quiet client/tests/fixtures/ && git ls-files --others --exclude-standard client/tests/fixtures/ | grep -q . && exit 1 || true. Or simpler: check for any untracked .msgpack files.

   Updated target:

   fixtures-check: fixtures
   	@STALE=0; \
   	if ! git diff --quiet client/tests/fixtures/; then \
   		STALE=1; \
   		echo "Modified fixtures:"; \
   		git diff --stat client/tests/fixtures/; \
   	fi; \
   	UNTRACKED=$$(git ls-files --others --exclude-standard client/tests/fixtures/); \
   	if [ -n "$$UNTRACKED" ]; then \
   		STALE=1; \
   		echo "New (untracked) fixtures:"; \
   		echo "$$UNTRACKED"; \
   	fi; \
   	if [ "$$STALE" -eq 1 ]; then \
   		echo ""; \
   		echo "FIXTURES STALE — run 'make fixtures' and commit the results."; \
   		exit 1; \
   	fi; \
   	echo "Fixtures: up to date"
   

2. Fixture path fragility: gen_fixtures.rs uses ../client/tests/fixtures/msgpack (relative to server/). This works in the monorepo checkout and CI checkout. If the path ever breaks, make fixtures itself will fail — which is caught before the diff check.

3. Determinism: gen_fixtures.rs uses fixed values (no timestamps, no random data). Verified in Round 2 — same source always produces same output.

Answering Hoshe's R2-OQ-04

Fixture staleness in make pre-pr — separate make pre-pr-full to keep basic pre-PR fast?

No. The fixture check adds ~10-15 seconds and requires only a server build (which make pre-pr already does in step 2). The incremental cost is negligible. Separating it into pre-pr-full means developers skip it — defeating the purpose. Keep it in the standard make pre-pr chain.

---

Summary

Spec	Status	Blocked On	Implementable Now?
make pre-pr	Final	Nothing	Yes
make perf-baseline	Final	Gauntlet (build_gauntlet_app)	Tooling: yes. Benchmark test: after Gauntlet
Golden file workflow	Final	Gauntlet + serde_json on ObserverSnapshot	Tooling: yes. Tests: after Gauntlet
CI pipeline	Final, deferred	Lead greenlight + self-hosted runner	Documented, ~1 day to wire
Fixture staleness	Final	Nothing	Yes

Immediate implementation order:

1. fixtures-check target (prerequisite for pre-pr)
2. make pre-pr target (immediate developer value)
3. tooling/perf-compare + tooling/perf-update scripts (ready for when benchmarks exist)
4. make golden-diff / make golden-update targets (ready for when Gauntlet ships)
5. CI workflow file (ready for when lead greenlights)