jpmschweitzerandClaude Opus 4.6
a87c95a6eb
3-round workshop with 7 agents (Tyre, Dudley, Stig, Hoshe, Justine, Gestalt, Ozzie) plus Qatux documenting. Produced: - 59-item prioritized test backlog (60 tickets under epic #455) - Gauntlet test world spec: 7 rooms + hub, 48 entities - Test client binary spec (tooling/test-client/) - Determinism fixes (3 patches, ~22 lines) - Server --test-mode + --port 0 design - Content cross-reference validation (9 checks) - make pre-pr pipeline (6-step) - 38 client tests prioritized - Anti-tedium features (reset plate, hub teleport, WRONG button) - Human tester walkthrough - CI pipeline design (deferred but documented) Sprint 8 scope: ~17.75 team-days across 26 tickets. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
997 lines
35 KiB
Markdown
997 lines
35 KiB
Markdown
# Justine — Round 3: Final Tooling Specifications
|
|
|
|
**Workshop:** QA Strategy & Test Architecture
|
|
**Track:** 5 (Content Scaling & CI Pipeline)
|
|
**Round:** 3 (Prioritization — build-ready specs)
|
|
**Date:** 2026-02-17
|
|
|
|
These specifications are implementation-ready. A developer should be able to build each target by reading this document alone.
|
|
|
|
---
|
|
|
|
## 1. `make pre-pr` — Final Specification
|
|
|
|
### Reconciliation
|
|
|
|
Hoshe proposed 6 steps; I proposed 5 in Round 2. The difference: Hoshe lists `check-fact-ids` as a separate step; I proposed absorbing it into `validate-content`. Since absorption hasn't been implemented yet, **keep them separate for now**. When Phase 2 of the content validator ships, `check-fact-ids` becomes redundant and the step count drops to 5.
|
|
|
|
Tyre argued fixture staleness should be BLOCKER, not WARNING. **Accepted.** Stale fixtures mean client tests run against outdated protocol data — every passing client test becomes a false positive. BLOCKER is correct.
|
|
|
|
### Makefile Additions
|
|
|
|
```makefile
|
|
# --- Pre-PR verification ---
|
|
# Run before pushing any PR. Chains all checks in dependency order.
|
|
# Fails fast on first error. Total: ~90-180s on incremental build.
|
|
|
|
.PHONY: pre-pr pre-pr-server pre-pr-client pre-pr-content fixtures-check
|
|
|
|
pre-pr: lint build test validate-content check-fact-ids fixtures-check
|
|
@echo ""
|
|
@echo "=== PRE-PR: ALL CHECKS PASSED ==="
|
|
@echo "Safe to create PR."
|
|
|
|
# Branch-specific variants
|
|
pre-pr-server: lint-server build-server test-server fixtures-check
|
|
@echo "Server pre-PR checks PASSED."
|
|
|
|
pre-pr-client: lint-client build-client test-client
|
|
@echo "Client pre-PR checks PASSED."
|
|
|
|
pre-pr-content: validate-content check-fact-ids
|
|
@echo "Content pre-PR checks PASSED."
|
|
|
|
# Fixture staleness check — BLOCKER (Tyre R2: stale fixtures = false positive client tests)
|
|
fixtures-check: fixtures
|
|
@if git diff --quiet client/tests/fixtures/; then \
|
|
echo "Fixtures: up to date"; \
|
|
else \
|
|
echo ""; \
|
|
echo "FIXTURES STALE — protocol changed but fixtures not committed."; \
|
|
echo "The following fixture files differ from the committed version:"; \
|
|
git diff --stat client/tests/fixtures/; \
|
|
echo ""; \
|
|
echo "To fix: stage and commit the updated fixtures:"; \
|
|
echo " git add client/tests/fixtures/"; \
|
|
echo " git commit -m \"chore(fixtures): regenerate for protocol changes\""; \
|
|
exit 1; \
|
|
fi
|
|
```
|
|
|
|
### Execution Chain
|
|
|
|
```
|
|
make pre-pr
|
|
│
|
|
├── 1. lint lint-server (clippy + fmt) + lint-client (GDScript check)
|
|
│ Duration: ~15s
|
|
│ Catches: clippy warnings, fmt violations, GDScript errors
|
|
│ Failure: exits immediately, no point building broken code
|
|
│
|
|
├── 2. build build-server (cargo build) + build-client (godot --headless --quit)
|
|
│ Duration: ~30-90s (incremental), ~5-7min (clean)
|
|
│ Catches: compilation errors both sides
|
|
│
|
|
├── 3. test test-server (cargo nextest) + test-client (gdUnit4 headless)
|
|
│ Duration: ~15-30s
|
|
│ Catches: unit + integration test failures
|
|
│
|
|
├── 4. validate-content YAML schema validation (existing Python script)
|
|
│ Duration: ~2-5s
|
|
│ Catches: malformed YAML, schema violations
|
|
│ Future: cross-reference validation (Phase 2)
|
|
│
|
|
├── 5. check-fact-ids Fact ID resolution against knowledge catalogs
|
|
│ Duration: ~2s
|
|
│ Catches: dangling fact_id references
|
|
│ Future: absorbed into validate-content Phase 2
|
|
│
|
|
└── 6. fixtures-check Regenerate fixtures + git diff --exit-code
|
|
Duration: ~10-15s
|
|
Catches: stale protocol fixtures (BLOCKER)
|
|
Requires: server build (already done in step 2)
|
|
```
|
|
|
|
### Failure Output Examples
|
|
|
|
**Lint failure (step 1):**
|
|
```
|
|
cd server && cargo clippy -- -D warnings
|
|
error: unused variable `x`
|
|
--> src/simulation/movement.rs:42:9
|
|
make: *** [lint-server] Error 1
|
|
```
|
|
|
|
**Fixture staleness (step 6):**
|
|
```
|
|
FIXTURES STALE — protocol changed but fixtures not committed.
|
|
The following fixture files differ from the committed version:
|
|
client/tests/fixtures/msgpack/snapshot_one_npc.msgpack | Bin 45 -> 52 bytes
|
|
client/tests/fixtures/msgpack/snapshot_v2_full.msgpack | Bin 89 -> 96 bytes
|
|
|
|
To fix: stage and commit the updated fixtures:
|
|
git add client/tests/fixtures/
|
|
git commit -m "chore(fixtures): regenerate for protocol changes"
|
|
make: *** [fixtures-check] Error 1
|
|
```
|
|
|
|
**Success:**
|
|
```
|
|
--- lint: done ---
|
|
--- build: done ---
|
|
--- test: done ---
|
|
Validated 47 files, 3 skipped, 0 errors
|
|
check-fact-ids: OK — 42 references validated against 42 canonical facts
|
|
Fixtures: up to date
|
|
|
|
=== PRE-PR: ALL CHECKS PASSED ===
|
|
Safe to create PR.
|
|
```
|
|
|
|
### Duration Budget
|
|
|
|
| Step | Incremental | Clean Cache | Notes |
|
|
|------|------------|-------------|-------|
|
|
| lint | ~15s | ~15s | No build dependency |
|
|
| build | ~30s | ~5-7min | Rust incremental build is fast |
|
|
| test | ~15-30s | ~15-30s | Tests compile quickly once build exists |
|
|
| validate-content | ~2-5s | ~2-5s | Python, no build dependency |
|
|
| check-fact-ids | ~2s | ~2s | Bash grep, no build dependency |
|
|
| fixtures-check | ~10-15s | ~10-15s | Runs gen_fixtures (server build already cached) |
|
|
| **Total** | **~90s** | **~8min** | Fast enough for every PR |
|
|
|
|
### What pre-pr Does NOT Include
|
|
|
|
- Layer 3 subprocess tests (slow, nightly-tier — `make test-layer3`)
|
|
- Performance benchmarks (machine-dependent — `make perf-baseline`)
|
|
- Gauntlet golden file tests (depends on Gauntlet implementation — `make golden-diff`)
|
|
- Content RON conversion (`make content-ron` — deferred per R2-OQ-03, not yet needed for validation)
|
|
|
|
For deeper verification:
|
|
```makefile
|
|
# Optional: run everything including slow tests
|
|
pre-pr-deep: pre-pr test-layer3 perf-baseline golden-diff
|
|
@echo "=== DEEP VERIFICATION: ALL CHECKS PASSED ==="
|
|
```
|
|
|
|
---
|
|
|
|
## 2. `make perf-baseline` — Final Specification
|
|
|
|
### Dependency
|
|
|
|
**Blocked on Gauntlet implementation.** The perf benchmark needs `build_gauntlet_app(seed)` to construct the test world. Until the Gauntlet ships, the shadowcast benchmark (`shadowcast_bench.rs`) is the only available benchmark. The tooling is designed to accommodate both.
|
|
|
|
### Baseline File: `tests/perf/baseline.json`
|
|
|
|
Checked into the repo. Updated explicitly by the developer.
|
|
|
|
```json
|
|
{
|
|
"_meta": {
|
|
"format_version": 1,
|
|
"updated_at": "2026-02-17T14:30:00Z",
|
|
"updated_by": "developer-name",
|
|
"commit": "abc123f",
|
|
"machine": "workstation-01",
|
|
"rust_version": "1.82.0",
|
|
"build_profile": "release"
|
|
},
|
|
"benchmarks": {
|
|
"gauntlet_100_ticks": {
|
|
"median_ms": 142.3,
|
|
"min_ms": 138.1,
|
|
"max_ms": 156.7,
|
|
"runs": 5,
|
|
"description": "Gauntlet map, seed 42, 100 simulation ticks"
|
|
},
|
|
"shadowcast_150x150_30pct": {
|
|
"median_ms": 12.4,
|
|
"min_ms": 11.8,
|
|
"max_ms": 14.2,
|
|
"runs": 5,
|
|
"description": "Shadowcast, 150x150 map, 30% walls, 1000 iterations"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Rust Benchmark Test: `server/tests/gauntlet_perf.rs`
|
|
|
|
```rust
|
|
//! Performance benchmarks for the Gauntlet.
|
|
//! Run with: cargo test --release --test gauntlet_perf -- --ignored --nocapture
|
|
//! Outputs structured PERF_RESULT lines for tooling/perf-compare to parse.
|
|
|
|
use std::time::Instant;
|
|
|
|
#[test]
|
|
#[ignore]
|
|
fn gauntlet_100_ticks_5_runs() {
|
|
let mut times_ms: Vec<f64> = Vec::new();
|
|
|
|
for run in 0..5 {
|
|
let mut app = build_gauntlet_app(42); // seed 42, deterministic
|
|
|
|
let start = Instant::now();
|
|
for _ in 0..100 {
|
|
app.update();
|
|
}
|
|
let elapsed = start.elapsed().as_secs_f64() * 1000.0;
|
|
times_ms.push(elapsed);
|
|
eprintln!("PERF_RUN: gauntlet_100_ticks run={} ms={:.2}", run, elapsed);
|
|
}
|
|
|
|
times_ms.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
|
let median = times_ms[2]; // middle of 5
|
|
let min = times_ms[0];
|
|
let max = times_ms[4];
|
|
|
|
eprintln!("PERF_RESULT: gauntlet_100_ticks median={:.2} min={:.2} max={:.2} runs=5",
|
|
median, min, max);
|
|
}
|
|
```
|
|
|
|
Output format (machine-parseable):
|
|
```
|
|
PERF_RUN: gauntlet_100_ticks run=0 ms=141.23
|
|
PERF_RUN: gauntlet_100_ticks run=1 ms=138.12
|
|
PERF_RUN: gauntlet_100_ticks run=2 ms=142.87
|
|
PERF_RUN: gauntlet_100_ticks run=3 ms=156.71
|
|
PERF_RUN: gauntlet_100_ticks run=4 ms=139.44
|
|
PERF_RESULT: gauntlet_100_ticks median=141.23 min=138.12 max=156.71 runs=5
|
|
```
|
|
|
|
### Comparison Script: `tooling/perf-compare`
|
|
|
|
```bash
|
|
#!/usr/bin/env bash
|
|
# Compare performance results against committed baseline.
|
|
# Usage: tooling/perf-compare <results_file> [baseline_file]
|
|
# results_file: raw output from cargo test (contains PERF_RESULT lines)
|
|
# baseline_file: defaults to tests/perf/baseline.json
|
|
#
|
|
# Exit codes: 0 = all pass, 1 = warning(s), 2 = failure(s)
|
|
set -euo pipefail
|
|
|
|
REPO_ROOT="$(git rev-parse --show-toplevel)"
|
|
RESULTS="${1:?Usage: perf-compare <results_file> [baseline_file]}"
|
|
BASELINE="${2:-$REPO_ROOT/tests/perf/baseline.json}"
|
|
|
|
WARN_THRESHOLD=15 # % regression = warning
|
|
FAIL_THRESHOLD=30 # % regression = failure
|
|
|
|
if [ ! -f "$BASELINE" ]; then
|
|
echo "No baseline at $BASELINE — run 'make perf-baseline-update' to create."
|
|
exit 0
|
|
fi
|
|
|
|
# Parse PERF_RESULT lines from test output
|
|
# Format: PERF_RESULT: <name> median=<float> min=<float> max=<float> runs=<int>
|
|
EXIT_CODE=0
|
|
|
|
echo ""
|
|
echo "=== Performance Comparison ==="
|
|
echo "Baseline: $(python3 -c "import json; d=json.load(open('$BASELINE')); print(d['_meta']['commit'], '('+d['_meta']['updated_at'][:10]+')', 'on', d['_meta']['machine'])")"
|
|
echo ""
|
|
printf "%-30s %12s %12s %10s %8s\n" "Benchmark" "Baseline" "Current" "Delta" "Status"
|
|
printf "%s\n" "$(printf '%.0s─' {1..76})"
|
|
|
|
while IFS= read -r line; do
|
|
NAME=$(echo "$line" | sed 's/.*PERF_RESULT: //' | awk '{print $1}')
|
|
CURRENT=$(echo "$line" | grep -oP 'median=\K[0-9.]+')
|
|
|
|
# Look up baseline
|
|
BASELINE_VAL=$(python3 -c "
|
|
import json, sys
|
|
d = json.load(open('$BASELINE'))
|
|
b = d.get('benchmarks', {}).get('$NAME', {})
|
|
print(b.get('median_ms', 'N/A'))
|
|
" 2>/dev/null)
|
|
|
|
if [ "$BASELINE_VAL" = "N/A" ]; then
|
|
printf "%-30s %12s %10.1fms %10s %8s\n" "$NAME" "N/A" "$CURRENT" "—" "NEW"
|
|
continue
|
|
fi
|
|
|
|
DELTA=$(python3 -c "print(f'{(($CURRENT - $BASELINE_VAL) / $BASELINE_VAL) * 100:.1f}')")
|
|
DELTA_ABS=$(python3 -c "print(abs(($CURRENT - $BASELINE_VAL) / $BASELINE_VAL) * 100)")
|
|
|
|
if python3 -c "exit(0 if $CURRENT < $BASELINE_VAL else 1)" 2>/dev/null; then
|
|
STATUS="PASS"
|
|
elif python3 -c "exit(0 if $DELTA_ABS < $WARN_THRESHOLD else 1)" 2>/dev/null; then
|
|
STATUS="PASS"
|
|
elif python3 -c "exit(0 if $DELTA_ABS < $FAIL_THRESHOLD else 1)" 2>/dev/null; then
|
|
STATUS="WARN"
|
|
[ "$EXIT_CODE" -lt 1 ] && EXIT_CODE=1
|
|
else
|
|
STATUS="FAIL"
|
|
EXIT_CODE=2
|
|
fi
|
|
|
|
printf "%-30s %10.1fms %10.1fms %+9.1f%% %8s\n" \
|
|
"$NAME" "$BASELINE_VAL" "$CURRENT" "$DELTA" "$STATUS"
|
|
|
|
done < <(grep "^PERF_RESULT:" "$RESULTS")
|
|
|
|
echo ""
|
|
case $EXIT_CODE in
|
|
0) echo "Overall: PASS" ;;
|
|
1) echo "Overall: WARNING — investigate regressions above 15%" ;;
|
|
2) echo "Overall: FAIL — regression(s) exceed 30% threshold" ;;
|
|
esac
|
|
|
|
exit $EXIT_CODE
|
|
```
|
|
|
|
### Makefile Targets
|
|
|
|
```makefile
|
|
# --- Performance benchmarks (local, release builds only) ---
|
|
|
|
.PHONY: perf-baseline perf-baseline-update
|
|
|
|
perf-baseline:
|
|
@echo "Building server (release)..."
|
|
@cd server && cargo build --release 2>&1 | tail -1
|
|
@echo "Running benchmarks (5 runs each)..."
|
|
@cd server && cargo test --release --test gauntlet_perf -- --ignored --nocapture \
|
|
2>&1 | tee ../.cache/perf_raw.txt
|
|
@tooling/perf-compare .cache/perf_raw.txt
|
|
|
|
perf-baseline-update:
|
|
@echo "Building server (release)..."
|
|
@cd server && cargo build --release 2>&1 | tail -1
|
|
@echo "Running benchmarks (5 runs each)..."
|
|
@cd server && cargo test --release --test gauntlet_perf -- --ignored --nocapture \
|
|
2>&1 | tee ../.cache/perf_raw.txt
|
|
@tooling/perf-update .cache/perf_raw.txt tests/perf/baseline.json
|
|
@echo ""
|
|
@echo "Baseline updated. Review and commit tests/perf/baseline.json"
|
|
@git diff --stat tests/perf/baseline.json
|
|
```
|
|
|
|
### Baseline Update Script: `tooling/perf-update`
|
|
|
|
```bash
|
|
#!/usr/bin/env bash
|
|
# Update the performance baseline file from benchmark results.
|
|
# Usage: tooling/perf-update <results_file> <baseline_file>
|
|
set -euo pipefail
|
|
|
|
RESULTS="${1:?Usage: perf-update <results_file> <baseline_file>}"
|
|
BASELINE="${2:?Usage: perf-update <results_file> <baseline_file>}"
|
|
|
|
python3 -c "
|
|
import json, sys, os, subprocess, datetime
|
|
|
|
results_file = '$RESULTS'
|
|
baseline_file = '$BASELINE'
|
|
|
|
# Parse PERF_RESULT lines
|
|
benchmarks = {}
|
|
with open(results_file) as f:
|
|
for line in f:
|
|
if 'PERF_RESULT:' not in line:
|
|
continue
|
|
parts = line.strip().split()
|
|
name = parts[1]
|
|
vals = {}
|
|
for p in parts[2:]:
|
|
k, v = p.split('=')
|
|
vals[k] = float(v) if '.' in v else int(v)
|
|
benchmarks[name] = {
|
|
'median_ms': vals['median'],
|
|
'min_ms': vals['min'],
|
|
'max_ms': vals['max'],
|
|
'runs': vals['runs'],
|
|
}
|
|
|
|
# Get metadata
|
|
commit = subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD']).decode().strip()
|
|
rust_ver = subprocess.check_output(['rustc', '--version']).decode().strip().split()[1]
|
|
machine = os.uname().nodename
|
|
|
|
baseline = {
|
|
'_meta': {
|
|
'format_version': 1,
|
|
'updated_at': datetime.datetime.now(datetime.timezone.utc).isoformat(),
|
|
'updated_by': os.environ.get('USER', 'unknown'),
|
|
'commit': commit,
|
|
'machine': machine,
|
|
'rust_version': rust_ver,
|
|
'build_profile': 'release',
|
|
},
|
|
'benchmarks': benchmarks,
|
|
}
|
|
|
|
with open(baseline_file, 'w') as f:
|
|
json.dump(baseline, f, indent=2)
|
|
f.write('\n')
|
|
|
|
print(f'Wrote {len(benchmarks)} benchmark(s) to {baseline_file}')
|
|
"
|
|
```
|
|
|
|
### Human-Readable Output
|
|
|
|
Normal run:
|
|
```
|
|
=== Performance Comparison ===
|
|
Baseline: abc123f (2026-02-15) on workstation-01
|
|
|
|
Benchmark Baseline Current Delta Status
|
|
────────────────────────────────────────────────────────────────────────────
|
|
gauntlet_100_ticks 142.3ms 148.7ms +4.5% PASS
|
|
shadowcast_150x150_30pct 12.4ms 12.1ms -2.4% PASS
|
|
|
|
Overall: PASS
|
|
```
|
|
|
|
Regression detected:
|
|
```
|
|
gauntlet_100_ticks 142.3ms 203.1ms +42.7% FAIL
|
|
Exceeds 30% threshold. Likely regression.
|
|
|
|
Overall: FAIL — regression(s) exceed 30% threshold
|
|
```
|
|
|
|
### Thresholds
|
|
|
|
| Delta | Status | Action |
|
|
|-------|--------|--------|
|
|
| <15% or faster | PASS | Normal variance |
|
|
| 15-30% slower | WARN | Investigate. May be noise or real regression |
|
|
| >30% slower | FAIL | Almost certainly a regression. Profile before merging |
|
|
|
|
### Machine Variance Protection
|
|
|
|
- **Machine tag in baseline**: Baseline records which machine it was set on. If the machine differs, the comparison script prints a warning: `"WARNING: baseline was measured on workstation-01, current machine is laptop-02. Results may not be comparable."`
|
|
- **Release builds only**: Debug builds have 10-20x performance variance. The Makefile targets use `--release`.
|
|
- **Median of 5**: Absorbs outliers from background processes, thermal throttling.
|
|
|
|
---
|
|
|
|
## 3. Golden File Workflow — Final Specification
|
|
|
|
### Overview
|
|
|
|
Golden files are canonical `ObserverSnapshot` outputs from the Gauntlet at fixed ticks with a fixed seed. They're checked into the repo as sorted JSON. When server behavior changes, the golden files change — and the diff shows exactly what changed.
|
|
|
|
### Dependency
|
|
|
|
**Blocked on Gauntlet implementation.** Requires `build_gauntlet_app(seed)` from Dudley/Tyre and `serde_json` derives on `ObserverSnapshot`.
|
|
|
|
### File Layout
|
|
|
|
```
|
|
tests/
|
|
golden/
|
|
gauntlet_tick_0.json # ObserverSnapshot at tick 0, seed 42
|
|
gauntlet_tick_10.json # ObserverSnapshot at tick 10, seed 42
|
|
gauntlet_tick_100.json # ObserverSnapshot at tick 100, seed 42
|
|
```
|
|
|
|
### JSON Format
|
|
|
|
Sorted keys, pretty-printed, deterministic. Generated with `serde_json::to_string_pretty()` and sorted keys (`#[serde(sort_maps)]` or post-processing):
|
|
|
|
```json
|
|
{
|
|
"current_monologue": null,
|
|
"entities": [
|
|
{
|
|
"entity_id": 1,
|
|
"kind": "Player",
|
|
"observation": "Visible",
|
|
"relationship": "Unknown",
|
|
"visibility": "Forward",
|
|
"x": 16.5,
|
|
"y": 16.5,
|
|
"z": 0
|
|
}
|
|
],
|
|
"game_time": {
|
|
"day": 0,
|
|
"day_phase": "Morning",
|
|
"tick_rate": "Full",
|
|
"time_of_day": 0
|
|
},
|
|
"nearby_interactions": [],
|
|
"pending_recognitions": [],
|
|
"player_facing": "North",
|
|
"player_inventory": [],
|
|
"player_stance": "Walk",
|
|
"tick": 0,
|
|
"version": 7,
|
|
"visible_tiles": []
|
|
}
|
|
```
|
|
|
|
**Why JSON, not MessagePack:**
|
|
- Human-readable in `git diff` — PR reviewers see exactly what changed
|
|
- Sorted keys = deterministic output regardless of Rust struct field order
|
|
- `serde_json` is already a dev-dependency in the server crate
|
|
- Standard format — no custom tooling for basic inspection
|
|
|
|
### Golden File Generator: `server/tests/gauntlet_golden_gen.rs`
|
|
|
|
```rust
|
|
//! Regenerate Gauntlet golden files.
|
|
//! Run with: cargo test --test gauntlet_golden_gen -- --ignored --nocapture
|
|
|
|
use settled_reach_server::bridge::types::ObserverSnapshot;
|
|
use std::fs;
|
|
use std::path::Path;
|
|
|
|
const GOLDEN_DIR: &str = "../tests/golden";
|
|
const SEED: u64 = 42;
|
|
const TICKS: &[u64] = &[0, 10, 100];
|
|
|
|
fn write_golden(name: &str, snapshot: &ObserverSnapshot) {
|
|
let dir = Path::new(GOLDEN_DIR);
|
|
fs::create_dir_all(dir).expect("create golden dir");
|
|
let path = dir.join(format!("{}.json", name));
|
|
let json = serde_json::to_string_pretty(snapshot).expect("serialize to JSON");
|
|
fs::write(&path, &json).expect("write golden file");
|
|
eprintln!("Wrote {} ({} bytes)", path.display(), json.len());
|
|
}
|
|
|
|
#[test]
|
|
#[ignore]
|
|
fn regenerate_golden_files() {
|
|
for &tick in TICKS {
|
|
let mut app = build_gauntlet_app(SEED);
|
|
for _ in 0..tick {
|
|
app.update();
|
|
}
|
|
let snapshot = extract_observer_snapshot(&app);
|
|
write_golden(&format!("gauntlet_tick_{}", tick), &snapshot);
|
|
}
|
|
}
|
|
```
|
|
|
|
### Golden File Comparator: `server/tests/gauntlet_golden.rs`
|
|
|
|
This is the test that runs during `make golden-diff`. It loads the checked-in golden file, runs the Gauntlet fresh, and compares field-by-field.
|
|
|
|
```rust
|
|
//! Compare current Gauntlet output against committed golden files.
|
|
//! Run with: cargo test --test gauntlet_golden -- --nocapture
|
|
//! On mismatch: prints structured diff and fails.
|
|
|
|
use settled_reach_server::bridge::types::*;
|
|
use std::path::Path;
|
|
|
|
const GOLDEN_DIR: &str = "../tests/golden";
|
|
const SEED: u64 = 42;
|
|
|
|
fn load_golden(name: &str) -> ObserverSnapshot {
|
|
let path = Path::new(GOLDEN_DIR).join(format!("{}.json", name));
|
|
let content = std::fs::read_to_string(&path)
|
|
.unwrap_or_else(|_| panic!("Golden file not found: {}. Run 'make golden-update'.", path.display()));
|
|
serde_json::from_str(&content)
|
|
.unwrap_or_else(|e| panic!("Failed to parse {}: {}", path.display(), e))
|
|
}
|
|
|
|
fn diff_snapshots(golden: &ObserverSnapshot, actual: &ObserverSnapshot) -> Vec<String> {
|
|
let mut diffs = Vec::new();
|
|
|
|
// Scalar fields
|
|
if golden.version != actual.version {
|
|
diffs.push(format!(" version: {} -> {}", golden.version, actual.version));
|
|
}
|
|
if golden.tick != actual.tick {
|
|
diffs.push(format!(" tick: {} -> {}", golden.tick, actual.tick));
|
|
}
|
|
if golden.player_facing != actual.player_facing {
|
|
diffs.push(format!(" player_facing: {:?} -> {:?}", golden.player_facing, actual.player_facing));
|
|
}
|
|
if golden.player_stance != actual.player_stance {
|
|
diffs.push(format!(" player_stance: {:?} -> {:?}", golden.player_stance, actual.player_stance));
|
|
}
|
|
|
|
// Game time
|
|
if golden.game_time != actual.game_time {
|
|
diffs.push(format!(" game_time: {:?} -> {:?}", golden.game_time, actual.game_time));
|
|
}
|
|
|
|
// Entity count
|
|
if golden.entities.len() != actual.entities.len() {
|
|
diffs.push(format!(" entities.count: {} -> {}", golden.entities.len(), actual.entities.len()));
|
|
}
|
|
|
|
// Per-entity comparison (both sorted by entity_id per determinism fixes)
|
|
let max_len = golden.entities.len().max(actual.entities.len());
|
|
for i in 0..max_len {
|
|
match (golden.entities.get(i), actual.entities.get(i)) {
|
|
(Some(g), Some(a)) => {
|
|
let label = format!("entities[{}](id:{})", i, g.entity_id);
|
|
if g.entity_id != a.entity_id {
|
|
diffs.push(format!(" {}.entity_id: {} -> {}", label, g.entity_id, a.entity_id));
|
|
}
|
|
if (g.x - a.x).abs() > 0.01 || (g.y - a.y).abs() > 0.01 {
|
|
diffs.push(format!(" {}: ({},{}) -> ({},{}) <- POSITION", label, g.x, g.y, a.x, a.y));
|
|
}
|
|
if g.z != a.z {
|
|
diffs.push(format!(" {}.z: {} -> {}", label, g.z, a.z));
|
|
}
|
|
if g.kind != a.kind {
|
|
diffs.push(format!(" {}.kind: {:?} -> {:?}", label, g.kind, a.kind));
|
|
}
|
|
if g.visibility != a.visibility {
|
|
diffs.push(format!(" {}.visibility: {:?} -> {:?} <- VISIBILITY", label, g.visibility, a.visibility));
|
|
}
|
|
if g.relationship != a.relationship {
|
|
diffs.push(format!(" {}.relationship: {:?} -> {:?}", label, g.relationship, a.relationship));
|
|
}
|
|
if g.observation != a.observation {
|
|
diffs.push(format!(" {}.observation: {:?} -> {:?}", label, g.observation, a.observation));
|
|
}
|
|
}
|
|
(Some(g), None) => {
|
|
diffs.push(format!(" REMOVED: entities[{}] {{ id:{}, kind:{:?}, pos:({},{}) }}", i, g.entity_id, g.kind, g.x, g.y));
|
|
}
|
|
(None, Some(a)) => {
|
|
diffs.push(format!(" ADDED: entities[{}] {{ id:{}, kind:{:?}, pos:({},{}) }}", i, a.entity_id, a.kind, a.x, a.y));
|
|
}
|
|
(None, None) => {}
|
|
}
|
|
}
|
|
|
|
// Visible tiles (summary only — per-tile diff is too noisy)
|
|
if golden.visible_tiles.len() != actual.visible_tiles.len() {
|
|
diffs.push(format!(" visible_tiles.count: {} -> {}", golden.visible_tiles.len(), actual.visible_tiles.len()));
|
|
}
|
|
|
|
// Interactions
|
|
if golden.nearby_interactions.len() != actual.nearby_interactions.len() {
|
|
diffs.push(format!(" nearby_interactions.count: {} -> {}", golden.nearby_interactions.len(), actual.nearby_interactions.len()));
|
|
}
|
|
|
|
// Inventory
|
|
if golden.player_inventory != actual.player_inventory {
|
|
diffs.push(format!(" player_inventory: {:?} -> {:?}", golden.player_inventory, actual.player_inventory));
|
|
}
|
|
|
|
// Monologue
|
|
if golden.current_monologue != actual.current_monologue {
|
|
diffs.push(format!(" current_monologue: {:?} -> {:?}", golden.current_monologue, actual.current_monologue));
|
|
}
|
|
|
|
diffs
|
|
}
|
|
|
|
#[test]
|
|
fn gauntlet_tick_0_matches_golden() {
|
|
compare_golden("gauntlet_tick_0", 0);
|
|
}
|
|
|
|
#[test]
|
|
fn gauntlet_tick_10_matches_golden() {
|
|
compare_golden("gauntlet_tick_10", 10);
|
|
}
|
|
|
|
#[test]
|
|
fn gauntlet_tick_100_matches_golden() {
|
|
compare_golden("gauntlet_tick_100", 100);
|
|
}
|
|
|
|
fn compare_golden(name: &str, ticks: u64) {
|
|
let golden = load_golden(name);
|
|
let mut app = build_gauntlet_app(SEED);
|
|
for _ in 0..ticks {
|
|
app.update();
|
|
}
|
|
let actual = extract_observer_snapshot(&app);
|
|
|
|
let diffs = diff_snapshots(&golden, &actual);
|
|
if !diffs.is_empty() {
|
|
eprintln!();
|
|
eprintln!("=== Golden File Mismatch: {} ===", name);
|
|
eprintln!("Seed: {}, Ticks: {}", SEED, ticks);
|
|
eprintln!();
|
|
eprintln!("CHANGED FIELDS ({}):", diffs.len());
|
|
for d in &diffs {
|
|
eprintln!("{}", d);
|
|
}
|
|
eprintln!();
|
|
eprintln!("To update golden files: make golden-update");
|
|
eprintln!("Then review: git diff tests/golden/");
|
|
panic!("{} field(s) differ from golden file", diffs.len());
|
|
}
|
|
}
|
|
```
|
|
|
|
### Makefile Targets
|
|
|
|
```makefile
|
|
# --- Golden file operations ---
|
|
|
|
.PHONY: golden-diff golden-update
|
|
|
|
# Compare current Gauntlet output against committed golden files.
|
|
# Prints structured diff on mismatch.
|
|
golden-diff:
|
|
cd server && cargo test --test gauntlet_golden -- --nocapture
|
|
|
|
# Regenerate golden files from current server behavior.
|
|
# Review the diff before committing.
|
|
golden-update:
|
|
cd server && cargo test --test gauntlet_golden_gen -- --ignored --nocapture
|
|
@echo ""
|
|
@echo "Golden files regenerated. Review changes:"
|
|
@git diff --stat tests/golden/
|
|
@echo ""
|
|
@echo "If changes are expected, commit:"
|
|
@echo " git add tests/golden/"
|
|
@echo " git commit -m \"chore(golden): update for <reason>\""
|
|
```
|
|
|
|
### Developer Workflow
|
|
|
|
1. Developer changes server logic
|
|
2. `make golden-diff` — shows structured diff if anything changed
|
|
3. Developer reviews the diff: "Yes, I moved NPC guard-1 by one tile, this is expected"
|
|
4. `make golden-update` — regenerates golden files
|
|
5. `git diff tests/golden/` — final review of JSON changes
|
|
6. Commit the updated golden files alongside the code change
|
|
|
|
---
|
|
|
|
## 4. CI Pipeline Design — Deferred but Documented
|
|
|
|
When the lead greenlights CI, this is the ready-to-implement specification. Based on my Round 1 proposal, Hoshe's 3-tier model, and Tyre's adjustments (15min PR budget, fixture staleness BLOCKER, content scaling in nightly).
|
|
|
|
### Workflow File: `.gitea/workflows/ci.yaml`
|
|
|
|
```yaml
|
|
name: CI
|
|
on:
|
|
push:
|
|
branches: ['*']
|
|
pull_request:
|
|
branches: [main]
|
|
schedule:
|
|
- cron: '0 3 * * *' # Nightly at 03:00 UTC
|
|
|
|
jobs:
|
|
# ──────────────────────────────────────────────
|
|
# TIER 1: Commit checks (every push, ~2 min)
|
|
# ──────────────────────────────────────────────
|
|
commit-checks:
|
|
runs-on: self-hosted
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- name: Lint server
|
|
run: make lint-server
|
|
- name: Lint client
|
|
run: make lint-client
|
|
- name: Validate content (schema)
|
|
run: make validate-content
|
|
- name: Check fact IDs
|
|
run: make check-fact-ids
|
|
|
|
# ──────────────────────────────────────────────
|
|
# TIER 2: PR checks (merge gate, <15 min)
|
|
# ──────────────────────────────────────────────
|
|
server-build-test:
|
|
if: github.event_name == 'pull_request'
|
|
needs: commit-checks
|
|
runs-on: self-hosted
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- name: Cache cargo
|
|
uses: actions/cache@v4
|
|
with:
|
|
path: |
|
|
~/.cargo/registry
|
|
~/.cargo/git
|
|
server/target
|
|
key: cargo-${{ hashFiles('server/Cargo.lock') }}
|
|
- name: Build server
|
|
run: make build-server
|
|
- name: Test server
|
|
run: make test-server
|
|
- name: Generate fixtures
|
|
run: make fixtures
|
|
- name: Check fixture staleness (BLOCKER)
|
|
run: |
|
|
if ! git diff --quiet client/tests/fixtures/; then
|
|
echo "::error::Fixtures are stale. Run 'make fixtures' and commit."
|
|
git diff --stat client/tests/fixtures/
|
|
exit 1
|
|
fi
|
|
- name: Upload fixtures
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: msgpack-fixtures
|
|
path: client/tests/fixtures/msgpack/
|
|
|
|
client-build-test:
|
|
if: github.event_name == 'pull_request'
|
|
needs: [commit-checks, server-build-test]
|
|
runs-on: self-hosted
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- name: Download fixtures
|
|
uses: actions/download-artifact@v4
|
|
with:
|
|
name: msgpack-fixtures
|
|
path: client/tests/fixtures/msgpack/
|
|
- name: Build client
|
|
run: make build-client
|
|
- name: Test client
|
|
run: make test-client
|
|
|
|
# ──────────────────────────────────────────────
|
|
# TIER 3: Nightly (deep validation, <30 min)
|
|
# ──────────────────────────────────────────────
|
|
nightly:
|
|
if: github.event_name == 'schedule'
|
|
runs-on: self-hosted
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- name: Full build
|
|
run: make build
|
|
- name: All tests
|
|
run: make test
|
|
- name: Layer 3 subprocess test
|
|
run: cd server && cargo test --test layer3 -- --ignored --nocapture
|
|
- name: Golden file check
|
|
run: |
|
|
make golden-diff || {
|
|
echo "::warning::Golden file mismatch detected"
|
|
cd server && cargo test --test gauntlet_golden -- --nocapture 2>&1 || true
|
|
}
|
|
- name: Performance benchmark
|
|
run: |
|
|
make perf-baseline 2>&1 | tee .cache/perf_output.txt
|
|
# Annotate but don't fail
|
|
grep "FAIL\|WARN" .cache/perf_output.txt && \
|
|
echo "::warning::Performance regression detected" || true
|
|
- name: Content scaling stress test
|
|
run: cd server && cargo test --test content_scaling -- --ignored --nocapture
|
|
```
|
|
|
|
### Runner Requirements
|
|
|
|
| Dependency | How to Provide |
|
|
|-----------|---------------|
|
|
| Rust stable + clippy + rustfmt | Pre-installed on self-hosted runner via `rustup` |
|
|
| cargo-nextest | Pre-installed: `cargo install cargo-nextest --locked` |
|
|
| Godot 4.6 headless | Pre-installed to `/usr/local/bin/godot4` on runner |
|
|
| Python 3 + jsonschema + pyyaml | Pre-installed: `pip install jsonschema pyyaml` |
|
|
|
|
**Self-hosted runner is required** for:
|
|
- Pre-installed Godot (no download step)
|
|
- Stable performance baselines (no co-tenancy variance)
|
|
- Access to internal network (Gitea at `git.schweitz.internal`)
|
|
|
|
### Merge-Blocking Policy (Gitea Branch Protection)
|
|
|
|
| Job | Required for Merge? | Rationale |
|
|
|-----|-------------------|-----------|
|
|
| `commit-checks` | **Yes** | Lint + content validation are fast gates |
|
|
| `server-build-test` | **Yes** | Server tests + fixture staleness are correctness gates |
|
|
| `client-build-test` | **Yes** | Client tests verify the rendering contract |
|
|
| `nightly` | **No** | Deep tests are informational, not blocking |
|
|
|
|
Gitea branch protection settings for `main`:
|
|
- Required status checks: `commit-checks`, `server-build-test`, `client-build-test`
|
|
- Require 1 approval
|
|
- Dismiss stale approvals on new pushes
|
|
|
|
### Estimated Wiring Effort
|
|
|
|
~1 day. The Makefile targets already exist. The workflow file is the only new artifact. Self-hosted runner setup is separate infrastructure work (~0.5 day).
|
|
|
|
---
|
|
|
|
## 5. Fixture Staleness Check — Final Specification
|
|
|
|
### Status: BLOCKER
|
|
|
|
Confirmed BLOCKER, not WARNING. Tyre's Round 2 argument is definitive: stale fixtures mean client tests run against outdated protocol data. Every passing client test is a false positive. This is exactly the class of bug (wire format mismatch, like Bug #4) that the entire serialization testing track exists to prevent.
|
|
|
|
### How It Works
|
|
|
|
```
|
|
make fixtures-check
|
|
│
|
|
├── 1. Run make fixtures
|
|
│ → cd server && cargo test --test gen_fixtures -- --ignored
|
|
│ → Writes .msgpack files to client/tests/fixtures/msgpack/
|
|
│
|
|
└── 2. Check for uncommitted changes
|
|
→ git diff --quiet client/tests/fixtures/
|
|
→ Exit 0: fixtures match committed versions (PASS)
|
|
→ Exit 1: fixtures differ from committed versions (FAIL)
|
|
```
|
|
|
|
### Where It Runs
|
|
|
|
| Context | How | Blocking? |
|
|
|---------|-----|-----------|
|
|
| `make pre-pr` | Step 6 of 6 | Yes — pre-pr fails |
|
|
| `make pre-pr-server` | Final step | Yes — server changes affect fixtures |
|
|
| CI PR tier (future) | `server-build-test` job | Yes — merge blocked |
|
|
| CI nightly (future) | Not separately — covered by PR tier | N/A |
|
|
| `make pre-pr-client` | **Not included** | No — client doesn't generate fixtures |
|
|
| `make pre-pr-content` | **Not included** | No — content changes don't affect fixtures |
|
|
|
|
### Makefile Target (repeated from Section 1 for standalone reference)
|
|
|
|
```makefile
|
|
fixtures-check: fixtures
|
|
@if git diff --quiet client/tests/fixtures/; then \
|
|
echo "Fixtures: up to date"; \
|
|
else \
|
|
echo ""; \
|
|
echo "FIXTURES STALE — protocol changed but fixtures not committed."; \
|
|
echo "The following fixture files differ from the committed version:"; \
|
|
git diff --stat client/tests/fixtures/; \
|
|
echo ""; \
|
|
echo "To fix: stage and commit the updated fixtures:"; \
|
|
echo " git add client/tests/fixtures/"; \
|
|
echo " git commit -m \"chore(fixtures): regenerate for protocol changes\""; \
|
|
exit 1; \
|
|
fi
|
|
```
|
|
|
|
### Edge Cases
|
|
|
|
1. **New fixture files (untracked):** `git diff --quiet` does NOT detect untracked files. If `gen_fixtures.rs` adds a new fixture, it won't be flagged by `git diff`. Mitigation: use `git diff --quiet client/tests/fixtures/ && git ls-files --others --exclude-standard client/tests/fixtures/ | grep -q . && exit 1 || true`. Or simpler: check for any untracked `.msgpack` files.
|
|
|
|
Updated target:
|
|
```makefile
|
|
fixtures-check: fixtures
|
|
@STALE=0; \
|
|
if ! git diff --quiet client/tests/fixtures/; then \
|
|
STALE=1; \
|
|
echo "Modified fixtures:"; \
|
|
git diff --stat client/tests/fixtures/; \
|
|
fi; \
|
|
UNTRACKED=$$(git ls-files --others --exclude-standard client/tests/fixtures/); \
|
|
if [ -n "$$UNTRACKED" ]; then \
|
|
STALE=1; \
|
|
echo "New (untracked) fixtures:"; \
|
|
echo "$$UNTRACKED"; \
|
|
fi; \
|
|
if [ "$$STALE" -eq 1 ]; then \
|
|
echo ""; \
|
|
echo "FIXTURES STALE — run 'make fixtures' and commit the results."; \
|
|
exit 1; \
|
|
fi; \
|
|
echo "Fixtures: up to date"
|
|
```
|
|
|
|
2. **Fixture path fragility:** `gen_fixtures.rs` uses `../client/tests/fixtures/msgpack` (relative to `server/`). This works in the monorepo checkout and CI checkout. If the path ever breaks, `make fixtures` itself will fail — which is caught before the diff check.
|
|
|
|
3. **Determinism:** `gen_fixtures.rs` uses fixed values (no timestamps, no random data). Verified in Round 2 — same source always produces same output.
|
|
|
|
### Answering Hoshe's R2-OQ-04
|
|
|
|
> Fixture staleness in `make pre-pr` — separate `make pre-pr-full` to keep basic pre-PR fast?
|
|
|
|
**No.** The fixture check adds ~10-15 seconds and requires only a server build (which `make pre-pr` already does in step 2). The incremental cost is negligible. Separating it into `pre-pr-full` means developers skip it — defeating the purpose. Keep it in the standard `make pre-pr` chain.
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
| Spec | Status | Blocked On | Implementable Now? |
|
|
|------|--------|-----------|-------------------|
|
|
| `make pre-pr` | Final | Nothing | **Yes** |
|
|
| `make perf-baseline` | Final | Gauntlet (`build_gauntlet_app`) | Tooling: yes. Benchmark test: after Gauntlet |
|
|
| Golden file workflow | Final | Gauntlet + `serde_json` on `ObserverSnapshot` | Tooling: yes. Tests: after Gauntlet |
|
|
| CI pipeline | Final, deferred | Lead greenlight + self-hosted runner | Documented, ~1 day to wire |
|
|
| Fixture staleness | Final | Nothing | **Yes** |
|
|
|
|
**Immediate implementation order:**
|
|
1. `fixtures-check` target (prerequisite for pre-pr)
|
|
2. `make pre-pr` target (immediate developer value)
|
|
3. `tooling/perf-compare` + `tooling/perf-update` scripts (ready for when benchmarks exist)
|
|
4. `make golden-diff` / `make golden-update` targets (ready for when Gauntlet ships)
|
|
5. CI workflow file (ready for when lead greenlights)
|