jpmschweitzer/settled-reach
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
11b29baefed11262a9e786d930388b39b80098aa
settled-reach/docs/workshops/test-architecture/hoshe-round1.md
T
jpmschweitzerandClaude Opus 4.6 563a295a90 docs(docs): add frontmatter to test-architecture workshop 
Standardized YAML frontmatter on all 20 files.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 00:01:23 +01:00

35 KiB
Raw Blame History

title, description, type, status, workshop, agent, round, created
title description type status workshop agent round created
Hoshe — Round 1 Analysis: Boundary Values, Layer 3, CI, Content Validation Hoshe's round 1 analysis of serialization boundary values, Layer 3 testing, and content validation workshop archived test-architecture hoshe 1 2026-02-17

Hoshe — Round 1 Analysis: Boundary Values, Layer 3, CI, Content Validation

Workshop: QA Strategy & Test Architecture Tracks: 4 (Serialization & Integration) + 5 (Content Scaling & CI) Date: 2026-02-17 Spec references: D-030 (testability architecture), D-020 (IPC/MessagePack protocol)

TRACK 4 — Serialization & Integration

T4-H1: MessagePack Boundary Value Test Matrix

The GDScript encoder (messagepack.gd:69-95) uses cascading if/elif branches to select the MessagePack integer format. Each branch boundary is a potential off-by-one bug site — exactly where bug #4 lived.

I traced every branch transition against the MessagePack spec. The encoder's branch order determines the effective ranges:

Branch Condition (GDScript) Effective range (after earlier branches steal values) MsgPack format Header byte
1 -32 <= v <= 127 -32 to 127 positive fixint / negative fixint 0x00-0x7f / 0xe0-0xff
2 -128 <= v < 128 -128 to -33 int 8 0xd0
3 0 <= v <= 255 128 to 255 uint 8 0xcc
4 -32768 <= v < 32768 -32768 to -129 AND 256 to 32767 int 16 0xd1
5 0 <= v <= 65535 32768 to 65535 uint 16 0xcd
6 -2^31 <= v < 2^31 -2147483648 to -32769 AND 65536 to 2147483647 int 32 0xd2
7 0 <= v <= 2^32-1 2147483648 to 4294967295 uint 32 0xce
8 -2^63 <= v < 2^63 -9223372036854775808 to -2147483649 AND 4294967296 to 9223372036854775807 int 64 0xd3
9 else Godot int is 64-bit signed, so this branch is unreachable for valid Godot ints uint 64 0xcf

Note on branch 4: Positive values 256-32767 are encoded as int_16, not uint_16. This is spec-valid (MessagePack allows any format that fits the value) but differs from what rmp_serde produces for unsigned Rust types. Cross-language roundtrip tests MUST cover this divergence — rmp_serde must accept int_16-encoded positive values when deserializing into u64.

Complete Boundary Value Matrix

Every row is a test case. Values are chosen at format transition boundaries (value-1, value, value+1).

Test ID Value Expected format Header byte Payload bytes Why this value matters
Positive fixint boundaries
BV-P01 0 pos fixint 0x00 (none) Zero — minimum positive fixint
BV-P02 1 pos fixint 0x01 (none) Smallest nonzero positive
BV-P03 126 pos fixint 0x7e (none) One below boundary
BV-P04 127 pos fixint 0x7f (none) MAX positive fixint — Bug #4 was here
fixint → uint_8 transition
BV-P05 128 uint 8 0xcc 0x80 MIN uint_8 — Bug #4: this was encoded as -128
BV-P06 129 uint 8 0xcc 0x81 One above boundary
BV-P07 254 uint 8 0xcc 0xfe One below max uint_8
BV-P08 255 uint 8 0xcc 0xff MAX uint_8
uint_8 → int_16 transition (NOTE: encoder uses int_16, not uint_16, for 256-32767)
BV-P09 256 int 16 0xd1 0x01 0x00 MIN int_16 positive — format widens to 2 bytes
BV-P10 257 int 16 0xd1 0x01 0x01 One above boundary
BV-P11 32766 int 16 0xd1 0x7f 0xfe One below max int_16
BV-P12 32767 int 16 0xd1 0x7f 0xff MAX int_16 positive
int_16 → uint_16 transition
BV-P13 32768 uint 16 0xcd 0x80 0x00 MIN uint_16 — exceeds int_16 max
BV-P14 32769 uint 16 0xcd 0x80 0x01 One above boundary
BV-P15 65534 uint 16 0xcd 0xff 0xfe One below max uint_16
BV-P16 65535 uint 16 0xcd 0xff 0xff MAX uint_16
uint_16 → int_32 transition
BV-P17 65536 int 32 0xd2 0x00 0x01 0x00 0x00 MIN int_32 positive
BV-P18 65537 int 32 0xd2 0x00 0x01 0x00 0x01 One above boundary
BV-P19 2147483646 int 32 0xd2 0x7f 0xff 0xff 0xfe One below max int_32
BV-P20 2147483647 int 32 0xd2 0x7f 0xff 0xff 0xff MAX int_32 (2^31-1)
int_32 → uint_32 transition
BV-P21 2147483648 uint 32 0xce 0x80 0x00 0x00 0x00 MIN uint_32 (2^31)
BV-P22 4294967294 uint 32 0xce 0xff 0xff 0xff 0xfe One below max uint_32
BV-P23 4294967295 uint 32 0xce 0xff 0xff 0xff 0xff MAX uint_32 (2^32-1)
uint_32 → int_64 transition
BV-P24 4294967296 int 64 0xd3 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x00 MIN int_64 positive (2^32)
BV-P25 9223372036854775807 int 64 0xd3 0x7f 0xff 0xff 0xff 0xff 0xff 0xff 0xff MAX int_64 (2^63-1, Godot max int)
Negative fixint boundaries
BV-N01 -1 neg fixint 0xff (none) Most common negative value
BV-N02 -31 neg fixint 0xe1 (none) One above min fixint
BV-N03 -32 neg fixint 0xe0 (none) MIN negative fixint
fixint → int_8 transition
BV-N04 -33 int 8 0xd0 0xdf Exceeds fixint — first value requiring int_8
BV-N05 -34 int 8 0xd0 0xde One below boundary
BV-N06 -127 int 8 0xd0 0x81 One above min int_8
BV-N07 -128 int 8 0xd0 0x80 MIN int_8
int_8 → int_16 transition
BV-N08 -129 int 16 0xd1 0xff 0x7f Exceeds int_8 — first value requiring int_16
BV-N09 -130 int 16 0xd1 0xff 0x7e One below boundary
BV-N10 -32767 int 16 0xd1 0x80 0x01 One above min int_16
BV-N11 -32768 int 16 0xd1 0x80 0x00 MIN int_16
int_16 → int_32 transition
BV-N12 -32769 int 32 0xd2 0xff 0xff 0x7f 0xff Exceeds int_16
BV-N13 -2147483647 int 32 0xd2 0x80 0x00 0x00 0x01 One above min int_32
BV-N14 -2147483648 int 32 0xd2 0x80 0x00 0x00 0x00 MIN int_32 (-2^31)
int_32 → int_64 transition
BV-N15 -2147483649 int 64 0xd3 0xff 0xff 0xff 0xff 0x7f 0xff 0xff 0xff Exceeds int_32 (-2^31-1)
BV-N16 -9223372036854775808 int 64 0xd3 0x80 0x00 0x00 0x00 0x00 0x00 0x00 0x00 MIN int_64 (-2^63, Godot min int)

Total: 41 boundary values.

Game-relevant values (sanity check)

These specific values appear in the protocol and must be tested even though they may not sit on format boundaries:

Value Protocol field Expected format
0 tick at game start pos fixint
10 tick at 1 second pos fixint
42 tick in test fixtures pos fixint
128 tick at ~13 seconds — BUG #4 SITE uint 8
500 tick in v2_full fixture int 16 (via branch 4)
999 tick in multi_entity fixture int 16
100 entity_id in test fixtures pos fixint
720 time_of_day (noon) int 16
1440 time_of_day (end of day) int 16

Observation: Asymmetric encoding between GDScript and Rust

The GDScript encoder picks int_16 for values 256-32767 (positive values in the signed range). The Rust encoder (rmp_serde) picks uint_16 for the same values. Both are valid per the MessagePack spec, but this means GDScript-encoded bytes and Rust-encoded bytes for the same logical value will differ at the byte level.

This is NOT a bug, but it means:

  • Byte-for-byte golden file comparison between GDScript-encoded and Rust-encoded data will FAIL for values 256-32767 (and similarly for 65536-2^31-1).
  • Golden files must be direction-specific: "Rust encodes this snapshot" (canonical) vs "GDScript encodes this input" (canonical).
  • Decoders on both sides MUST accept both signed and unsigned encodings for the same value. Tests should verify this explicitly.

T4-H2: Where Should Boundary Tests Live?

Recommendation: Both, but with different scopes.

Layer Location What it tests Speed Frequency
Encode-only (GDScript) client/tests/test_msgpack_boundaries.gd GDScript encoder produces correct format for each boundary value Fast (~100ms) Every commit
Encode-only (Rust) server/tests/serialization.rs (extend) Rust encoder produces correct format for each boundary value Fast (~50ms) Every commit
Decode cross-language Fixtures: Rust-generated .msgpack files decoded by GDScript GDScript decoder handles Rust-encoded boundary values Medium (~2s) Every PR
Decode cross-language Fixtures: GDScript-generated .msgpack files decoded by Rust Rust decoder handles GDScript-encoded boundary values Medium (~2s) Every PR
Full roundtrip bridge_tcp.rs extension: send boundary-valued snapshots over TCP, receive on GDScript side End-to-end wire format agreement Slow (~5s) Nightly / pre-merge

Rationale: Bug #4 was an encode-side bug (GDScript encoder chose wrong format). A GDScript-only encode test would have caught it. But the WORST bugs are decode mismatches — where one side encodes a value that the other side decodes as a different value. Cross-language fixtures catch those. The roundtrip catches protocol-level issues (framing, batching).

Concrete test functions:

GDScript (test_msgpack_boundaries.gd):

func test_boundary_positive_fixint_max() -> void:
    # BV-P04: 127 must encode as single byte 0x7f (positive fixint)
    var result = Messagepack.encode(127)
    assert_that(result.value.size()).is_equal(1)
    assert_that(result.value[0]).is_equal(0x7f)

func test_boundary_uint8_min() -> void:
    # BV-P05: 128 must encode as uint_8 (0xcc, 0x80) — Bug #4 regression guard
    var result = Messagepack.encode(128)
    assert_that(result.value.size()).is_equal(2)
    assert_that(result.value[0]).is_equal(0xcc)
    assert_that(result.value[1]).is_equal(0x80)

func test_boundary_roundtrip_all() -> void:
    # Every boundary value must survive encode→decode roundtrip
    var boundaries = [0, 1, 126, 127, 128, 129, 254, 255, 256, 257,
                      32766, 32767, 32768, 32769, 65534, 65535, 65536,
                      -1, -31, -32, -33, -34, -127, -128, -129, -130,
                      -32767, -32768, -32769]
    for v in boundaries:
        var encoded = Messagepack.encode(v)
        assert_that(encoded.status).is_null()
        var decoded = Messagepack.decode(encoded.value)
        assert_that(decoded.status).is_null()
        assert_that(decoded.value).is_equal(v)

Rust (serialization.rs addition):

#[test]
fn boundary_values_roundtrip() {
    let boundaries: Vec<i64> = vec![
        0, 1, 126, 127, 128, 129, 254, 255, 256, 257,
        32766, 32767, 32768, 32769, 65534, 65535, 65536, 65537,
        2147483646, 2147483647, 2147483648, 4294967294, 4294967295, 4294967296,
        -1, -31, -32, -33, -34, -127, -128, -129, -130,
        -32767, -32768, -32769,
        -2147483647, -2147483648, -2147483649,
    ];
    for val in boundaries {
        let bytes = rmp_serde::to_vec(&val).unwrap();
        let decoded: i64 = rmp_serde::from_slice(&bytes).unwrap();
        assert_eq!(decoded, val, "boundary value {} did not roundtrip", val);
    }
}

T4-H3: Should gen_fixtures.rs Generate Boundary Value Fixtures?

Yes. Concrete proposal:

Add a generate_boundary_fixtures() function to gen_fixtures.rs that generates two types of fixtures:

  1. Raw integer boundary fixtures — one .msgpack file per boundary value, containing just the encoded integer. Filename encodes the expected value: boundary_int_127.msgpack, boundary_int_128.msgpack, boundary_int_neg128.msgpack, etc.

  2. Snapshot boundary fixtures — snapshots with tick values at critical boundaries. These test the field-level encoding within a struct context:

    • snapshot_tick_127.msgpack — tick at max positive fixint
    • snapshot_tick_128.msgpack — tick at min uint_8 (Bug #4 regression)
    • snapshot_tick_32768.msgpack — tick at min uint_16
    • snapshot_tick_65536.msgpack — tick at min int_32
    • snapshot_entity_id_boundary.msgpack — entity_id values at boundaries

Implementation sketch for gen_fixtures.rs:

#[test]
#[ignore]
fn generate_boundary_fixtures() {
    // Raw integer boundary values
    let boundaries: Vec<(i64, &str)> = vec![
        (127, "boundary_int_127"),
        (128, "boundary_int_128"),
        (255, "boundary_int_255"),
        (256, "boundary_int_256"),
        (32767, "boundary_int_32767"),
        (32768, "boundary_int_32768"),
        (65535, "boundary_int_65535"),
        (65536, "boundary_int_65536"),
        (-32, "boundary_int_neg32"),
        (-33, "boundary_int_neg33"),
        (-128, "boundary_int_neg128"),
        (-129, "boundary_int_neg129"),
        (-32768, "boundary_int_neg32768"),
        (-32769, "boundary_int_neg32769"),
    ];

    for (value, name) in &boundaries {
        write_fixture(name, &rmp_serde::to_vec(value).unwrap());
    }

    // Snapshot with tick at the Bug #4 boundary
    let snapshot_128 = fixture_snapshot(128, vec![]);
    write_fixture("snapshot_tick_128", &rmp_serde::to_vec_named(&snapshot_128).unwrap());

    // Snapshot with large entity_id
    let snapshot_large_id = fixture_snapshot(0, vec![VisibleEntity {
        entity_id: 65536,  // uint_32 boundary
        x: 0.0, y: 0.0, z: 0,
        kind: EntityKind::Npc,
        visibility: VisibilitySector::Forward,
        relationship: RelationshipState::Unknown,
        observation: EntityVisibility::Visible,
    }]);
    write_fixture("snapshot_entity_id_65536", &rmp_serde::to_vec_named(&snapshot_large_id).unwrap());
}

Client-side verification (test_msgpack_boundaries.gd):

func test_boundary_fixture_int_128() -> void:
    # Bug #4 regression: Rust encodes tick=128 as uint_8, GDScript must decode to 128 (not -128)
    var bytes = _load_fixture("boundary_int_128")
    var result = Messagepack.decode(bytes)
    assert_that(result.value).is_equal(128)

func test_boundary_fixture_snapshot_tick_128() -> void:
    var bytes = _load_fixture("snapshot_tick_128")
    var snapshot = Protocol.decode_snapshot(bytes)
    assert_that(snapshot.tick).is_equal(128)

Workflow: make fixtures regenerates all fixtures (existing + boundary). Client tests verify them. This is D-030 Layer 1: fixture-based cross-language validation.

T4-H4: Golden File Approach Tied to the Gauntlet

Proposal: Gauntlet Snapshot Golden File Pipeline

The golden file flow:

[1. Server generates]          [2. Committed to repo]        [3. Client verifies]

cargo test --test               client/tests/fixtures/        make test-client
  gen_gauntlet_golden           gauntlet/                       ↓
  -- --ignored                    tick_0.msgpack              test_gauntlet_golden.gd
       ↓                         tick_10.msgpack                verifies each fixture
  Boots Gauntlet map              tick_0.json (human-readable    decodes to expected
  with seed=42                    diff companion)                field values
  Runs to tick 0, 10
  Serializes ObserverSnapshot
  from fixed player position
  Writes .msgpack + .json

Key design decisions:

  1. Format: ObserverSnapshot, not full ECS world. The snapshot is what the client sees — it's the contract surface. Internal ECS changes that don't affect the snapshot are allowed without golden file breakage. This keeps the golden file stable and focused.

  2. JSON companion for diff. Each .msgpack fixture gets a parallel .json file containing the same data in human-readable form. When the golden file breaks, git diff on the JSON shows exactly what changed (e.g., "entity_id changed from 100 to 101" or "new field zone_id appeared").

  3. Fixed player position. The Gauntlet spec defines a named position per room (e.g., GAUNTLET.occlusion_corridor.observer_position = (15, 10, 0)). Golden files are generated from that exact position. Moving the observer changes what's visible.

  4. Seed pinning. seed=42 (or whatever the Gauntlet uses). The golden file test MUST fail if the seed produces different output, which catches non-determinism bugs.

CI integration:

# Pseudo-CI pipeline (Gitea Actions or equivalent)
golden-file-check:
  steps:
    - cargo build --release
    - cargo test --test gen_gauntlet_golden -- --ignored
    # Generates fresh .msgpack files in a temp directory
    - diff client/tests/fixtures/gauntlet/tick_0.msgpack /tmp/gauntlet_fresh/tick_0.msgpack
    # If diff is non-empty, the golden file is stale → FAIL
    # Developer must regenerate and commit updated golden files

When golden files legitimately change:

  1. Developer changes server logic that affects snapshot output
  2. Developer runs make fixtures-gauntlet (regenerates golden files)
  3. Developer inspects the JSON diff: git diff client/tests/fixtures/gauntlet/tick_0.json
  4. If the diff is expected, commit the updated golden files
  5. PR review includes the golden file diff as mandatory review artifact

Breakage detection granularity:

The golden file test should not just do byte comparison. It should decode both the golden file and the fresh snapshot, then compare field by field:

#[test]
fn gauntlet_tick_0_matches_golden() {
    let golden: ObserverSnapshot = load_golden("tick_0");
    let fresh: ObserverSnapshot = run_gauntlet_to_tick(0);

    assert_eq!(golden.tick, fresh.tick);
    assert_eq!(golden.entities.len(), fresh.entities.len());
    for (g, f) in golden.entities.iter().zip(fresh.entities.iter()) {
        assert_eq!(g.entity_id, f.entity_id, "entity_id mismatch");
        assert!((g.x - f.x).abs() < 0.01, "x mismatch for entity {}", g.entity_id);
        assert!((g.y - f.y).abs() < 0.01, "y mismatch for entity {}", g.entity_id);
    }
    // ... etc for all fields
}

This gives actionable error messages instead of "binary files differ."

T4-H5: Minimum Viable Layer 3 Test (Real Subprocess Integration)

Current state assessment:

Layer Status What exists
Layer 1 (fixtures) Complete gen_fixtures.rs generates, test_protocol.gd + serialization.rs verify
Layer 2 (bridge) Partial bridge_tcp.rs + bridge_ipc.rs test roundtrip. game_loop.rs tests full pipeline. But all run within one process (server as thread, not subprocess).
Layer 3 (real subprocess) Missing No test launches the actual settled-reach-server binary as a child process

Why Layer 3 matters: Bug #1 (server never sends snapshots in live mode) was caused by read_framed() blocking the bevy Update schedule. This bug only manifests with the real binary running as a subprocess — the in-process thread tests in game_loop.rs don't reproduce it because the thread shares memory and doesn't have the same blocking behavior.

Minimum viable Layer 3 test specification:

Test name: server_subprocess_sends_snapshot_on_connect
File: tests/integration_subprocess.rs (or tests/layer3.rs)

Setup:
  1. cargo build the server binary (or use pre-built from CI artifact)
  2. Launch server binary as child process: `Command::new("target/debug/settled-reach-server")`
     with args: --test-mode --port 0 (random port, printed to stdout)
  3. Parse port from server stdout
  4. Connect to server via TCP

Test steps:
  1. Send one PlayerInput (MoveNorth, tick 0) via write_framed
  2. Read one ObserverSnapshot via read_framed with 5-second timeout
  3. Assert: snapshot.version == PROTOCOL_VERSION
  4. Assert: snapshot.tick == 0
  5. Assert: snapshot.entities.len() >= 1 (at least the player)
  6. Assert: player entity has kind == Player

Teardown:
  1. Drop TCP connection
  2. Kill child process (SIGTERM)
  3. Wait for exit with timeout

Duration budget: < 10 seconds (including server startup)
Frequency: Nightly + pre-merge (D-030 says "slow, daily/pre-merge")

Implementation requirements:

  1. Server needs --test-mode flag. This flag should:

    • Load the proof room (or Gauntlet) instead of requiring content files
    • Print the listening port to stdout: LISTENING:9876
    • Use a fixed seed for determinism
    • Exit after first client disconnects (or after timeout)

  2. Server needs --port 0 support. Bind to random available port and print it. Prevents test flakiness from port conflicts.

  3. Test function:

use std::process::{Command, Stdio};
use std::io::{BufRead, BufReader as StdBufReader};
use std::net::TcpStream;
use std::time::Duration;

#[test]
#[ignore] // Layer 3: slow, run with --ignored or in CI nightly
fn server_subprocess_sends_snapshot_on_connect() {
    // Build the server binary
    let status = Command::new("cargo")
        .args(["build", "--bin", "settled-reach-server"])
        .status()
        .expect("cargo build failed");
    assert!(status.success(), "server build failed");

    // Launch server as subprocess
    let mut server = Command::new("target/debug/settled-reach-server")
        .args(["--test-mode", "--port", "0"])
        .stdout(Stdio::piped())
        .stderr(Stdio::inherit())
        .spawn()
        .expect("failed to launch server");

    // Parse port from stdout
    let stdout = server.stdout.take().unwrap();
    let mut lines = StdBufReader::new(stdout).lines();
    let port: u16 = loop {
        let line = lines.next().expect("server stdout ended").expect("read line");
        if let Some(port_str) = line.strip_prefix("LISTENING:") {
            break port_str.trim().parse().expect("parse port");
        }
    };

    // Connect as client
    let stream = TcpStream::connect(format!("127.0.0.1:{}", port))
        .expect("connect to server");
    stream.set_read_timeout(Some(Duration::from_secs(5))).unwrap();

    let mut reader = std::io::BufReader::new(stream.try_clone().unwrap());
    let mut writer = std::io::BufWriter::new(stream);

    // Send input
    let inputs = vec![PlayerInput {
        tick: 0,
        action: PlayerAction::MoveNorth,
    }];
    let payload = rmp_serde::to_vec_named(&inputs).expect("serialize");
    write_framed(&mut writer, &payload).expect("send input");

    // Receive snapshot
    let response = read_framed(&mut reader)
        .expect("read snapshot")
        .expect("not EOF — server must send snapshot after receiving input");

    let snapshot: ObserverSnapshot =
        rmp_serde::from_slice(&response).expect("deserialize snapshot");

    // Assertions
    assert_eq!(snapshot.version, PROTOCOL_VERSION,
        "snapshot version mismatch — protocol incompatibility");
    assert_eq!(snapshot.tick, 0);
    assert!(!snapshot.entities.is_empty(),
        "snapshot must contain at least the player entity");

    let player = snapshot.entities.iter()
        .find(|e| matches!(e.kind, EntityKind::Player))
        .expect("no Player entity in snapshot");
    assert!(player.x > 0.0, "player x must be positive");
    assert!(player.y > 0.0, "player y must be positive");

    // Cleanup
    drop(reader);
    drop(writer);
    server.kill().ok();
    server.wait().ok();
}

What this catches that Layer 2 doesn't:

  • Server binary startup issues (missing resources, config, panic on init)
  • Subprocess I/O blocking (Bug #1 class)
  • Protocol version mismatch between compiled server and test expectations
  • Real TCP behavior (Nagle's algorithm, buffer sizes, connection lifecycle)

TRACK 5 — Content Scaling & CI Pipeline

T5-H1: Why Don't Client Tests Run in CI?

Finding: There IS no CI pipeline at all. No .github/, .gitea/, or other CI configuration files exist in the repository. The Makefile has ci, ci-server, ci-client targets, but these are designed for local execution only.

The blockers for CI are, in order of priority:

  1. No CI runner configured. The project uses Gitea (git.schweitz.internal), which supports Gitea Actions (GitHub Actions compatible). But no workflow files exist. This is the primary blocker — it's wiring, not a technical limitation.

  2. Godot headless availability. make test-client uses $(GODOT) --headless --path client -s res://addons/gdUnit4/bin/GdUnitCmdTool.gd --ignoreHeadlessMode. This requires:

    • Godot 4.6 binary on the CI runner
    • The --headless flag (supported since Godot 4.0)
    • No GPU required (headless mode uses software rendering)
    • The --ignoreHeadlessMode flag for gdUnit4 (already present in Makefile)

    This is solvable: install Godot on the CI runner, or use the official Godot Docker image.

  3. gdUnit4 test runner stability in headless. gdUnit4 documents headless CI support via GdUnitCmdTool.gd. I haven't verified whether the current test suite actually passes headless — some tests may have implicit dependencies on window size, input events, or rendering state. This needs a verification run: make test-client on a headless machine.

  4. Test output format. D-030 specifies JSON summary output. gdUnit4 produces JUnit XML by default. The CI pipeline needs a format adapter — or we accept JUnit XML for CI (most CI systems parse it natively).

Recommended action: The first step is just running make ci on the Gitea runner. Everything else is refinement.

T5-H2: Minimal CI Pipeline — Tier Proposal

Tier Trigger Tests Duration budget Rationale
Commit (fast feedback) Every push to any branch make lint-server (clippy + fmt), make lint-client (GDScript error check), make validate-content (YAML schema), make check-fact-ids < 2 min Catches syntax errors, formatting, broken content. No compilation needed for content checks.
PR (merge gate) PR opened or updated Everything in Commit tier + make build (both server + client), make test-server (cargo nextest), make test-client (gdUnit4 headless), make fixtures + verify no diff (fixture staleness check) < 10 min Full build + test. Fixture staleness check catches protocol changes that weren't regenerated.
Nightly (deep validation) Scheduled, 1x/day on main Everything in PR tier + Layer 3 subprocess test (cargo test --test layer3 -- --ignored), Gauntlet golden file regeneration + diff check, Content load-test (boot server with full content, tick 100 times), Performance benchmark (Gauntlet 100 ticks, assert < time budget) < 30 min Slow tests that catch subtle integration bugs. Performance regression detection.

Pipeline configuration (Gitea Actions):

# .gitea/workflows/ci.yaml

name: CI
on:
  push:
    branches: ['*']
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 3 * * *'  # Nightly at 03:00

jobs:
  commit-checks:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4
      - run: make lint-server
      - run: make lint-client
      - run: make validate-content
      - run: make check-fact-ids

  pr-checks:
    if: github.event_name == 'pull_request'
    needs: commit-checks
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4
      - run: make build
      - run: make test-server
      - run: make test-client
      - run: make fixtures
      - run: git diff --exit-code client/tests/fixtures/
        # Fails if fixtures are stale

  nightly:
    if: github.event_name == 'schedule'
    needs: commit-checks
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4
      - run: make build
      - run: make test-server
      - run: make test-client
      - run: cargo test --test layer3 -- --ignored
      - run: make test-gauntlet-golden  # (new target, TBD)
      - run: make test-content-load     # (new target, TBD)

Critical gate: PR tier is the merge gate. A PR cannot merge if any PR-tier test fails. Nightly failures generate alerts but don't block work.

T5-H3: Minimum Runtime Content Validation

Current state: make validate-content only checks YAML structure against JSON Schema. It does NOT verify that the content can actually be loaded and used by the server.

Minimum runtime validation — "boot and tick" test:

Test name: content_loads_and_ticks_without_panic
File: server/tests/content_loading.rs (extend existing)

Setup:
  1. Load content from content/campaigns/ using ContentLoader
  2. Build a minimal bevy App with SimulationPlugin + ContentPlugin
  3. Insert loaded content as resources

Test steps:
  1. app.update() — tick once
  2. Assert: no panic (the test itself succeeding is the assertion)
  3. Assert: SimulationTime.tick == 1
  4. Assert: at least 1 entity with PlayerCharacter component exists
  5. Assert: at least 1 entity with NPC-related components exists (if content defines NPCs)

Duration: < 5 seconds
Frequency: Every PR

Graduated runtime validation layers:

Level What it checks Test type Duration
1. Boot Content loads without panic Unit test in content_loading.rs < 2s
2. Tick 10 ticks complete without panic Integration test < 3s
3. Snapshot ObserverSnapshot generates with expected entity count Integration test < 5s
4. Cross-reference All entity_ref in dialogue resolves to spawned entities Dedicated validation test < 3s
5. Stress 100 ticks with max-NPC content pack, no tick exceeds 50ms Performance test < 30s

Level 1 already partially exists in content_loading.rs. We should extend it to Level 3 as the minimum for PR gating.

Concrete Level 3 test:

#[test]
fn content_produces_valid_snapshot() {
    let mut app = build_app_with_content("content/campaigns/meridian");

    // Tick 10 times
    for _ in 0..10 {
        app.update();
    }

    // Generate snapshot from player position
    let snapshot = extract_observer_snapshot(&app);

    assert!(snapshot.tick > 0, "simulation must have advanced");
    assert!(!snapshot.entities.is_empty(), "snapshot must contain entities");

    // Verify player exists in snapshot
    let has_player = snapshot.entities.iter().any(|e| matches!(e.kind, EntityKind::Player));
    assert!(has_player, "player must be visible in own snapshot");

    // Verify NPC count matches content expectation
    let npc_count = snapshot.entities.iter().filter(|e| matches!(e.kind, EntityKind::Npc)).count();
    assert!(npc_count >= 1, "at least one NPC should be visible from spawn position");
}

T5-H4: Content Scaling Test — "Adding an NPC Doesn't Break Anything"

The test: When a new NPC is added to a district, the game must still boot, tick, and produce valid snapshots.

Test specification:

Test name: adding_npc_to_district_preserves_functionality
File: server/tests/content_scaling.rs (new)

Approach: Comparative testing
  1. Load baseline content → boot → tick 10 → snapshot (baseline)
  2. Load baseline content + 1 extra NPC → boot → tick 10 → snapshot (modified)
  3. Compare: modified snapshot should be a SUPERSET of baseline

Assertions:
  - Modified boots without panic
  - Modified ticks 10 times without panic
  - Modified snapshot has >= baseline entity count
  - Modified snapshot has exactly baseline.npcs + 1 NPCs
  - All baseline NPCs still present (by entity name/stable_id)
  - Player entity unchanged between baseline and modified
  - No tick exceeds 50ms (performance regression check)
  - All nearby_interactions for existing NPCs still present

Concrete implementation:

#[test]
fn adding_npc_preserves_existing_entities() {
    // Baseline: load district with known NPCs
    let baseline_app = build_app_with_content("content/campaigns/meridian");
    let baseline_snapshot = run_and_snapshot(&baseline_app, 10);
    let baseline_npc_ids: HashSet<u64> = baseline_snapshot.entities.iter()
        .filter(|e| matches!(e.kind, EntityKind::Npc))
        .map(|e| e.entity_id)
        .collect();

    // Modified: same district + one extra NPC injected programmatically
    let mut modified_app = build_app_with_content("content/campaigns/meridian");
    inject_test_npc(&mut modified_app, "test_extra_npc", TilePosition::new(20, 20, 0));
    let modified_snapshot = run_and_snapshot(&modified_app, 10);
    let modified_npc_ids: HashSet<u64> = modified_snapshot.entities.iter()
        .filter(|e| matches!(e.kind, EntityKind::Npc))
        .map(|e| e.entity_id)
        .collect();

    // All baseline NPCs still exist
    assert!(baseline_npc_ids.is_subset(&modified_npc_ids),
        "existing NPCs disappeared after adding new NPC");

    // Exactly one more NPC
    assert_eq!(modified_npc_ids.len(), baseline_npc_ids.len() + 1,
        "expected exactly one additional NPC");

    // Player still exists and at same position
    let baseline_player = find_player(&baseline_snapshot);
    let modified_player = find_player(&modified_snapshot);
    assert_eq!(baseline_player.x, modified_player.x);
    assert_eq!(baseline_player.y, modified_player.y);
}

Scaling test matrix:

Scenario Baseline Modified Key assertion
+1 NPC 3 NPCs 4 NPCs All original NPCs present
+5 NPCs 3 NPCs 8 NPCs No tick > 50ms
+15 NPCs 3 NPCs 18 NPCs (Crowd Plaza scale) No tick > 100ms, no panic
+1 location 1 district 1 district + 1 location YAML Content loads, entities spawn correctly
+1 item 9 items 10 items Inventory interaction still works (9-slot limit not exceeded by world items)

Content-level validation (complement to runtime tests):

Add to make validate-content:

  • Entity reference check: Every entity_ref in dialogue/monologue YAML resolves to a defined NPC profile
  • Location slug check: Every location field matches a defined location YAML
  • Fact ID check: Already exists as make check-fact-ids
  • District capacity check: Warn if a district exceeds 15 NPCs (Crowd Plaza performance boundary)

Summary of Deliverables

Deliverable Track Status
Boundary value test matrix (41 values) T4-H1 Complete
Boundary test placement recommendation T4-H2 Complete
gen_fixtures.rs extension specification T4-H3 Complete
Golden file pipeline specification T4-H4 Complete
Layer 3 test specification with code T4-H5 Complete
CI blocker analysis T5-H1 Complete
CI pipeline tier proposal (commit/PR/nightly) T5-H2 Complete
Runtime content validation specification T5-H3 Complete
Content scaling test specification T5-H4 Complete

Open Questions for Round 2

  1. For Dudley: Does the server binary support --test-mode and --port 0 flags? If not, what's the minimum change to add them? Layer 3 depends on this.
  2. For Dudley: The GDScript encoder uses int_16 for positive values 256-32767 while rmp_serde uses uint_16. Does rmp_serde::from_slice::<u64>() accept int_16-encoded positive values? I believe it does, but this needs explicit verification.
  3. For Tyre/Justine: The Gitea instance at git.schweitz.internal — does it have Gitea Actions enabled? Is there a self-hosted runner available? CI pipeline design depends on this.
  4. For Tyre: The golden file JSON companion format — should it be the full ObserverSnapshot serialized to JSON, or a reduced "diff-friendly" format with sorted keys and abbreviated fields?
  5. For Justine: Fixture staleness check in CI (make fixtures && git diff --exit-code) — is this robust enough, or do we need a content-addressed hash approach?
Reference in New Issue View Git Blame Copy Permalink