settled-reach/docs/workshops/test-architecture/test-architecture-workshop-brief.md

QA Strategy & Test Architecture Workshop Brief

Goal: Design a comprehensive testing strategy — test world, automation framework, content scaling safety, and CI pipeline — that catches integration boundary bugs before they reach manual playtesting, and scales safely as content grows. Produce concrete specifications and a prioritized implementation backlog.

Participants: Tyre, Hoshe, Dudley, Stig, Justine, Gestalt, Ozzie

Note for Tyre: You may spawn Troblum as a sparring partner for feasibility questions if you need a second opinion on architecture or technology choices.

Context: Sprint 6-7 development exposed a pattern of bugs that unit tests don't catch and manual make game testing finds too late. The bugs live at integration boundaries (client-server, encode-decode, ECS schedule ordering, pause/unpause state transitions). D-030 defined the test architecture but implementation has lagged. Simultaneously, the project is about to scale content generation (more NPCs, locations, items, districts) — we need confidence that new content won't introduce regressions. This workshop designs the QA infrastructure to keep everything tight and under control.

The Bug Catalogue

Real bugs from Sprint 6-7. Every test specification produced by this workshop must prevent at least one class of these from recurring.

#	Bug	Root Cause	Category	Status
1	Server never sends snapshots in live mode	read_framed() blocking TCP read stalls entire bevy Update schedule	IPC / schedule integration	Fixed
2	Camera doesn't center on player at startup	Client receives no snapshot until first keystroke (consequence of #1)	Client-server startup sequencing	Fixed
3	Player moves while game is paused	process_player_input processes movement regardless of TickRate	State transition / input filtering	Fixed
4	MessagePack encodes tick 128 as -128	Signed int8 branch uses <= instead of < for upper bound	Serialization boundary	Fixed (all int sizes)
5	Monologue lost when snapshots overwrite	Latest-wins snapshot buffer drops one-shot events	Event delivery reliability	Fixed (carry-forward)
6	Snapshot overwrite warning spam	Server ticks faster than client consumes	Rate mismatch	Fixed (carry-forward eliminates warning)

The Test World — "The Gauntlet"

A purpose-built, permanently stable server-side map designed as a QA playground. Not a game level — a mechanical test suite in map form. The Gauntlet is the golden file anchor: its layout never changes, so regression tests against it are reliable. When new systems are added, new rooms are added to the Gauntlet — existing rooms stay frozen.

Dynamic/procedural maps get separate, fuzzier assertion-based tests (e.g., "player can see at least 3 NPCs" not "player sees kael at (12,7)").

Proposed Rooms

Room	Systems Exercised	Key Assertions
Inventory Warehouse	Pickup, CarriedBy, inventory grid, 9-slot limit	Take when full, drop, re-take, slot assignment stability
Occlusion Corridor	LOS, shadowcasting, vision cone, perception modes	NPC behind wall: invisible in Visual, detected in Sensor, heat signature in IR, footsteps in Sound. Each perception mode yields different ObserverSnapshot content
Interaction Gallery	One entity per ObjectType + one multi-verb NPC	Single-verb Interact, multi-verb scroll/selection, verb priority ordering, sprint suppression clears interaction buffer
Crowd Plaza	15+ NPCs at various relationship states/distances	Entity rendering at density, cognitive delay overlap, D-033 color spread, performance (tick budget)
Fog Theater	Open area → corridor → room with controlled LOS transitions	Fog layer transitions, peripheral dimming, remembered entities after LOS exit, exploration texture persistence
Dialogue Room	NPCs at different trust tiers, one with contradiction	Full dialogue tree traversal, walk-away mid-dialogue, confrontation trigger, monologue during cognitive delay
Pause Chamber	TickRate toggle, state transitions	Movement discarded while paused, pause/unpause round-trip, UI state during pause, stance toggle during pause
Zone Gate (provisioned)	Zone transition, entity persistence, camera behavior	Reserved space in the map layout. Implementation deferred until multi-map system exists. Test specifications written as future contracts — "when zone transitions ship, these assertions must pass."

Design Constraints

• Stable coordinates: Every entity, wall, and item has a fixed position documented in the Gauntlet spec. Regression tests reference these positions by name (e.g., GAUNTLET.occlusion_npc_behind_wall), not raw coordinates.
• Deterministic content: The Gauntlet loads from a fixed content pack with a fixed seed. No procedural variation. Same seed → same world state at tick 0.
• Additive only: Existing rooms are never modified. New systems get new rooms appended to the map. This preserves all existing golden files.
• Launchable: make test-world boots server with the Gauntlet map and connects the client. make test-world-headless boots server only for automated tests.

Workshop Tracks

Track 1: Test World Design (Gestalt, Dudley, Tyre, Ozzie)

Design the Gauntlet map — room layout, entity placement, scenario coverage, and the human tester walkthrough flow.

Questions for Gestalt:

1. Review the proposed room list. What mechanical interactions are missing? Think about system combinations — what happens when the player sprints through the Crowd Plaza into the Occlusion Corridor? When they try to interact while in peripheral vision?
2. What makes a good "stress test" room? Maximum entity density? Maximum interaction depth? Both?
3. The Gauntlet is stable, but dynamic maps need fuzzy tests. What invariants should hold for ANY valid map? (e.g., "player spawn is always reachable", "at least one NPC is within interaction range within 10 tiles of spawn")

Questions for Dudley:

1. The current proof room (spawn_proof_room in content_loader.rs) is a 30x30 box with 3 NPCs. How do we extend this into the Gauntlet? Separate content YAML, or a programmatic builder function?
2. Each room needs a known "tick 0 state" for golden file comparison. How do we snapshot this — serialize the full ECS world, or just the ObserverSnapshot from a fixed player position?
3. The Gauntlet needs entities with specific knowledge graph states (contradiction for Dialogue Room, different trust tiers). How do we inject these at load time?

Questions for Tyre:

1. Should the Gauntlet be a content pack (YAML/RON loaded by the content loader) or a Rust builder function (like the current proof room)? Content pack is more maintainable but harder to set up precise ECS state.
2. The "reserved Zone Gate" — what's the minimum architectural provision? A marker in the map? An empty module with trait stubs?
3. Performance budget: what tick time is acceptable for the Gauntlet with 15+ NPCs? When should we raise an alarm?

Questions for Ozzie:

1. A human tester walks through the Gauntlet room by room. What's the optimal flow? Linear corridor connecting rooms, or hub-and-spoke from a central area?
2. What should the tester see on screen that tells them "this room is testing X"? Signs? HUD overlay? A printed checklist they follow?
3. The text output mode (see Track 3) replaces visual rendering with structured text. What information does a human tester need in text form to verify "this looks correct"?

Track 2: Deterministic Gameplay (Tyre, Dudley)

The project uses SimRng (ChaCha20, seeded) for randomness, but full determinism — same inputs + same seed = identical game state — has not been validated or hardened. This track evaluates whether to commit to determinism now.

The position: Hardening determinism now is cheaper than retrofitting later. Every system added without determinism constraints makes the retrofit harder. But there may be gameplay or performance costs.

Questions for Tyre:

1. Pros and cons of committing to determinism now. What breaks? What do we gain? What's the ongoing maintenance cost? Consider: HashMap iteration order (non-deterministic in Rust), floating-point operations (platform-dependent), bevy system ordering (parallel execution), external I/O timing.
2. Where is determinism already broken today? Audit the critical path: process_player_input → validate_movement → compute_observer_snapshot. Which of these use HashMap, f32 arithmetic, or unordered queries?
3. What would a "determinism test" look like? Run the Gauntlet with seed X and inputs [A, B, C], serialize world state at tick 50, compare against golden file. Is this feasible with bevy_ecs?
4. Performance implications: replacing HashMap with BTreeMap everywhere? Deterministic float alternatives?

Questions for Dudley:

1. Which ECS queries are order-dependent today? If two NPCs are equidistant from the player, does the system process them in a stable order?
2. The SimRng is seeded but is it consumed in a deterministic order? If system execution order varies, RNG calls happen in different order → different outcomes.
3. What's the minimum change set to make the server deterministic for the Gauntlet? Can we scope it to "deterministic for single-player, single-thread" as a starting point?

Track 3: Test Automation & Text Renderer (Tyre, Dudley, Stig)

Two automation tools: scripted input replay with snapshot assertions, and a text-based game state renderer for human verification without a GPU.

Scripted replay concept:

# test_inventory_full.replay
@gauntlet seed=42
spawn_at inventory_warehouse_entrance
move_to crate_1     # pathfind or explicit directions
interact crate_1 Take
assert inventory.count == 1
repeat 8: interact crate_N Take   # fill all 9 slots
interact crate_10 Take
assert inventory.count == 9       # didn't overflow
assert snapshot.nearby_interactions[crate_10].verbs contains "Take"
assert snapshot.nearby_interactions[crate_10].verbs["Take"].available == true
# ^ server should still offer Take even when inventory full — client greys it out

Text renderer concept:

=== Tick 42 | Gauntlet: Occlusion Corridor ===
Player (15,10) facing East | Stance: Walk | TickRate: Full
Visible entities:
  npc:guard-1     (18,10) Forward  relationship:Neutral  visible
  npc:worker-2    (20,10) Forward  relationship:Unknown  remembered
  [wall at (17,10) blocks LOS to npc:hidden-1 at (19,12)]
Fog: 31 visible, 58 explored, 412 unexplored
Interactions: guard-1 [Talk(1), Observe(2)] distance=3
Inventory: 2/9 [keycard(slot-0), manifest(slot-3)]
Monologue: none | Dialogue: none

Questions for Tyre:

1. Scripted replay requires a "test client" — a Rust program that connects to the server over TCP, sends scripted inputs, and reads snapshots. Is this a new binary, a test harness in tests/, or an extension of the existing game_loop integration test?
2. The text renderer — should it live server-side (format the ObserverSnapshot before sending), client-side (format after receiving), or as a standalone tool that reads snapshots from a file/pipe?
3. What assertion language? Custom DSL (like above), Rust test macros, or a data-driven approach (YAML expected-state files compared against actual)?

Questions for Dudley:

1. The server already produces ObserverSnapshot with all the data needed for text rendering. What's missing? (Hint: wall positions in the LOS path aren't in the snapshot — only visible tiles.)
2. The scripted replay needs to inject inputs at specific ticks. The current InputQueue accepts PlayerInput { tick, action }. Can we pre-load a sequence from a file?
3. How do we handle "wait for condition" in scripts? (e.g., "wait until cognitive delay completes" — variable tick count depending on entity distance)

Questions for Stig:

1. The text renderer replaces visual output for automated testing. But human testers also need a checklist — "in the Occlusion Corridor, verify: NPC behind wall is NOT visible, NPC in front of wall IS visible, peripheral NPC is dimmed." Where does this checklist live? In the Gauntlet spec? Generated from room metadata?
2. Client-side text rendering: should the client have a --text-mode flag that replaces the Godot renderer with a terminal-output formatter? Or is this purely a server-side tool?
3. The client tests (gdUnit4) are structural — they check scene tree state, not pixels. What client-side properties are worth asserting beyond what the text renderer shows? (Camera position, z-layer ordering, node visibility, modulate alpha values?)

Track 4: Serialization & Integration Testing (Hoshe, Dudley)

The MessagePack boundary bug (bug #4) was fixed across all int sizes. This track ensures it never recurs and extends coverage to the full wire protocol.

Current state:

• test_protocol.gd and serialization.rs test roundtrips within their respective languages
• bridge_tcp.rs tests cross-language roundtrip for snapshots and inputs
• Boundary value bugs at int8/16/32/64 are now fixed, but no tests specifically exercise boundary values
• gen_fixtures.rs generates MessagePack fixtures but not boundary value fixtures

Questions for Hoshe:

1. Design a boundary value test matrix for the MessagePack encoder. Every power-of-two boundary where the format changes, both positive and negative. Include the exact values.
2. Should boundary tests live in the client (GDScript), server (Rust), or both? Cross-language roundtrip tests catch the real bugs but are slower.
3. The gen_fixtures.rs test generates MessagePack fixtures — should it generate boundary value fixtures that the client can verify?
4. Propose a "golden file" approach tied to the Gauntlet: server generates canonical MessagePack bytes for a known Gauntlet snapshot, client verifies it decodes to the same values. How would this work in CI?
5. D-030 defined three IPC test layers. Layer 1 (fixtures) and partial Layer 2 (bridge) exist. Layer 3 (real subprocess integration) is missing. What's the minimum viable Layer 3 test?

Questions for Dudley:

1. The Rust PlayerInput uses tick: u64 but the client sends Godot's signed int (63-bit range). What's the actual valid range, and should the server reject negative ticks explicitly?
2. rmp_serde silently rejects -128 for a u64 field. Should the bridge layer pre-validate and log, or is the current error-and-skip behavior acceptable?
3. process_player_input has no test for pause-state filtering. Propose the test cases for the pause guard (movement discarded, unpause accepted, stance toggle during pause — allowed or not?).
4. The EntityRegistry / StableId mapping is a correctness boundary. Are there enough tests for entity lifecycle (spawn, register, lookup, despawn)?

Track 5: Content Scaling & CI Pipeline (Hoshe, Justine, Tyre)

When content scales from 3 NPCs to 30, from 1 district to 5, from 9 items to 50 — what breaks? This track designs the safety net.

Content validation layers:

1. Schema validation (exists: make validate-content) — YAML structure correctness
2. Cross-reference validation (partial) — entity refs resolve, fact_ids exist, location slugs match, dialogue pool tags use valid enums
3. Load-test validation (doesn't exist) — boot the server with a stress content pack, tick 100 times, no panics, snapshots arrive within frame budget
4. Regression snapshots — the Gauntlet golden file: seed X produces this exact snapshot at tick 10. Any code/content change that breaks it is flagged.

Questions for Hoshe:

1. The client tests don't run in CI yet. What's the blocker — headless Godot availability, test runner stability, or just wiring?
2. Propose a minimal CI pipeline: which tests run on every commit, which are PR-only, which are nightly?
3. The current make validate-content checks YAML schema but not runtime behavior. What's the minimum runtime validation — "boot server, load content, tick once, no panics"?
4. Content scaling: what's the test for "adding a new NPC to the transit district doesn't break anything"? Load the district, verify entity count, verify interactions still work?

Questions for Justine:

1. The server runs cargo nextest for Rust tests. The client runs make test-client for gdUnit4. Neither runs in CI. What's the minimum CI pipeline that runs both?
2. The Gauntlet golden file test needs both server build + test world content. How should CI manage this artifact? Build once, cache, reuse across test jobs?
3. Performance regression detection: the Gauntlet should complete 100 ticks within a time budget. How do we track this across commits without noise from CI machine variance?
4. When a golden file breaks, the developer needs to see what changed. Diff format for ObserverSnapshot comparison — structured diff, or just "expected vs actual" dump?

Questions for Tyre:

1. The Gauntlet golden file is the anchor for regression testing. What format — serialized ECS world state, or ObserverSnapshot at a fixed position? Snapshot is smaller and closer to what the client sees; world state catches server-only bugs.
2. Fuzzy tests for dynamic maps: what invariants should always hold regardless of procedural generation? (Reachability, minimum NPC count, spawn safety, no overlapping entities?)
3. How do we test content combinations? NPC A's dialogue references NPC B's secret — if B is removed from the content pack, A's dialogue breaks. Is this a schema problem, a load-time validation problem, or a runtime test?

Workshop Format

Round 1 (analysis): Each participant answers their track questions independently. Reference existing code, D-030, and the bug catalogue. Produce concrete test specifications (function names, assertions, setup), not general principles.

Pause after Round 1 for lead review of summaries, open decisions, and questions.

Round 2 (synthesis): Cross-review and integration. Tyre validates Stig's client proposals for architectural soundness. Hoshe validates Dudley's server proposals for coverage completeness. All participants weigh in on the deterministic gameplay decision. Reconcile overlapping proposals across tracks.

Pause after Round 2 for lead review.

Round 3 (prioritization): Rank all proposed tests and infrastructure by implementation priority. Produce the final test backlog, concrete specs for top implementations, and the CI pipeline design.

Pause after Round 3 for lead review and closing.

Output

• Decision on deterministic gameplay commitment (with documented pros/cons)
• Gauntlet test world specification (room layout, entity placement, golden file format)
• Prioritized test backlog (tickets) covering the bug catalogue + content scaling
• Concrete test specifications for the top implementations
• Text renderer specification
• Human tester checklist template for the Gauntlet walkthrough
• CI pipeline design (what runs when, what gates what)
• Scripted replay framework specification
• Content validation pipeline design