settled-reach/docs/workshops/llm-voice-pipeline/tyre-round3.md

title, description, type, status, workshop, agent, round, created

title	description	type	status	workshop	agent	round	created
Tyre Round 3: Implementation Specification	Spike 1 implementation spec, model test plan, spike 2 outline, and hardware detection design	workshop	archived	llm-voice-pipeline	tyre	3	2026-03-07

Tyre Round 3: Implementation Specification

Domain: Technical architecture Round: 3 — Decision & Implementation Plan Deliverables: Spike 1 implementation spec, model test plan, Spike 2 outline, hardware detection design

---

1. Spike 1 Implementation Spec: sr-voice CLI Tool

Purpose

A standalone Rust CLI that loads a GGUF model, accepts prompts, and returns generated text. No game integration, no queue, no cache. This is the plumbing that Jeroen, Mellanie, and Paula will feed manually-crafted prompts through to answer: "does this even play?"

Crate structure

server/
  sr-voice/
    Cargo.toml
    src/
      main.rs          # CLI entry point
      inference.rs     # Model loading and generation wrapper
      prompt.rs        # Prompt file parsing and construction

sr-voice is a separate crate in the server workspace, not compiled into the game binary. It depends on llama-cpp-rs (or llama-cpp-2 — see build notes) and produces a standalone binary: sr-voice.

Cargo.toml dependencies

[package]
name = "sr-voice"
version = "0.1.0"
edition = "2021"

[dependencies]
llama-cpp-2 = { version = "0.1", features = ["metal", "vulkan"] }
# Note: "metal" and "vulkan" are optional features, compile-time gated.
# CPU-only is the default and always available.
clap = { version = "4", features = ["derive"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"

[features]
default = []
gpu-metal = ["llama-cpp-2/metal"]
gpu-vulkan = ["llama-cpp-2/vulkan"]

Build note on llama-cpp-rs vs llama-cpp-2: Both wrap the same C library. llama-cpp-2 is the more actively maintained fork as of early 2026 and has cleaner safe Rust wrappers. Evaluate both at spike start; pick whichever compiles cleanly on Linux + macOS + Windows without manual C++ toolchain intervention. Pin the llama.cpp commit hash in Cargo.toml to prevent upstream API breaks.

Build dependency: Requires a C/C++ compiler (gcc/clang/MSVC). CMake is pulled in by the llama.cpp build system. This is a compile-time dependency, not a runtime dependency — the final binary is self-contained.

CLI interface

sr-voice --model <path-to-gguf> [OPTIONS] [PROMPT_FILE]

Options:
  --model <path>        Path to GGUF model file (required)
  --threads <n>         CPU threads for inference (default: physical_cores - 1)
  --ctx-size <n>        Context window size in tokens (default: 512)
  --max-tokens <n>      Maximum output tokens (default: 64)
  --temperature <f>     Sampling temperature (default: 0.7)
  --top-p <f>           Top-p sampling (default: 0.9)
  --seed <n>            RNG seed for sampling (default: random)
  --json                Output as JSON: {"input": "...", "output": "...", "tokens_per_sec": N}
  --batch               Process multiple prompts from a JSONL file (one per line)
  --benchmark           Run 5 inference calls and report avg tokens/sec

PROMPT_FILE:
  Read prompt from file (plain text). If omitted, reads from stdin.

Core function signatures

// inference.rs

/// Configuration for the inference engine.
pub struct InferenceConfig {
    pub model_path: PathBuf,
    pub n_threads: u32,
    pub ctx_size: u32,
    pub seed: Option<u32>,
}

/// A loaded model ready for inference.
pub struct InferenceEngine {
    // Wraps llama_model + llama_context from llama-cpp-2.
    // Model is loaded once; context is reused across calls.
    model: LlamaModel,
    ctx: LlamaContext,
}

impl InferenceEngine {
    /// Load a GGUF model from disk. Returns an error if the model
    /// doesn't fit in available RAM or the file is invalid.
    ///
    /// Typical load time: 2-5 seconds for a 2B Q4 model from SSD.
    pub fn load(config: &InferenceConfig) -> Result<Self, VoiceError>;

    /// Run inference on a prompt string. Returns the generated text.
    ///
    /// `max_tokens`: maximum output tokens (stops early on EOS).
    /// `temperature`: sampling temperature (0.0 = greedy, 1.0 = creative).
    /// `top_p`: nucleus sampling threshold.
    pub fn generate(
        &mut self,
        prompt: &str,
        max_tokens: u32,
        temperature: f32,
        top_p: f32,
    ) -> Result<GenerationResult, VoiceError>;
}

/// Result of a single inference call.
pub struct GenerationResult {
    /// Generated text (stripped of prompt echo).
    pub text: String,
    /// Number of tokens generated.
    pub tokens_generated: u32,
    /// Wall-clock time for generation (excludes prompt processing).
    pub generation_time_ms: u64,
    /// Tokens per second (generation phase only).
    pub tokens_per_sec: f32,
    /// Wall-clock time for prompt processing (prefill).
    pub prefill_time_ms: u64,
}

pub enum VoiceError {
    ModelLoadFailed(String),
    InferenceFailed(String),
    OutOfMemory,
    InvalidModel(String),
}

// prompt.rs

/// A structured prompt payload for the spike test matrix.
/// Parsed from JSON files that Mellanie/Paula/Jeroen prepare.
#[derive(Debug, Deserialize)]
pub struct PromptPayload {
    /// Unique ID for tracking results.
    pub id: String,
    /// Content type being re-voiced.
    pub content_type: ContentType,
    /// The fully assembled prompt string (system + injectors + base text).
    pub prompt: String,
    /// The original base text (for output comparison).
    pub base_text: String,
    /// Expected semantic core (for tell payloads — optional).
    pub semantic_core: Option<String>,
}

#[derive(Debug, Deserialize)]
pub enum ContentType {
    Behavior,
    Tell,
    Dialogue,
}

Batch mode for the test matrix

The --batch flag processes a JSONL file where each line is a PromptPayload JSON object. Output is JSONL with the original payload + generated text + timing:

{"id":"rural-farmer-1","content_type":"Behavior","base_text":"tends crops in the field","output":"works the irrigation channels before morning rotation","tokens_per_sec":8.2,"prefill_ms":340,"generation_ms":3650}
{"id":"nervous-tell-1","content_type":"Tell","base_text":"shifts weight and checks the time without reason","output":"shifts from foot to foot, void-ward glances at the clock","tokens_per_sec":7.9,"prefill_ms":380,"generation_ms":3800}

This enables Mellanie and Paula to prepare prompt files, run them through both models, and compare output side by side. The JSON output feeds directly into a comparison spreadsheet or diff tool.

What Spike 1 does NOT include

• No game integration
• No queue or priority system
• No cache
• No thread pool management
• No prompt construction logic (prompts are hand-crafted by the content team for the spike)
• No save/load of voiced content
• No Godot interaction

---

2. Model Test Plan

Candidates

Model	Parameters	Q4_K_M Size	Why it's here
Gemma 2 2B	2.6B	~1.5 GB	Primary candidate. Google origin. Good instruction following for size.
Phi-3-mini	3.8B	~2.2 GB	Fallback candidate. Microsoft origin. Better quality, larger footprint.

Addressing the Phi-3 size discrepancy: The original proposal called both "2B class." This is incorrect. Phi-3-mini is 3.8B parameters — nearly 50% larger. This matters for:

• RAM: +700MB at Q4 (+47% over Gemma 2B)
• Throughput: ~30% slower decode due to larger weight matrix
• Install size: +700MB in the distribution bundle

Phi-3-mini is the quality fallback, not a peer candidate. If Gemma 2B passes the quality bar, Phi-3 is unnecessary. If Gemma 2B fails, Phi-3 tells us whether more parameters solve the problem or the task itself is wrong for small models.

Test matrix

Prompt payloads (prepared by Mellanie/Paula/Jeroen — Tyre provides the structure):

ID	Content type	Base text	Zone	Culture	Traits	Mood	TellCategory	Notes
B-01	Behavior	"tends crops in the field"	rural	van-maanens-star	Bold, Honest	Neutral	—	Simple role action
B-02	Behavior	"checks a manifest against a handheld scanner, lips moving"	industrial	van-maanens-star	Curious, Social	Neutral	—	Detailed role action
B-03	Behavior	"catches {target}'s eye and nods across the room"	industrial	van-maanens-star	Social, Compassionate	Neutral	—	Relationship behavior — named target preservation
B-04	Behavior	"talks past {target} without making eye contact"	industrial	van-maanens-star	Deceptive, Bold	Neutral	—	Negative relationship — social signal preservation
B-05	Behavior	"sits alone in the break room rubbing the back of her neck, datapad face-down on the table"	industrial	van-maanens-star	—	Stressed	—	Long-form atmospheric behavior
T-01	Tell	"shifts weight and checks the time without reason"	any	van-maanens-star	—	—	Nervous	Nervous fidget — phenomenon must survive
T-02	Tell	"affects exaggerated calm"	any	van-maanens-star	Deceptive	—	Guarded	Suppression — constrained re-voicing test
T-03	Tell	"checks surroundings repeatedly"	any	van-maanens-star	Cautious	—	Nervous	Surveillance — must not become avoidance
D-01	Dialogue	"You need a keycard for that door."	industrial	van-maanens-star	Bold	Neutral	—	Simple informational dialogue
D-02	Dialogue	"I haven't seen Kael since second shift. Why?"	industrial	van-maanens-star	Suspicious, Cautious	Guarded	Guarded	Dialogue with active tell context — tell shapes tone
D-03	Dialogue	"The cargo manifest doesn't match what's in bay seven."	industrial	van-maanens-star	Honest, Curious	Alert	—	Information-bearing dialogue — must preserve factual content

11 payloads total. Each run through both models = 22 outputs per prompt template variant.

Prompt template variants to test

For each payload, test 2-3 prompt template variants to find the optimal instruction format:

Variant 1 — Instruction-first:

[System instruction]
[Culture injector with closed vocabulary]
[Personality + mood]
[Tell context if applicable]
Rephrase: "[base text]"

Variant 2 — Few-shot:

[System instruction]
[Culture injector]

Examples:
Base: "repairs equipment by hand" → Voiced: "strips the housing down and rebuilds it, no manual needed"
Base: "arranges goods on a portable display" → Voiced: "squares the goods on the fold-out, everything where it should be"

[Personality + mood]
[Tell context if applicable]
Rephrase: "[base text]"

Variant 3 — Negative-constraint-heavy (Miri's recommendation):

[System instruction]
[Culture injector]
DO NOT use: military ranks, sir/ma'am, religious references, quips, banter.
DO NOT reference: religion, sports, nationality, Earth-origin social structures.
Technology terms ONLY: insert, span gate, horizon gate, void, the Reach.
Exclamations ONLY: "void take it", "stars", "blood and void", "cold vacuum", "damn all".

[Personality + mood]
Rephrase: "[base text]"

Evaluation criteria

Each output is scored on 5 axes (1-5 scale, scored by Paula and Mellanie independently):

Criterion	What it measures	Pass threshold
Register accuracy	Does it sound like Van Maanen's Star working-class? Not formal, not quippy, not military.	>= 3
Oath preservation	If an exclamation appears, is it from the canonical list? No franchise bleed?	>= 4 (hard requirement)
Semantic preservation	Does the output preserve the action/information from the base text?	>= 4 (hard requirement)
Named target survival	For B-03/B-04: does the {target} name survive in the output?	Pass/Fail
Tell phenomenon class	For T-01/T-02/T-03: does the tell still express the same TellCategory?	Pass/Fail

Spike 1 success criteria:

• At least one model achieves >= 3 average on register accuracy across all payloads
• Both hard requirements (oath, semantic) pass on >= 9/11 payloads
• Named target survival: pass on both relationship payloads
• Tell phenomenon class: pass on all 3 tell payloads

If Gemma 2B meets these criteria, it's the selected model. If only Phi-3 meets them, we accept the RAM/size tradeoff and document why. If neither meets them, the spike has failed and we fall back to base-text-only (Proposal A without re-voicing, which is the current system with scaling managed by content authoring).

Benchmark protocol

On Spike 1 hardware (developer machine), record for each model:

• Tokens/sec at Q4_K_M with --threads set to physical_cores - 1
• RAM usage during inference (peak RSS)
• Model load time from SSD

On a representative minimum-spec machine (if available — otherwise note the hardware used and extrapolate using Troblum's bandwidth formula):

• Same metrics
• Thermal behavior during 5-minute sustained inference run

---

3. Spike 2 Outline: Integration Architecture

Spike 2 wires the validated Spike 1 runner into the game. This outline is ticket-ready, not full implementation design.

3.1 Inference thread pool

┌──────────────────────────────────────────┐
│ Game Process                              │
│                                           │
│  ┌───────────────┐   ┌────────────────┐  │
│  │ Simulation    │   │ Voice Pipeline │  │
│  │ (server)      │   │ (separate pool)│  │
│  │               │   │                │  │
│  │ World gen ────┼──>│ Work queue     │  │
│  │ NPC spawn     │   │ InferenceEngine│  │
│  │ Tick loop     │   │ Voice cache    │  │
│  └───────────────┘   └────────────────┘  │
│                                           │
│  ┌───────────────┐                        │
│  │ Godot client  │ <── reads cache ──┘    │
│  └───────────────┘                        │
└──────────────────────────────────────────┘

Thread count: 1 dedicated inference thread. llama.cpp uses its own internal threading (set to physical_cores - world_gen_threads - 1). The inference thread owns the InferenceEngine instance — no model sharing across threads.

Priority: Below-normal OS thread priority. Inference yields to simulation and rendering.

3.2 Work queue

/// A single unit of work for the voice pipeline.
pub struct VoiceWorkItem {
    /// Cache key for storing the result.
    pub cache_key: VoiceCacheKey,
    /// Priority tier (lower number = higher priority).
    pub priority: VoicePriority,
    /// The fully constructed prompt string.
    pub prompt: String,
    /// Maximum output tokens.
    pub max_tokens: u32,
}

pub enum VoicePriority {
    /// P0: Plot-critical NPCs the player is about to interact with.
    Critical = 0,
    /// P1: NPCs in the current zone the player may interact with.
    High = 1,
    /// P2: NPCs in adjacent/anticipated zones.
    Standard = 2,
    /// P3: Ambient NPCs in distant zones (opportunistic).
    Background = 3,
}

Queue implementation: crossbeam-channel bounded channel (capacity: 256). Items sorted by priority. Producer: the world generation system, triggered by ZonePopulated event. Consumer: the inference thread.

Backpressure: If the queue is full, new items are dropped silently — the game continues with base text. No blocking the simulation thread.

Zone transition pause: When the simulation emits a ZoneTransitionStart event, the inference thread pauses (drains current item, then waits). Resumes on ZoneTransitionComplete. This prevents CPU contention during the loading spike.

3.3 Voice cache

pub struct VoiceCacheKey {
    pub world_seed: u64,
    pub culture_id: String,       // "van-maanens-star"
    pub npc_stable_id: StableId,
    pub content_type: ContentType, // Behavior | Dialogue
    pub content_index: u8,         // which behavior/dialogue line
}

pub struct VoiceCache {
    /// In-memory cache for current session.
    entries: BTreeMap<VoiceCacheKey, String>,
    /// Model identifier used to generate these entries.
    model_id: String,
}

Persistence: Written to user://voice_cache/{seed}.msgpack on zone transition or autosave. Loaded on game start if seed matches. Format version tag for migration.

Invalidation: Full cache invalidation on: seed change, model update (game patch). Per-NPC invalidation on: relationship change that affects behavior text.

Baked content: Hub zone voiced content ships as a game asset at res://voice_baked/{zone_id}.msgpack. Loaded into the cache on zone entry. Never regenerated at runtime.

3.4 Tell-as-context prompt construction

Per Jeroen's decision: tells are passthrough (never re-voiced), but they INFORM the re-voicing prompt for behaviors and dialogue.

/// Build the re-voicing prompt for an NPC's behavior or dialogue.
fn build_prompt(
    base_text: &str,
    content_type: ContentType,
    culture: &CultureProfile,
    npc: &NpcBlueprint,
    active_tell: Option<TellCategory>,  // from DerivedTellState
) -> String {
    let mut prompt = String::with_capacity(512);

    // System instruction
    prompt.push_str(SYSTEM_INSTRUCTION);

    // Culture injector (from NpcBlueprint.cultural_markers — per Miri's recommendation)
    prompt.push_str(&format_culture_injector(&npc.cultural_markers, culture));

    // Personality injector
    prompt.push_str(&format_personality(&npc.traits));

    // Tell-as-context: if the NPC has an active tell, inject it as mood/state context
    // The tell itself is NOT being re-voiced — it's informing the tone
    if let Some(tell) = active_tell {
        prompt.push_str(&format_tell_context(tell));
        // e.g., "The character is currently guarded and evasive.
        //        Their dialogue should reflect this state without stating it directly."
    }

    // Negative constraints (franchise bleed prevention)
    prompt.push_str(NEGATIVE_CONSTRAINTS);

    // Base text to re-voice
    match content_type {
        ContentType::Behavior => {
            prompt.push_str(&format!("\nRephrase this action: \"{}\"", base_text));
        }
        ContentType::Dialogue => {
            prompt.push_str(&format!("\nRephrase this dialogue line: \"{}\"", base_text));
        }
    }

    prompt
}

fn format_tell_context(tell: TellCategory) -> String {
    match tell {
        TellCategory::Nervous => {
            "\nState: The character is anxious. Their speech is clipped, distracted.\n".into()
        }
        TellCategory::Angry => {
            "\nState: The character is angry. Their speech is terse, barely controlled.\n".into()
        }
        TellCategory::Friendly => {
            "\nState: The character is warm and open. Their speech is relaxed.\n".into()
        }
        TellCategory::Guarded => {
            "\nState: The character is guarded. They deflect and keep things vague.\n".into()
        }
        TellCategory::RoutineDeviation => {
            "\nState: The character is preoccupied. Something else is on their mind.\n".into()
        }
    }
}

This is the critical design: the TellCategory enum flows into the prompt as a mood/state modifier, not as content to re-voice. The tell behavior string stays untouched. The dialogue and ambient behaviors around the tell are colored by the NPC's state.

3.5 Observer integration

The observer snapshot system already reads DerivedTellState and observable_behaviors. Integration point:

Observer reads NPC behavior string:
  1. Check voice cache for (seed, culture, npc_id, behavior_index)
  2. If cache hit → use voiced string
  3. If cache miss → use base text string (fallback)
  4. Tell state → always from DerivedTellState (passthrough, never from cache)

No changes to the wire format (ObserverSnapshot). The client doesn't know or care whether the behavior string was voiced or base text.

3.6 Baked content generation

A build-time step that runs the inference engine on all hub zone NPCs:

# Build tool (not the game binary)
sr-voice-bake \
  --model models/gemma-2b-q4.gguf \
  --zones content/global/van-maanens-star-*.ron \
  --culture content/global/culture-van-maanens-star.ron \
  --output client/assets/voice_baked/ \
  --seed 0  # baked content uses seed 0 as the canonical reference

Output: one .msgpack file per zone containing all voiced behavior and dialogue strings. Checked into the repository (text-only, compresses to ~20-50KB per zone). Human-reviewed by Paula/Mellanie before ship.

---

4. Hardware Detection Design

Layer 1: RAM check (can the model load?)

On first toggle of "AI-Enhanced Dialogue":

fn check_ram_available() -> RamCheckResult {
    let available_mb = get_available_system_ram_mb();
    let model_size_mb = 1600; // Gemma 2B Q4 + KV cache overhead

    if available_mb < model_size_mb {
        RamCheckResult::InsufficientRam {
            available_mb,
            required_mb: model_size_mb,
        }
    } else {
        RamCheckResult::Ok
    }
}

User-facing message if insufficient:

"AI-Enhanced Dialogue requires approximately 1.6 GB of free RAM. Your system currently has {available_mb} MB available. The feature may cause instability. Enable anyway?"

Player can always override. No hard block.

Layer 2: Time-per-token benchmark (is inference useful?)

If RAM check passes, run a 5-token benchmark on first enable:

fn benchmark_inference(engine: &mut InferenceEngine) -> BenchmarkResult {
    let test_prompt = "Rephrase: \"walks down the corridor.\"";
    let result = engine.generate(test_prompt, 5, 0.7, 0.9)?;
    let tpt_ms = result.generation_time_ms as f32 / result.tokens_generated as f32;

    BenchmarkResult {
        tokens_per_sec: result.tokens_per_sec,
        time_per_token_ms: tpt_ms,
    }
}

Thresholds:

Tokens/sec	Recommendation	User message
>= 5 t/s	Full enable	"AI-Enhanced Dialogue is active."
2-5 t/s	Enable with warning	"AI-Enhanced Dialogue is active. On your hardware, voiced content will generate slowly. Some NPCs may show plain text until generation catches up."
< 2 t/s	Recommend disable	"Your hardware generates voiced content very slowly. We recommend disabling AI-Enhanced Dialogue for the best experience. Enable anyway?"

No hard floor. Player can always choose to run it. The benchmark runs once, result is cached in user settings. Player can re-run from the settings menu.

Layer 3: Runtime monitoring

During gameplay, the inference thread monitors its own throughput:

// In the inference thread main loop:
if current_tokens_per_sec < 1.0 {
    // Sustained very-slow inference — likely thermal throttle or power saver
    pause_inference();
    notify_ui("AI dialogue generation paused — system is running slowly.");
    // Resume after 60 seconds or on user action
}

Battery/power-saver detection: On Windows, check GetSystemPowerStatus(). If on battery with power saver active, auto-pause inference and show notification. On Linux/macOS, check /sys/class/power_supply/ or equivalent. Resume when plugged in or power mode changes.

Settings UI

[Settings > Audio & Dialogue]

AI-Enhanced Dialogue: [ON / OFF]
  Status: Active (8.2 tokens/sec)

  [Re-run benchmark]

  Note: When enabled, NPC dialogue and behaviors are enhanced with
  culture-specific voice. This uses additional CPU resources.
  Disable if you experience performance issues.

---

Effort Estimates

Work item	Sprints	Dependencies
Spike 1: sr-voice CLI tool	1	None — can start immediately
Spike 1: Prompt crafting + model testing	1	sr-voice CLI (Mellanie/Paula/Jeroen run the tests)
Spike 2: Queue + cache + thread pool	1.5	Spike 1 model selection
Spike 2: Tell-as-context prompt construction	0.5	Queue infrastructure
Spike 2: Observer integration	0.5	Cache system
Spike 2: Baked content generation tool	0.5	Queue + cache
Hardware detection system	0.5	InferenceEngine (from Spike 1)
Total	5.5	Spike 1 and 2 are sequential; sub-items within each spike are partially parallelizable

Spike 1 can start next sprint. The CLI tool is self-contained Rust with no game dependencies. While the content team runs manual prompt tests, Spike 2 infrastructure design can begin in parallel.