settled-reach/docs/workshops/llm-voice-pipeline/troblum-round2.md

title, description, type, status, workshop, agent, round, created

title	description	type	status	workshop	agent	round	created
Troblum Round 2: Proposal Evaluation and Infrastructure Answers	Infrastructure evaluation of throughput, token budgets, and hardware detection	workshop	archived	llm-voice-pipeline	troblum	2	2026-03-07

Troblum — Round 2: Proposal Evaluation & Infrastructure Answers

Workshop: LLM Voice Pipeline Domain: Infrastructure / Performance Round: 2 — Convergent Evaluation Date: 2026-03-07

---

Q-R1-04: Throughput Impact of 150-token vs. 500-token Prompts

The question is precise; the answer requires splitting inference into its two phases.

Phase 1: Prefill (prompt processing)

Prefill is parallel — all prompt tokens processed simultaneously across available CPU cores. On 4-6 core AVX2 hardware from 2019+, prefill throughput for a 2B Q4 model runs approximately 300–600 tokens/sec.

Prompt length	Prefill time (i5-9400 class)
150 tokens	0.25–0.5 seconds
500 tokens	0.85–1.7 seconds

The prefill penalty for 500-token vs. 150-token prompts: approximately 0.6–1.2 seconds per call.

Phase 2: Generation (decode)

Generation is sequential — one token at a time, memory-bandwidth limited. On the i5-9400 class at Gemma 2B Q4_K_M, decode runs 7–9 tokens/sec (see Q-R1-05 below for full derivation). Prompt length does not affect decode speed — only output length matters.

Output length	Generation time @ 8 t/s
30 tokens (behavior line)	3.75 seconds
50 tokens (dialogue line)	6.25 seconds

Total per-task comparison

Task type	Prompt	Output	Total @ i5-9400
Behavior (Proposal A/B)	150 t	30 t	4.0–4.5 seconds
Dialogue (Proposal C)	500 t	50 t	7.1–8.0 seconds

Dialogue tasks take approximately 1.8–2× longer than behavior tasks. The multiplier is driven mostly by longer output (50 vs. 30 tokens) with the longer prompt adding a secondary fixed cost per call.

Impact on zone pre-voicing time

Proposal A (behaviors only, i5-9400, 8 t/s average):

• Rural zone: 4 NPCs × 3 behaviors × 4.2s = ~50 seconds
• Industrial zone: 10 NPCs × 3 behaviors × 4.2s = ~2.1 minutes

Proposal C (behaviors + dialogue, same hardware, behaviors at 4.2s, dialogue at 7.5s):

• Assume 3 behavior lines + 4 dialogue lines per NPC
• Rural zone: 4 NPCs × (3 × 4.2s + 4 × 7.5s) = 4 × (12.6 + 30) = ~170 seconds (~2.8 min)
• Industrial zone: 10 NPCs × 42.6s = ~7.1 minutes

The industrial zone at 7 minutes is at the edge of comfortable for background pre-voicing. If the player spends at least 10 minutes in a zone (likely for plot-critical locations), pre-voicing completes before meaningful NPC interaction. For transit zones the player moves through quickly, it will not catch up — base text fallback will be visible.

This is a workable design IF the queue prioritizes by interaction likelihood, not just plot criticality. If the player sprints through an industrial zone to reach a specific NPC, that NPC's lines must be in the P0 queue. P2 ambient NPCs in the same zone can remain unvoiced without player impact.

---

Q-R1-05: Is 4-Core 2019+ CPU Sufficient for Gemma 2B Q4?

Short answer: Yes. With specific numbers.

Derivation for i5-9400

The bottleneck for llama.cpp decode on CPU is memory bandwidth. Each decode step reads the full set of model weights.

• Gemma 2B Q4_K_M weight size on-disk and in-RAM: ~1.5 GB
• Intel i5-9400 memory bandwidth: DDR4-2666 dual-channel ≈ 42 GB/s
• Theoretical tokens/sec: 42 GB/s ÷ 1.5 GB = 28 tokens/sec (theoretical ceiling)
• Real-world efficiency factor (cache pressure, OS overhead, threading): ~25–35%
• Estimated real-world decode speed: 7–10 tokens/sec

Derivation for Ryzen 5 3600

• Memory bandwidth: DDR4-3200 dual-channel ≈ 51 GB/s
• Theoretical ceiling: 51 ÷ 1.5 = 34 tokens/sec
• Same efficiency factor: ~8–12 tokens/sec

Additionally: the Ryzen 5 3600 has a 32 MB L3 cache. For a 1.5 GB model, L3 caching has minimal impact on decode (model weights far exceed L3 capacity). The bandwidth advantage is the real differentiator.

Working estimates: i5-9400: 7–9 t/s. Ryzen 5 3600: 9–12 t/s.

Is this sufficient?

Yes. At 8 t/s (midpoint for i5-9400):

Zone	NPCs	Pre-voice time (behaviors only)	Pre-voice time (behaviors + dialogue)
Rural (density 2)	2–4	25–50 sec	85–170 sec
Industrial (density 6)	6–12	75–150 sec	250–510 sec

A player spending 3+ minutes in any zone will have behavior pre-voicing complete before meaningful NPC interaction. This is a comfortable margin for immersive-sim play patterns.

One caveat: these are desktop CPUs. The i5-9400 is a 65W chip with no power-management constraints in normal operation. If the minimum-spec assumption includes laptops with throttled performance (45W TDP, thermal limits), effective decode speed drops to 4–6 t/s. At 4 t/s, industrial zone pre-voicing (behaviors only) takes 5 minutes — still workable but tight. This is the scenario where thermal monitoring in the queue scheduler becomes mandatory, not optional.

---

Proposal C RAM Concern: Qwen2.5-3B at Q4

Single-model scenario

If Proposal C uses only one model (Gemma 2B for both behaviors and dialogue):

• Model RAM: 1.5 GB
• Game + OS: 1.7–2.65 GB
• Total peak: 3.2–4.15 GB
• Headroom on 8 GB: 3.85–4.8 GB — No concern.

Dual-model scenario (Gemma 2B for behaviors + Qwen2.5-3B for dialogue)

If the spike shows 2B quality is insufficient for dialogue and 3B is required, the RAM calculation depends on loading strategy:

Simultaneous loading (both models in RAM at once):

• 1.5 GB (Gemma 2B Q4) + 2.0 GB (Qwen2.5-3B Q4) = 3.5 GB total for models
• Plus game + OS: 1.7–2.65 GB
• Total peak: 5.2–6.15 GB
• Headroom: 1.85–2.8 GB

This headroom is tight. During zone transitions with active world generation AND both models loaded, memory spikes could push into swap territory on minimum-spec machines. Not safe.

Sequential model loading (swap strategy — recommended):

• Only one model loaded at a time
• Batch behavior tasks → load Gemma 2B → process → unload → load Qwen2.5-3B → process dialogue tasks → unload
• Peak RAM at any time: 2.0 GB (larger model) + 2.65 GB (game) = 4.65 GB
• Headroom: 3.35 GB — comfortable.

Model swap latency: loading Gemma 2B Q4 from SSD takes 2–4 seconds. Qwen2.5-3B Q4: 3–6 seconds. If batching 20+ tasks per model-load cycle (which is realistic for a full zone), swap overhead is 5–10 seconds amortized over the batch — acceptable.

Conclusion: Qwen2.5-3B Q4 fits on 8GB alongside the game IF the queue implements sequential loading with batching. Simultaneous loading of both models is not safe on minimum spec. The queue scheduler must enforce single-model-at-a-time.

The Qwen constraint flag

The original proposed-llm-voice.md document states the project constraint: "no Meta/Chinese models." Qwen2.5-3B is by Alibaba (Chinese company). This appears to conflict with that constraint.

I don't know if this constraint has been formally dropped or if it's a oversight in the Round 2 proposals. Before committing to a Qwen2.5-3B dependency, someone needs to confirm whether the "no Chinese models" constraint still applies. If it does, the fallback for Proposal C's dialogue model is a non-Chinese 3B alternative — Phi-3-mini at 3.8B is the next candidate (though it's larger and slower), or the spike might demonstrate Gemma 2B Q4 is sufficient for dialogue after all.

Flagging this to the team. I'm not making the constraint decision; I'm noting the conflict.

---

Install Size: Acceptable? Optional Download?

Numbers by proposal

Proposal	Model(s)	Total model download
A or B	Gemma 2B Q4_K_M	~1.5 GB
C (single model)	Gemma 2B Q4_K_M	~1.5 GB
C (dual model)	Gemma 2B + Qwen2.5-3B Q4	~3.5 GB

Plus inference runtime: ~20–25 MB. Negligible.

For context, typical indie game install sizes are 2–8 GB. Adding 1.5 GB is a 20–75% install size increase depending on the base game. Adding 3.5 GB for dual-model Proposal C is potentially larger than the base game itself.

My position: optional download

Bundling the model in the base installer is the cleanest player experience but creates distribution problems:

1. itch.io: 2 GB per-file upload limit. A 1.5 GB base game + 1.5 GB model in one package exceeds this. Even split across two files, dual-model Proposal C (3.5 GB) is problematic.
2. Steam: No hard size limit, but the initial download perception matters. Players who don't plan to use "AI-Enhanced Dialogue" are paying the bandwidth cost involuntarily.
3. GoG, Epic: Similar concerns. Large downloads increase refund friction.

Recommended approach: make the model an optional in-game download, triggered when the player first enables "AI-Enhanced Dialogue."

Implementation: the game ships with base text fully functional. On feature enable, a download prompt: "AI-Enhanced Dialogue requires downloading a 1.5 GB language model. Download now?" Single download, stored in user data directory. The download is from the game's own servers (not cloud AI services) — this preserves the "no accounts, no cloud" guarantee from the proposal.

Pros: base install stays at game-only size, model download is opt-in, works on all distribution platforms.

Cons: first-time enable has friction (download wait). This is acceptable. The feature is a toggle, not a core gameplay requirement.

For baked hub content (pre-voiced at build time): these voiced lines ship as game data, not requiring the model. The player's first hours are already pre-voiced without any model download. The model download only matters for background generation of visited zones. This actually softens the first-enable friction further: hub zones work immediately, the download runs in the background for everything else.

---

Resolution Matrix

Which proposal do I recommend?

Proposal A (Conservative — Behaviors Only, Tells Locked).

Infrastructure reasoning:

• Single short-form prompt template (150 tokens) → predictable throughput, easy to benchmark
• Single model (Gemma 2B Q4_K_M) → no dual-model queue complexity, no model-swap scheduler, simpler memory management
• Tell passthrough is the safest mechanism from an infrastructure standpoint — zero risk of cache corruption from subtle tell corruption
• 1.5 GB install delta (or optional download) is the best case for distribution
• Scope is well-defined enough to write a deterministic spike with measurable pass/fail criteria

Proposal A does not solve dialogue scaling. That is a known limitation and an acceptable deferred problem. Prove the behavior pipeline first.

Can I live with Proposal B?

Yes, with one note.

Proposal B's constrained re-voicing (semantic core preservation) adds no infrastructure complexity. The semantic_core constraint is 20 additional tokens in the prompt — negligible throughput impact (~0.05 seconds per task). Two prompt templates are trivially maintained.

My note: the quality validation for constrained re-voicing is harder to automate than for passthrough. "Did the model preserve avoidance_behavior?" requires either human eval or a second LLM classifier. The infrastructure for build-time validation of baked content needs this: the spike design should include a validation pass that flags tells where the semantic core may not have survived. This is buildable but needs explicit scope in the spike plan.

Can I live with Proposal C?

Yes, conditionally.

Conditions:

1. The Qwen2.5-3B "no Chinese models" constraint conflict must be resolved before model selection. If Qwen is out, an alternative 3B model must be identified. Phi-3-mini at 3.8B is slower (worse on minimum spec) and larger (2.2 GB Q4).
2. If dual-model is adopted, the queue scheduler MUST implement sequential loading (swap strategy). Simultaneous model loading is not safe on minimum spec.
3. Install size must be addressed. Dual-model Proposal C at 3.5 GB additional is too large for bundle-in-base-install. Optional download required.
4. The dialogue spike (longer prompts, 50-token output, more complex constraints) must be scoped as a separate validation from the behavior spike. Combining both in a single spike plan risks making the success criteria too broad to evaluate cleanly.

The infrastructure complexity of Proposal C is manageable — it's more complexity, not prohibitive complexity. The queue scheduler is the only genuinely hard part.

Minimum changes to make non-preferred proposals acceptable

If Proposal B is chosen over A: Add an automated validation pass for semantic core preservation in the spike. At minimum, human spot-check of 10% of cached tells per culture. Define "corrupted tell" precisely before the spike, not after.

If Proposal C is chosen over A: (1) Resolve Qwen constraint. (2) Commit to optional download model delivery. (3) Implement sequential model swap in queue scheduler. (4) Run the dialogue spike separately from the behavior spike — separate prompts, separate models, separate success criteria.

---

Summary

Question	Answer
Q-R1-04: Throughput difference, 150t vs. 500t prompts	~1.8–2× longer per task. Behavior: 4–4.5s. Dialogue: 7–8s. Industrial zone: 2 min (A) vs. 7 min (C) on i5-9400.
Q-R1-05: i5-9400 / Ryzen 5 3600 sufficient?	Yes. 7–10 t/s decode for Gemma 2B Q4. Zone pre-voicing (behaviors) completes in 50s–2min. Comfortable margin for immersive-sim play patterns. Thermal throttling is the risk on laptops.
Qwen2.5-3B RAM on 8GB	Yes, fits with sequential model swap (3.35 GB headroom). Simultaneous loading is unsafe (1.85 GB headroom — too tight).
Install size: bundle or optional?	Optional download. 1.5 GB (single model) or 3.5 GB (dual) is too large to bundle invisibly. Trigger on first AI-Enhanced Dialogue enable. Hub baked content ships in game data — player's first hours unaffected.
Recommended proposal	A (behaviors only, tells locked, single model).

---

Troblum, 2026-03-07