diff --git a/docs/architecture.md b/docs/architecture.md index 01c2191..7ae494d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -24,8 +24,8 @@ tower-of-joy. Everything is local — no audio, transcript, or reply ever leaves ▼ │ Speaches (container, GPU) │ ┌────────────────────┐ │ • STT: faster-whisper │ │ Tatlock (butler) │ │ • TTS: Kokoro / Piper │ - │ tatlock.schweitz. │ │ also usable by Open WebUI, │ - │ internal :8000 │ │ Home Assistant, … │ + │ container name: │ │ also usable by Open WebUI, │ + │ tatlock:8000 │ │ Home Assistant, … │ └────────────────────┘ └─────────────────────────────┘ ``` @@ -103,19 +103,25 @@ we may adopt later for streaming transcription. extension, verified live; default voice `bm_george`, en-GB male). LAN-only like the Tatlock internal route — do not expose through NPM without auth. Register in `CONTAINERS.md`. -- **Measured** (live round trip through the gateway code, warm): STT ~0.3 s for a - ~3 s utterance; TTS ~1.9 s for a ~3 s sentence. Cold start after model TTL offload - adds ~5–10 s to the first request. +- **Measured** (live round trip, warm, 2026-08-07): STT ~0.30 s for a ~4.8 s utterance; + TTS ~0.24 s for a ~4.5 s sentence (real-time factor ~0.05). The first call after an + idle gap costs ~1.2 s; a full cold start after model TTL offload adds ~4 s. - **Why a shared layer instead of models inside the gateway**: one GPU-resident model instance serves the whole homelab. Open WebUI is currently configured with `AUDIO_STT_ENGINE=openai` / `AUDIO_TTS_ENGINE=openai` (OpenAI *cloud*) — pointing its audio base URL at Speaches makes it fully local with a config change. Home Assistant can share it too. Meanwhile the gateway image needs no CUDA and rebuilds in seconds. -- **VRAM budget**: RTX 2080 Ti, 11 GB, shared with Ollama (~3.6 GB in use as of - 2026-07). whisper `small` at int8 is <1 GB; Kokoro is a few hundred MB. Speaches' - model TTL offload keeps idle pressure near zero. If VRAM contention ever bites, - faster-whisper `small` on CPU is an acceptable fallback (int8, a few seconds per - utterance). +- **VRAM budget**: RTX 2080 Ti, 11,264 MiB, shared with Ollama. As of 2026-08-07 the + steady state is ~4.9 GB used / ~5.9 GB free with everything resident: `gemma4:e2b` + 1.9 GB and `nomic-embed-text` 0.3 GB (both pinned), whisper `small` int8 <1 GB, + Kokoro a few hundred MB. Speaches' model TTL offload keeps idle pressure near zero. + **This budget is not slack — it is the constraint.** On 2026-08-07 Tatlock was + deployed against `mistral-nemo:latest` (9.3 GB, 2 h keep-alive), which left 7 MiB + free and made every transcription fail with `CUDA failed with error out of memory` + while the Speaches container still reported healthy. Keep Tatlock's model at or below + ~4 GB resident, and check `nvidia-smi` free VRAM before changing it. If contention + ever bites anyway, faster-whisper `small` on CPU is an acceptable fallback (int8, a + few seconds per utterance). ### 4. Tatlock — existing backend (`/mnt/media/Projects/tatlock`) @@ -220,19 +226,22 @@ it in phase 5. ## Latency budget & streaming -Measured/known numbers that shape the design (Tatlock figures per tatlock CLAUDE.md, -GPU-resident benchmarks of 2026-07-14, gemma4:e2b at ~100 tok/s): +Measured 2026-08-07 against the deployed stack (`gemma4:e2b` at ~95 tok/s, GPU-resident): | Stage | Cost | |-------|------| -| STT (Speaches whisper `small`) | ~0.3 s warm (measured) | -| TTS (Speaches Kokoro) | ~1.9 s per ~3 s sentence, warm (measured) | -| Tatlock Steward analysis | ~6 s warm | -| **Tatlock, full local flow** | **11–25 s end-to-end** (librarian-routed ~20–25 s) | -| Tatlock cold start (>2 h idle) | +~8 s (`OLLAMA_KEEP_ALIVE=2h`) | +| STT (Speaches whisper `small`) | ~0.30 s warm, for ~4.8 s of audio | +| TTS (Speaches Kokoro) | ~0.24 s warm, for ~4.5 s of audio (RTF ~0.05) | +| **Tatlock, full local flow** | **~10–13 s end-to-end** for simple turns | +| Tatlock cold model load | +~36 s — avoided while the model is pinned | -(Older "~35 s Steward / ~2 min flow" figures were from a CPU-only driver-mismatch era — -do not plan against them.) +A Tatlock turn costs **3 sequential Ollama calls** (Steward routing → tool orchestration → +butler-tone synthesis) and ~710 generated tokens even for "what is 61 plus 12?". Most of +that is the model's own reasoning: gemma4 thinks by default, and the effort is spent three +times per turn. + +(Older figures — "~35 s Steward / ~2 min flow" from the CPU-only era, and "11–25 s full +flow" from 2026-07-14 — are superseded. Do not plan against them.) Speech is not the bottleneck — **Tatlock is**, by one to two orders of magnitude. Constraints this imposes: @@ -241,11 +250,11 @@ Constraints this imposes: sentence-by-sentence**, forwarding audio as each sentence is ready. The device starts speaking after the first sentence instead of waiting for the full reply — with streaming, first audio should land roughly at Steward-time + first-sentence-time, - well under the 11–25 s full-flow figure. The WS protocol already supports this: one + well under the ~10–13 s full-flow figure. The WS protocol already supports this: one `audio_start` … PCM … `audio_end` envelope with chunks arriving as they're synthesized — the device just plays a continuous stream. 2. **The `thinking` face state is a first-class feature**, not decoration — it's what - makes a 10–25 s Tatlock turn feel intentional instead of broken. Consider progress + makes a ~10 s Tatlock turn feel intentional instead of broken. Consider progress cues (e.g. surface Tatlock's reasoning summaries on-screen) later. 3. A **fast lane** may eventually be needed: MultiNet on-device commands for instant home-automation phrases, and/or a low-latency intent path in Tatlock itself. Out of