From 67bee80dc8423d7baa12c5790d8c748ef5a0796d Mon Sep 17 00:00:00 2001 From: Jeroen Schweitzer Date: Fri, 7 Aug 2026 15:05:15 +0200 Subject: [PATCH] docs(architecture): sync with measured 2026-08-07 state Every figure in the latency budget was stale, in both directions. TTS was listed at ~1.9 s per sentence but measures ~0.24 s warm for 4.5 s of audio; the full Tatlock flow was listed at 11-25 s but measures ~10-13 s for simple turns. Both sets of numbers predate the current model. The VRAM section now carries real figures and the reason they matter: on 2026-08-07 Tatlock ran against a 9.3 GB model, leaving 7 MiB free, and every transcription failed with CUDA out of memory while the Speaches container still reported healthy. The budget is the constraint, not slack. Also replaces the retired tatlock.schweitz.internal hostname in the topology diagram with the docker container name. Co-Authored-By: Claude --- docs/architecture.md | 51 ++++++++++++++++++++++++++------------------ 1 file changed, 30 insertions(+), 21 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 01c2191..7ae494d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -24,8 +24,8 @@ tower-of-joy. Everything is local — no audio, transcript, or reply ever leaves ▼ │ Speaches (container, GPU) │ ┌────────────────────┐ │ • STT: faster-whisper │ │ Tatlock (butler) │ │ • TTS: Kokoro / Piper │ - │ tatlock.schweitz. │ │ also usable by Open WebUI, │ - │ internal :8000 │ │ Home Assistant, … │ + │ container name: │ │ also usable by Open WebUI, │ + │ tatlock:8000 │ │ Home Assistant, … │ └────────────────────┘ └─────────────────────────────┘ ``` @@ -103,19 +103,25 @@ we may adopt later for streaming transcription. extension, verified live; default voice `bm_george`, en-GB male). LAN-only like the Tatlock internal route — do not expose through NPM without auth. Register in `CONTAINERS.md`. -- **Measured** (live round trip through the gateway code, warm): STT ~0.3 s for a - ~3 s utterance; TTS ~1.9 s for a ~3 s sentence. Cold start after model TTL offload - adds ~5–10 s to the first request. +- **Measured** (live round trip, warm, 2026-08-07): STT ~0.30 s for a ~4.8 s utterance; + TTS ~0.24 s for a ~4.5 s sentence (real-time factor ~0.05). The first call after an + idle gap costs ~1.2 s; a full cold start after model TTL offload adds ~4 s. - **Why a shared layer instead of models inside the gateway**: one GPU-resident model instance serves the whole homelab. Open WebUI is currently configured with `AUDIO_STT_ENGINE=openai` / `AUDIO_TTS_ENGINE=openai` (OpenAI *cloud*) — pointing its audio base URL at Speaches makes it fully local with a config change. Home Assistant can share it too. Meanwhile the gateway image needs no CUDA and rebuilds in seconds. -- **VRAM budget**: RTX 2080 Ti, 11 GB, shared with Ollama (~3.6 GB in use as of - 2026-07). whisper `small` at int8 is <1 GB; Kokoro is a few hundred MB. Speaches' - model TTL offload keeps idle pressure near zero. If VRAM contention ever bites, - faster-whisper `small` on CPU is an acceptable fallback (int8, a few seconds per - utterance). +- **VRAM budget**: RTX 2080 Ti, 11,264 MiB, shared with Ollama. As of 2026-08-07 the + steady state is ~4.9 GB used / ~5.9 GB free with everything resident: `gemma4:e2b` + 1.9 GB and `nomic-embed-text` 0.3 GB (both pinned), whisper `small` int8 <1 GB, + Kokoro a few hundred MB. Speaches' model TTL offload keeps idle pressure near zero. + **This budget is not slack — it is the constraint.** On 2026-08-07 Tatlock was + deployed against `mistral-nemo:latest` (9.3 GB, 2 h keep-alive), which left 7 MiB + free and made every transcription fail with `CUDA failed with error out of memory` + while the Speaches container still reported healthy. Keep Tatlock's model at or below + ~4 GB resident, and check `nvidia-smi` free VRAM before changing it. If contention + ever bites anyway, faster-whisper `small` on CPU is an acceptable fallback (int8, a + few seconds per utterance). ### 4. Tatlock — existing backend (`/mnt/media/Projects/tatlock`) @@ -220,19 +226,22 @@ it in phase 5. ## Latency budget & streaming -Measured/known numbers that shape the design (Tatlock figures per tatlock CLAUDE.md, -GPU-resident benchmarks of 2026-07-14, gemma4:e2b at ~100 tok/s): +Measured 2026-08-07 against the deployed stack (`gemma4:e2b` at ~95 tok/s, GPU-resident): | Stage | Cost | |-------|------| -| STT (Speaches whisper `small`) | ~0.3 s warm (measured) | -| TTS (Speaches Kokoro) | ~1.9 s per ~3 s sentence, warm (measured) | -| Tatlock Steward analysis | ~6 s warm | -| **Tatlock, full local flow** | **11–25 s end-to-end** (librarian-routed ~20–25 s) | -| Tatlock cold start (>2 h idle) | +~8 s (`OLLAMA_KEEP_ALIVE=2h`) | +| STT (Speaches whisper `small`) | ~0.30 s warm, for ~4.8 s of audio | +| TTS (Speaches Kokoro) | ~0.24 s warm, for ~4.5 s of audio (RTF ~0.05) | +| **Tatlock, full local flow** | **~10–13 s end-to-end** for simple turns | +| Tatlock cold model load | +~36 s — avoided while the model is pinned | -(Older "~35 s Steward / ~2 min flow" figures were from a CPU-only driver-mismatch era — -do not plan against them.) +A Tatlock turn costs **3 sequential Ollama calls** (Steward routing → tool orchestration → +butler-tone synthesis) and ~710 generated tokens even for "what is 61 plus 12?". Most of +that is the model's own reasoning: gemma4 thinks by default, and the effort is spent three +times per turn. + +(Older figures — "~35 s Steward / ~2 min flow" from the CPU-only era, and "11–25 s full +flow" from 2026-07-14 — are superseded. Do not plan against them.) Speech is not the bottleneck — **Tatlock is**, by one to two orders of magnitude. Constraints this imposes: @@ -241,11 +250,11 @@ Constraints this imposes: sentence-by-sentence**, forwarding audio as each sentence is ready. The device starts speaking after the first sentence instead of waiting for the full reply — with streaming, first audio should land roughly at Steward-time + first-sentence-time, - well under the 11–25 s full-flow figure. The WS protocol already supports this: one + well under the ~10–13 s full-flow figure. The WS protocol already supports this: one `audio_start` … PCM … `audio_end` envelope with chunks arriving as they're synthesized — the device just plays a continuous stream. 2. **The `thinking` face state is a first-class feature**, not decoration — it's what - makes a 10–25 s Tatlock turn feel intentional instead of broken. Consider progress + makes a ~10 s Tatlock turn feel intentional instead of broken. Consider progress cues (e.g. surface Tatlock's reasoning summaries on-screen) later. 3. A **fast lane** may eventually be needed: MultiNet on-device commands for instant home-automation phrases, and/or a low-latency intent path in Tatlock itself. Out of