docs(architecture): sync with measured 2026-08-07 state
Test, Build and Push / test-gateway (push) Successful in 37s
Test, Build and Push / release (push) Skipped
Test, Build and Push / build-gateway (push) Skipped

Every figure in the latency budget was stale, in both directions. TTS was
listed at ~1.9 s per sentence but measures ~0.24 s warm for 4.5 s of audio;
the full Tatlock flow was listed at 11-25 s but measures ~10-13 s for simple
turns. Both sets of numbers predate the current model.

The VRAM section now carries real figures and the reason they matter: on
2026-08-07 Tatlock ran against a 9.3 GB model, leaving 7 MiB free, and every
transcription failed with CUDA out of memory while the Speaches container
still reported healthy. The budget is the constraint, not slack.

Also replaces the retired tatlock.schweitz.internal hostname in the topology
diagram with the docker container name.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-08-07 15:05:15 +02:00
co-authored by Claude
parent 7c7ce1541c
commit 67bee80dc8
+30 -21
View File
@@ -24,8 +24,8 @@ tower-of-joy. Everything is local — no audio, transcript, or reply ever leaves
▼ │ Speaches (container, GPU) │
┌────────────────────┐ │ • STT: faster-whisper │
│ Tatlock (butler) │ │ • TTS: Kokoro / Piper │
tatlock.schweitz. │ │ also usable by Open WebUI, │
internal :8000 │ │ Home Assistant, … │
container name: │ │ also usable by Open WebUI, │
tatlock:8000 │ │ Home Assistant, … │
└────────────────────┘ └─────────────────────────────┘
```
@@ -103,19 +103,25 @@ we may adopt later for streaming transcription.
extension, verified live; default voice `bm_george`, en-GB male). LAN-only like the
Tatlock internal route — do not expose through NPM without auth. Register in
`CONTAINERS.md`.
- **Measured** (live round trip through the gateway code, warm): STT ~0.3 s for a
~3 s utterance; TTS ~1.9 s for a ~3 s sentence. Cold start after model TTL offload
adds ~510 s to the first request.
- **Measured** (live round trip, warm, 2026-08-07): STT ~0.30 s for a ~4.8 s utterance;
TTS ~0.24 s for a ~4.5 s sentence (real-time factor ~0.05). The first call after an
idle gap costs ~1.2 s; a full cold start after model TTL offload adds ~4 s.
- **Why a shared layer instead of models inside the gateway**: one GPU-resident model
instance serves the whole homelab. Open WebUI is currently configured with
`AUDIO_STT_ENGINE=openai` / `AUDIO_TTS_ENGINE=openai` (OpenAI *cloud*) — pointing its
audio base URL at Speaches makes it fully local with a config change. Home Assistant
can share it too. Meanwhile the gateway image needs no CUDA and rebuilds in seconds.
- **VRAM budget**: RTX 2080 Ti, 11 GB, shared with Ollama (~3.6 GB in use as of
2026-07). whisper `small` at int8 is <1 GB; Kokoro is a few hundred MB. Speaches'
model TTL offload keeps idle pressure near zero. If VRAM contention ever bites,
faster-whisper `small` on CPU is an acceptable fallback (int8, a few seconds per
utterance).
- **VRAM budget**: RTX 2080 Ti, 11,264 MiB, shared with Ollama. As of 2026-08-07 the
steady state is ~4.9 GB used / ~5.9 GB free with everything resident: `gemma4:e2b`
1.9 GB and `nomic-embed-text` 0.3 GB (both pinned), whisper `small` int8 <1 GB,
Kokoro a few hundred MB. Speaches' model TTL offload keeps idle pressure near zero.
**This budget is not slack — it is the constraint.** On 2026-08-07 Tatlock was
deployed against `mistral-nemo:latest` (9.3 GB, 2 h keep-alive), which left 7 MiB
free and made every transcription fail with `CUDA failed with error out of memory`
while the Speaches container still reported healthy. Keep Tatlock's model at or below
~4 GB resident, and check `nvidia-smi` free VRAM before changing it. If contention
ever bites anyway, faster-whisper `small` on CPU is an acceptable fallback (int8, a
few seconds per utterance).
### 4. Tatlock — existing backend (`/mnt/media/Projects/tatlock`)
@@ -220,19 +226,22 @@ it in phase 5.
## Latency budget & streaming
Measured/known numbers that shape the design (Tatlock figures per tatlock CLAUDE.md,
GPU-resident benchmarks of 2026-07-14, gemma4:e2b at ~100 tok/s):
Measured 2026-08-07 against the deployed stack (`gemma4:e2b` at ~95 tok/s, GPU-resident):
| Stage | Cost |
|-------|------|
| STT (Speaches whisper `small`) | ~0.3 s warm (measured) |
| TTS (Speaches Kokoro) | ~1.9 s per ~3 s sentence, warm (measured) |
| Tatlock Steward analysis | ~6 s warm |
| **Tatlock, full local flow** | **1125 s end-to-end** (librarian-routed ~2025 s) |
| Tatlock cold start (>2 h idle) | +~8 s (`OLLAMA_KEEP_ALIVE=2h`) |
| STT (Speaches whisper `small`) | ~0.30 s warm, for ~4.8 s of audio |
| TTS (Speaches Kokoro) | ~0.24 s warm, for ~4.5 s of audio (RTF ~0.05) |
| **Tatlock, full local flow** | **~1013 s end-to-end** for simple turns |
| Tatlock cold model load | +~36 s — avoided while the model is pinned |
(Older "~35 s Steward / ~2 min flow" figures were from a CPU-only driver-mismatch era —
do not plan against them.)
A Tatlock turn costs **3 sequential Ollama calls** (Steward routing → tool orchestration →
butler-tone synthesis) and ~710 generated tokens even for "what is 61 plus 12?". Most of
that is the model's own reasoning: gemma4 thinks by default, and the effort is spent three
times per turn.
(Older figures — "~35 s Steward / ~2 min flow" from the CPU-only era, and "1125 s full
flow" from 2026-07-14 — are superseded. Do not plan against them.)
Speech is not the bottleneck — **Tatlock is**, by one to two orders of magnitude.
Constraints this imposes:
@@ -241,11 +250,11 @@ Constraints this imposes:
sentence-by-sentence**, forwarding audio as each sentence is ready. The device starts
speaking after the first sentence instead of waiting for the full reply — with
streaming, first audio should land roughly at Steward-time + first-sentence-time,
well under the 1125 s full-flow figure. The WS protocol already supports this: one
well under the ~1013 s full-flow figure. The WS protocol already supports this: one
`audio_start` … PCM … `audio_end` envelope with chunks arriving as they're
synthesized — the device just plays a continuous stream.
2. **The `thinking` face state is a first-class feature**, not decoration — it's what
makes a 1025 s Tatlock turn feel intentional instead of broken. Consider progress
makes a ~10 s Tatlock turn feel intentional instead of broken. Consider progress
cues (e.g. surface Tatlock's reasoning summaries on-screen) later.
3. A **fast lane** may eventually be needed: MultiNet on-device commands for instant
home-automation phrases, and/or a low-latency intent path in Tatlock itself. Out of