From 0886c84d5ed90b0a080e36cfe2452e884cc8be5d Mon Sep 17 00:00:00 2001 From: Jeroen Schweitzer Date: Tue, 14 Jul 2026 17:14:44 +0200 Subject: [PATCH] Document Speaches speech layer, on-device processing limits, latency constraints - STT/TTS move to a shared Speaches container (OpenAI-format, GPU, port 8601 proposed); gateway becomes a thin orchestrator with pluggable speech backends (speaches default, embedded fallback) - Record the on-device ceiling: WakeNet wake word, VAD, ES7210 AEC, optional MultiNet fixed commands; open-vocabulary STT permanently out - Record the real latency bottleneck (Tatlock ~2 min full local flow): gateway must stream chat tokens and synthesize sentence-by-sentence - Plan reply_delta + barge-in protocol additions Co-Authored-By: Claude Fable 5 --- AGENTS.md | 8 ++- README.md | 41 ++++++------ docs/architecture.md | 153 +++++++++++++++++++++++++++++++++++++------ 3 files changed, 160 insertions(+), 42 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index aa69b77..14b77fd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,8 +11,12 @@ one repo: - `firmware/` — ESP-IDF (C, LVGL 9) app for the Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C (3.4" round 800×800 touch display, dual mics + ES7210 AEC, ES8311 codec + speaker). -- `gateway/` — Python FastAPI container on tower-of-joy doing STT (faster-whisper, GPU), - chat (Tatlock `/v1/chat/completions`), and TTS (Piper). Listens on port **8600**. +- `gateway/` — Python FastAPI container on tower-of-joy orchestrating STT → chat + (Tatlock `/v1/chat/completions`) → TTS. Listens on port **8600**. STT/TTS models live + in a shared **Speaches** container (proposed port 8601, OpenAI-format API), not in the + gateway image; `stt.py`/`tts.py` are pluggable backends (`speaches` default, + `embedded` fallback for dev). See docs/architecture.md — the scaffold currently + implements only `embedded`. The device and gateway speak a WebSocket protocol defined in `docs/architecture.md`. **That doc is the contract** — update it in the same change as any protocol edit on diff --git a/README.md b/README.md index 49ac9b4..9a64af5 100644 --- a/README.md +++ b/README.md @@ -25,28 +25,28 @@ happens on this server. ## Architecture ``` -┌─────────────────────┐ WebSocket (PCM audio + JSON events) -│ DeskLock device │◄──────────────────────────────────────────┐ -│ (ESP32-P4) │ │ -│ │ ┌──────────────────────────────────┴──┐ -│ • LVGL face │ │ DeskLock Gateway (container) │ -│ • Touch input │ │ on tower-of-joy │ -│ • Mic capture+AEC │ │ │ -│ • TTS playback │ │ • STT: faster-whisper (GPU) │ -└─────────────────────┘ │ • TTS: Piper │ - │ • Chat: Tatlock /v1/chat/completions│ - └──────────────────┬───────────────────┘ - │ HTTP (LAN, :8000) - ┌──────────┴──────────┐ - │ Tatlock (butler) │ - │ tatlock.schweitz. │ - │ internal │ - └─────────────────────┘ +┌──────────────────────┐ WebSocket: PCM audio + JSON events +│ DeskLock device │◄───────────────────────────────────┐ +│ (ESP32-P4) │ │ +│ • LVGL face │ ┌──────────────────────────────┴───────────┐ +│ • touch / wake word │ │ DeskLock Gateway (container, :8600) │ +│ • mic capture + AEC │ │ thin orchestrator — no ML dependencies │ +│ • TTS playback │ └───────┬──────────────────┬───────────────┘ +└──────────────────────┘ │ │ OpenAI-format HTTP + │ ▼ + HTTP (LAN) │ ┌─────────────────────────────┐ + ▼ │ Speaches (container, GPU) │ + ┌────────────────────┐ │ • STT: faster-whisper │ + │ Tatlock (butler) │ │ • TTS: Kokoro / Piper │ + │ tatlock.schweitz. │ │ also usable by Open WebUI, │ + │ internal :8000 │ │ Home Assistant, … │ + └────────────────────┘ └─────────────────────────────┘ ``` -Tatlock stays a text-only brain. The **gateway** is Tatlock's ears and mouth: it converts -speech to text on the way in and text to speech on the way out, keeping the device firmware -thin (audio transport + face rendering only). +Tatlock stays a text-only brain. The **gateway** orchestrates speech-to-text, chat, and +text-to-speech; the **Speaches** container owns the actual STT/TTS models on the GPU, +shared homelab-wide. The device firmware stays thin: audio transport, wake word, and +face rendering only. Everything runs on the LAN — no cloud in the voice path. See [docs/architecture.md](docs/architecture.md) for the full design. @@ -69,4 +69,5 @@ See [docs/architecture.md](docs/architecture.md) for the full design. - [Waveshare wiki: ESP32-P4-WIFI6-Touch-LCD-3.4C](https://www.waveshare.com/wiki/ESP32-P4-WIFI6-Touch-LCD-3.4C) - [Official examples repo (waveshareteam/ESP32-P4-WIFI6-Touch-LCD-XC)](https://github.com/waveshareteam/ESP32-P4-WIFI6-Touch-LCD-XC) - [BSP component: waveshare/esp32_p4_wifi6_touch_lcd_xc](https://components.espressif.com/components/waveshare/esp32_p4_wifi6_touch_lcd_xc) +- [Speaches — self-hosted OpenAI-compatible speech server](https://github.com/speaches-ai/speaches) - Tatlock backend: `/mnt/media/Projects/tatlock` — https://tatlock.schweitz.net diff --git a/docs/architecture.md b/docs/architecture.md index aa566b5..7ee01a9 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -2,10 +2,35 @@ ## Goal -A always-on, glanceable butler face in the living room. You speak to it; it relays your +An always-on, glanceable butler face in the living room. You speak to it; it relays your words to Tatlock and speaks the reply back, with a face that reflects what it's doing (idle, listening, thinking, speaking). It is deliberately a *thin* endpoint: all -intelligence lives in Tatlock, all heavy audio processing lives in the gateway. +intelligence lives in Tatlock, all heavy audio processing lives server-side on +tower-of-joy. Everything is local — no audio, transcript, or reply ever leaves the LAN. + +## System overview + +``` +┌──────────────────────┐ WebSocket: PCM audio + JSON events +│ DeskLock device │◄───────────────────────────────────┐ +│ (ESP32-P4) │ │ +│ • LVGL face │ ┌──────────────────────────────┴───────────┐ +│ • touch / wake word │ │ DeskLock Gateway (container, :8600) │ +│ • mic capture + AEC │ │ thin orchestrator — no ML dependencies │ +│ • TTS playback │ └───────┬──────────────────┬───────────────┘ +└──────────────────────┘ │ │ OpenAI-format HTTP + │ ▼ + HTTP (LAN) │ ┌─────────────────────────────┐ + ▼ │ Speaches (container, GPU) │ + ┌────────────────────┐ │ • STT: faster-whisper │ + │ Tatlock (butler) │ │ • TTS: Kokoro / Piper │ + │ tatlock.schweitz. │ │ also usable by Open WebUI, │ + │ internal :8000 │ │ Home Assistant, … │ + └────────────────────┘ └─────────────────────────────┘ +``` + +Tatlock stays a text-only brain. The **gateway** orchestrates Tatlock's ears and mouth; +the **speech layer** (Speaches) owns the actual STT/TTS models on the GPU. ## Components @@ -17,7 +42,8 @@ Responsibilities: `waveshare/esp32_p4_wifi6_touch_lcd_xc` BSP). Face states: - `idle` — subtle animation + clock (it's a desk clock when nobody's talking to it) - `listening` — visual feedback that the mic is hot - - `thinking` — Tatlock is working on a reply + - `thinking` — Tatlock is working on a reply (this state earns its keep; see + [Latency budget](#latency-budget--streaming)) - `speaking` — mouth/waveform animation synced to TTS playback - **Audio capture**: dual mics through the ES7210 (hardware echo cancellation reference from the playback path), 16 kHz 16-bit mono PCM. @@ -25,33 +51,103 @@ Responsibilities: - **Transport**: a single WebSocket to the gateway carrying binary PCM frames plus JSON control events (`state`, `transcript`, `reply_text`, errors). Device reconnects with backoff; face shows a disconnected state when the gateway is unreachable. -- **Interaction**: phase 1 is touch-to-talk (tap the face). Phase 2 adds esp-sr WakeNet - wake word on the P4 so the interaction is hands-free. + +**On-device speech processing — what runs on the P4 and what deliberately doesn't.** +The P4 (dual RISC-V @ 400 MHz, 32 MB PSRAM) has a hard ceiling; the split is: + +| On-device (planned) | Why | +|---------------------|-----| +| Wake word — esp-sr WakeNet (phase 2) | Must be local: always-listening audio should never leave the device until the wake word fires | +| Voice-activity detection (end-of-utterance) | Removes tap-to-stop; cheap on-device | +| Echo cancellation — ES7210 hardware | Enables barge-in while TTS is playing | +| esp-sr MultiNet fixed commands (optional, later) | ~200-phrase closed vocabulary recognized entirely on-device — instant "lights off"-style commands with zero round trip | + +Full open-vocabulary STT on-device is **out of scope permanently**: even whisper-tiny +needs hundreds of MB and orders of magnitude more compute than the P4 offers. Anything +open-ended goes to the speech layer. Non-responsibilities: no STT, no TTS, no conversation state. If it's not rendering, recording, or playing, it doesn't belong in firmware. -### 2. Gateway (`gateway/`) — container on tower-of-joy +### 2. Gateway (`gateway/`) — container on tower-of-joy, port 8600 -A FastAPI service bridging device audio to Tatlock text: +A FastAPI service bridging device audio to Tatlock text. It owns *orchestration*, not +models — the container stays a slim pure-Python image with no CUDA/ML dependencies: 1. Accepts the device WebSocket (`/ws/voice`). 2. Buffers inbound PCM until end-of-utterance (client-signalled in phase 1; VAD later). -3. **STT**: faster-whisper on the RTX 2080 Ti. -4. **Chat**: POSTs the transcript to Tatlock `/v1/chat/completions` - (`http://tatlock.schweitz.internal:8000`, OpenAI-compatible, streaming). Maintains the - conversation id so follow-ups have context. -5. **TTS**: Piper (fast, CPU-friendly, local) synthesizes the reply. -6. Streams reply PCM back to the device along with `reply_text` for on-screen display. +3. **STT**: POST to Speaches `/v1/audio/transcriptions`. +4. **Chat**: POST the transcript to Tatlock `/v1/chat/completions` + (`http://tatlock.schweitz.internal:8000`, OpenAI-compatible, **streaming**), + maintaining the conversation history so follow-ups have context. +5. **TTS**: as Tatlock's token stream completes each sentence, POST it to Speaches + `/v1/audio/speech` and forward the PCM immediately — see + [Latency budget](#latency-budget--streaming). + +`stt.py` / `tts.py` are pluggable backends selected by config +(`DESKLOCK_STT_BACKEND` / `DESKLOCK_TTS_BACKEND`): + +- `speaches` (default) — OpenAI-format HTTP to the shared speech container. +- `embedded` — in-process faster-whisper / Piper. Kept as a fallback so the gateway can + run standalone (dev on a laptop, speech container down), at the cost of a fat image. + +> **Status note:** the initial scaffold implements only the `embedded` path; the +> backend switch and Speaches client are the next gateway task. The gateway is stateless apart from in-flight conversations; it can restart freely. -### 3. Tatlock — existing backend (`/mnt/media/Projects/tatlock`) +### 3. Speech layer — Speaches (container, GPU) + +[Speaches](https://github.com/speaches-ai/speaches) (successor to faster-whisper-server) +is a self-hosted, OpenAI-API-compatible speech server: STT via faster-whisper, TTS via +Kokoro/Piper, dynamic model load/offload with a TTL, and a `/v1/realtime` WebSocket API +we may adopt later for streaming transcription. + +- **Deployment**: its own stack in `system-admin-toj/containers/stacks/`, GPU-enabled. + Proposed host port **8601** (verified free; register in `CONTAINERS.md` at deploy). + LAN-only like the Tatlock internal route — do not expose through NPM without auth. +- **Why a shared layer instead of models inside the gateway**: one GPU-resident model + instance serves the whole homelab. Open WebUI is currently configured with + `AUDIO_STT_ENGINE=openai` / `AUDIO_TTS_ENGINE=openai` (OpenAI *cloud*) — pointing its + audio base URL at Speaches makes it fully local with a config change. Home Assistant + can share it too. Meanwhile the gateway image needs no CUDA and rebuilds in seconds. +- **VRAM budget**: RTX 2080 Ti, 11 GB, shared with Ollama (~3.6 GB in use as of + 2026-07). whisper `small` at int8 is <1 GB; Kokoro is a few hundred MB. Speaches' + model TTL offload keeps idle pressure near zero. If VRAM contention ever bites, + faster-whisper `small` on CPU is an acceptable fallback (int8, a few seconds per + utterance). + +### 4. Tatlock — existing backend (`/mnt/media/Projects/tatlock`) Untouched by this project. DeskLock consumes its OpenAI-compatible API over the internal LAN (port 8000, bypassing the Authentik-protected public route). If device auth is needed later, the gateway holds the credential — never the firmware. +## Latency budget & streaming + +Measured/known numbers that shape the design: + +| Stage | Cost | +|-------|------| +| STT (whisper `small`, GPU) | a few hundred ms for a ~5 s utterance | +| TTS (Piper/Kokoro) | faster than realtime | +| **Tatlock, full local flow (gemma4)** | **~35 s Steward analysis warm; ~2 min end-to-end** (per tatlock CLAUDE.md) | + +Speech is not the bottleneck — **Tatlock is**, by two orders of magnitude. Constraints +this imposes: + +1. **The gateway must consume Tatlock's streaming response and synthesize + sentence-by-sentence**, forwarding audio as each sentence is ready. The device starts + speaking after the first sentence instead of waiting for the full reply. The WS + protocol already supports this: one `audio_start` … PCM … `audio_end` envelope with + chunks arriving as they're synthesized — the device just plays a continuous stream. +2. **The `thinking` face state is a first-class feature**, not decoration — it's what + makes a long Tatlock turn feel intentional instead of broken. Consider progress cues + (e.g. surface Tatlock's reasoning summaries on-screen) later. +3. A **fast lane** may eventually be needed: MultiNet on-device commands for instant + home-automation phrases, and/or a low-latency intent path in Tatlock itself. Out of + scope for now, but don't design it out. + ## WebSocket protocol (device ↔ gateway) Binary frames: raw 16 kHz s16le mono PCM (mic upstream, TTS downstream). @@ -65,10 +161,16 @@ gateway → device: {"type": "state", "value": "thinking"} gateway → device: {"type": "transcript", "text": "..."} gateway → device: {"type": "reply_text", "text": "..."} gateway → device: {"type": "audio_start", "sample_rate": 16000} -gateway → device: +gateway → device: (may arrive sentence-by-sentence; play as a stream) gateway → device: {"type": "audio_end"} ``` +Planned additions (documented before implemented, here first): + +- `reply_delta` (gateway → device): incremental reply text for on-screen streaming while + audio is synthesized. +- An interrupt event (device → gateway) for barge-in during playback (phase 4). + Keep this protocol documented here and mirrored in `firmware/` and `gateway/` constants — it is the one contract between the two halves of the repo. @@ -76,11 +178,22 @@ it is the one contract between the two halves of the repo. - **ESP-IDF native (not Arduino/ESPHome)**: the P4 + MIPI-DSI + esp-sr stack is only first-class in ESP-IDF; Waveshare recommends it, and the BSP targets it. -- **Gateway owns STT/TTS (not the device)**: the P4 could run small STT models, but - server-side whisper is dramatically better, and Piper voices beat embedded TTS. The - GPU is already there. Wake word is the only speech task that must be on-device. +- **Server-side STT/TTS, device does wake word + VAD + AEC only**: server whisper is + dramatically better than anything embeddable, the GPU is already there, and the P4 + physically can't run open-vocabulary STT. Wake word must be on-device (privacy: no + audio leaves the device until it fires). +- **STT/TTS as a shared Speaches service (not embedded in the gateway)**: one model + instance for the whole homelab (DeskLock, Open WebUI, potentially HA), slim gateway + image, models upgradable independently. `embedded` backend retained as a dev/fallback + mode. + - *Rejected — Wyoming protocol containers* (`wyoming-faster-whisper`/`wyoming-piper`): + native to Home Assistant's ecosystem, but Tatlock and Open WebUI already speak + OpenAI format, so Speaches fits the lab better. Revisit only if HA Assist becomes a + first-class consumer. + - *Rejected — cloud STT/TTS*: violates the local-first premise; also adds WAN latency + and per-minute cost. - **Separate gateway (not extending Tatlock)**: keeps Tatlock's API text-only and clean; - audio concerns (codecs, VAD, streaming) stay at the edge. The gateway is also where a - future second endpoint (kitchen, office) would connect. + audio concerns (codecs, VAD, streaming, sentence segmentation) stay at the edge. The + gateway is also where a future second endpoint (kitchen, office) would connect. - **Monorepo**: the WS protocol couples firmware and gateway; versioning them together avoids contract drift.