jpmschweitzer/desklock
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 4 Wiki Activity

Document Speaches speech layer, on-device processing limits, latency constraints

Browse Source 
- STT/TTS move to a shared Speaches container (OpenAI-format, GPU, port
  8601 proposed); gateway becomes a thin orchestrator with pluggable
  speech backends (speaches default, embedded fallback)
- Record the on-device ceiling: WakeNet wake word, VAD, ES7210 AEC,
  optional MultiNet fixed commands; open-vocabulary STT permanently out
- Record the real latency bottleneck (Tatlock ~2 min full local flow):
  gateway must stream chat tokens and synthesize sentence-by-sentence
- Plan reply_delta + barge-in protocol additions

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jpmschweitzer
2026-07-14 17:14:44 +02:00
co-authored by Claude Fable 5
parent 576fd7d237
commit 0886c84d5e
3 changed files with 160 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+6 -2
View File
@@ -11,8 +11,12 @@ one repo:
- `firmware/` — ESP-IDF (C, LVGL 9) app for the Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C - `firmware/` — ESP-IDF (C, LVGL 9) app for the Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C
(3.4" round 800×800 touch display, dual mics + ES7210 AEC, ES8311 codec + speaker). (3.4" round 800×800 touch display, dual mics + ES7210 AEC, ES8311 codec + speaker).
- `gateway/` — Python FastAPI container on tower-of-joy doing STT (faster-whisper, GPU), - `gateway/` — Python FastAPI container on tower-of-joy orchestrating STT → chat
chat (Tatlock `/v1/chat/completions`), and TTS (Piper). Listens on port **8600**. (Tatlock `/v1/chat/completions`) → TTS. Listens on port **8600**. STT/TTS models live
in a shared **Speaches** container (proposed port 8601, OpenAI-format API), not in the
gateway image; `stt.py`/`tts.py` are pluggable backends (`speaches` default,
`embedded` fallback for dev). See docs/architecture.md — the scaffold currently
implements only `embedded`.
The device and gateway speak a WebSocket protocol defined in `docs/architecture.md`. The device and gateway speak a WebSocket protocol defined in `docs/architecture.md`.
**That doc is the contract** — update it in the same change as any protocol edit on **That doc is the contract** — update it in the same change as any protocol edit on
README.md
+21 -20
View File
@@ -25,28 +25,28 @@ happens on this server.
## Architecture ## Architecture
``` ```
┌─────────────────────┐ WebSocket (PCM audio + JSON events) ┌──────────────────────┐ WebSocket: PCM audio + JSON events
│ DeskLock device │◄──────────────────────────────────────────┐ │ DeskLock device │◄───────────────────────────────────┐
│ (ESP32-P4) │ │ │ (ESP32-P4) │ │
│ │ ┌──────────────────────────────────┴──┐ │ • LVGL face │ ┌──────────────────────────────┴───────────┐
│ • LVGL face │ │ DeskLock Gateway (container) │ │ • touch / wake word │ │ DeskLock Gateway (container, :8600) │
│ • Touch input │ │ on tower-of-joy │ │ • mic capture + AEC │ │ thin orchestrator — no ML dependencies │
│ • Mic capture+AEC │ │ │ │ • TTS playback │ └───────┬──────────────────┬───────────────┘
│ • TTS playback │ │ • STT: faster-whisper (GPU) │ └──────────────────────┘ │ │ OpenAI-format HTTP
└─────────────────────┘ │ • TTS: Piper │ │ ▼
│ • Chat: Tatlock /v1/chat/completions│ HTTP (LAN) │ ┌─────────────────────────────┐
└──────────────────┬───────────────────┘ ▼ │ Speaches (container, GPU) │
│ HTTP (LAN, :8000) ┌────────────────────┐ │ • STT: faster-whisper │
┌──────────┴──────────┐ │ Tatlock (butler) │ │ • TTS: Kokoro / Piper │
│ Tatlock (butler) │ │ tatlock.schweitz. │ │ also usable by Open WebUI, │
│ tatlock.schweitz. │ │ internal :8000 │ │ Home Assistant, … │
│ internal │ └────────────────────┘ └─────────────────────────────┘
└─────────────────────┘
``` ```
Tatlock stays a text-only brain. The **gateway** is Tatlock's ears and mouth: it converts Tatlock stays a text-only brain. The **gateway** orchestrates speech-to-text, chat, and
speech to text on the way in and text to speech on the way out, keeping the device firmware text-to-speech; the **Speaches** container owns the actual STT/TTS models on the GPU,
thin (audio transport + face rendering only). shared homelab-wide. The device firmware stays thin: audio transport, wake word, and
face rendering only. Everything runs on the LAN — no cloud in the voice path.
See [docs/architecture.md](docs/architecture.md) for the full design. See [docs/architecture.md](docs/architecture.md) for the full design.
@@ -69,4 +69,5 @@ See [docs/architecture.md](docs/architecture.md) for the full design.
- [Waveshare wiki: ESP32-P4-WIFI6-Touch-LCD-3.4C](https://www.waveshare.com/wiki/ESP32-P4-WIFI6-Touch-LCD-3.4C) - [Waveshare wiki: ESP32-P4-WIFI6-Touch-LCD-3.4C](https://www.waveshare.com/wiki/ESP32-P4-WIFI6-Touch-LCD-3.4C)
- [Official examples repo (waveshareteam/ESP32-P4-WIFI6-Touch-LCD-XC)](https://github.com/waveshareteam/ESP32-P4-WIFI6-Touch-LCD-XC) - [Official examples repo (waveshareteam/ESP32-P4-WIFI6-Touch-LCD-XC)](https://github.com/waveshareteam/ESP32-P4-WIFI6-Touch-LCD-XC)
- [BSP component: waveshare/esp32_p4_wifi6_touch_lcd_xc](https://components.espressif.com/components/waveshare/esp32_p4_wifi6_touch_lcd_xc) - [BSP component: waveshare/esp32_p4_wifi6_touch_lcd_xc](https://components.espressif.com/components/waveshare/esp32_p4_wifi6_touch_lcd_xc)
- [Speaches — self-hosted OpenAI-compatible speech server](https://github.com/speaches-ai/speaches)
- Tatlock backend: `/mnt/media/Projects/tatlock` — https://tatlock.schweitz.net - Tatlock backend: `/mnt/media/Projects/tatlock` — https://tatlock.schweitz.net
docs/architecture.md
+133 -20
View File
@@ -2,10 +2,35 @@
## Goal ## Goal
A always-on, glanceable butler face in the living room. You speak to it; it relays your An always-on, glanceable butler face in the living room. You speak to it; it relays your
words to Tatlock and speaks the reply back, with a face that reflects what it's doing words to Tatlock and speaks the reply back, with a face that reflects what it's doing
(idle, listening, thinking, speaking). It is deliberately a *thin* endpoint: all (idle, listening, thinking, speaking). It is deliberately a *thin* endpoint: all
intelligence lives in Tatlock, all heavy audio processing lives in the gateway. intelligence lives in Tatlock, all heavy audio processing lives server-side on
tower-of-joy. Everything is local — no audio, transcript, or reply ever leaves the LAN.
## System overview
```
┌──────────────────────┐ WebSocket: PCM audio + JSON events
│ DeskLock device │◄───────────────────────────────────┐
│ (ESP32-P4) │ │
│ • LVGL face │ ┌──────────────────────────────┴───────────┐
│ • touch / wake word │ │ DeskLock Gateway (container, :8600) │
│ • mic capture + AEC │ │ thin orchestrator — no ML dependencies │
│ • TTS playback │ └───────┬──────────────────┬───────────────┘
└──────────────────────┘ │ │ OpenAI-format HTTP
│ ▼
HTTP (LAN) │ ┌─────────────────────────────┐
▼ │ Speaches (container, GPU) │
┌────────────────────┐ │ • STT: faster-whisper │
│ Tatlock (butler) │ │ • TTS: Kokoro / Piper │
│ tatlock.schweitz. │ │ also usable by Open WebUI, │
│ internal :8000 │ │ Home Assistant, … │
└────────────────────┘ └─────────────────────────────┘
```
Tatlock stays a text-only brain. The **gateway** orchestrates Tatlock's ears and mouth;
the **speech layer** (Speaches) owns the actual STT/TTS models on the GPU.
## Components ## Components
@@ -17,7 +42,8 @@ Responsibilities:
`waveshare/esp32_p4_wifi6_touch_lcd_xc` BSP). Face states: `waveshare/esp32_p4_wifi6_touch_lcd_xc` BSP). Face states:
- `idle` — subtle animation + clock (it's a desk clock when nobody's talking to it) - `idle` — subtle animation + clock (it's a desk clock when nobody's talking to it)
- `listening` — visual feedback that the mic is hot - `listening` — visual feedback that the mic is hot
- `thinking` — Tatlock is working on a reply - `thinking` — Tatlock is working on a reply (this state earns its keep; see
[Latency budget](#latency-budget--streaming))
- `speaking` — mouth/waveform animation synced to TTS playback - `speaking` — mouth/waveform animation synced to TTS playback
- **Audio capture**: dual mics through the ES7210 (hardware echo cancellation reference - **Audio capture**: dual mics through the ES7210 (hardware echo cancellation reference
from the playback path), 16 kHz 16-bit mono PCM. from the playback path), 16 kHz 16-bit mono PCM.
@@ -25,33 +51,103 @@ Responsibilities:
- **Transport**: a single WebSocket to the gateway carrying binary PCM frames plus JSON - **Transport**: a single WebSocket to the gateway carrying binary PCM frames plus JSON
control events (`state`, `transcript`, `reply_text`, errors). Device reconnects with control events (`state`, `transcript`, `reply_text`, errors). Device reconnects with
backoff; face shows a disconnected state when the gateway is unreachable. backoff; face shows a disconnected state when the gateway is unreachable.
- **Interaction**: phase 1 is touch-to-talk (tap the face). Phase 2 adds esp-sr WakeNet
wake word on the P4 so the interaction is hands-free. **On-device speech processing — what runs on the P4 and what deliberately doesn't.**
The P4 (dual RISC-V @ 400 MHz, 32 MB PSRAM) has a hard ceiling; the split is:
| On-device (planned) | Why |
|---------------------|-----|
| Wake word — esp-sr WakeNet (phase 2) | Must be local: always-listening audio should never leave the device until the wake word fires |
| Voice-activity detection (end-of-utterance) | Removes tap-to-stop; cheap on-device |
| Echo cancellation — ES7210 hardware | Enables barge-in while TTS is playing |
| esp-sr MultiNet fixed commands (optional, later) | ~200-phrase closed vocabulary recognized entirely on-device — instant "lights off"-style commands with zero round trip |
Full open-vocabulary STT on-device is **out of scope permanently**: even whisper-tiny
needs hundreds of MB and orders of magnitude more compute than the P4 offers. Anything
open-ended goes to the speech layer.
Non-responsibilities: no STT, no TTS, no conversation state. If it's not rendering, Non-responsibilities: no STT, no TTS, no conversation state. If it's not rendering,
recording, or playing, it doesn't belong in firmware. recording, or playing, it doesn't belong in firmware.
### 2. Gateway (`gateway/`) — container on tower-of-joy ### 2. Gateway (`gateway/`) — container on tower-of-joy, port 8600
A FastAPI service bridging device audio to Tatlock text: A FastAPI service bridging device audio to Tatlock text. It owns *orchestration*, not
models — the container stays a slim pure-Python image with no CUDA/ML dependencies:
1. Accepts the device WebSocket (`/ws/voice`). 1. Accepts the device WebSocket (`/ws/voice`).
2. Buffers inbound PCM until end-of-utterance (client-signalled in phase 1; VAD later). 2. Buffers inbound PCM until end-of-utterance (client-signalled in phase 1; VAD later).
3. **STT**: faster-whisper on the RTX 2080 Ti. 3. **STT**: POST to Speaches `/v1/audio/transcriptions`.
4. **Chat**: POSTs the transcript to Tatlock `/v1/chat/completions` 4. **Chat**: POST the transcript to Tatlock `/v1/chat/completions`
(`http://tatlock.schweitz.internal:8000`, OpenAI-compatible, streaming). Maintains the (`http://tatlock.schweitz.internal:8000`, OpenAI-compatible, **streaming**),
conversation id so follow-ups have context. maintaining the conversation history so follow-ups have context.
5. **TTS**: Piper (fast, CPU-friendly, local) synthesizes the reply. 5. **TTS**: as Tatlock's token stream completes each sentence, POST it to Speaches
6. Streams reply PCM back to the device along with `reply_text` for on-screen display. `/v1/audio/speech` and forward the PCM immediately — see
[Latency budget](#latency-budget--streaming).
`stt.py` / `tts.py` are pluggable backends selected by config
(`DESKLOCK_STT_BACKEND` / `DESKLOCK_TTS_BACKEND`):
- `speaches` (default) — OpenAI-format HTTP to the shared speech container.
- `embedded` — in-process faster-whisper / Piper. Kept as a fallback so the gateway can
run standalone (dev on a laptop, speech container down), at the cost of a fat image.
> **Status note:** the initial scaffold implements only the `embedded` path; the
> backend switch and Speaches client are the next gateway task.
The gateway is stateless apart from in-flight conversations; it can restart freely. The gateway is stateless apart from in-flight conversations; it can restart freely.
### 3. Tatlock — existing backend (`/mnt/media/Projects/tatlock`) ### 3. Speech layer — Speaches (container, GPU)
[Speaches](https://github.com/speaches-ai/speaches) (successor to faster-whisper-server)
is a self-hosted, OpenAI-API-compatible speech server: STT via faster-whisper, TTS via
Kokoro/Piper, dynamic model load/offload with a TTL, and a `/v1/realtime` WebSocket API
we may adopt later for streaming transcription.
- **Deployment**: its own stack in `system-admin-toj/containers/stacks/`, GPU-enabled.
Proposed host port **8601** (verified free; register in `CONTAINERS.md` at deploy).
LAN-only like the Tatlock internal route — do not expose through NPM without auth.
- **Why a shared layer instead of models inside the gateway**: one GPU-resident model
instance serves the whole homelab. Open WebUI is currently configured with
`AUDIO_STT_ENGINE=openai` / `AUDIO_TTS_ENGINE=openai` (OpenAI *cloud*) — pointing its
audio base URL at Speaches makes it fully local with a config change. Home Assistant
can share it too. Meanwhile the gateway image needs no CUDA and rebuilds in seconds.
- **VRAM budget**: RTX 2080 Ti, 11 GB, shared with Ollama (~3.6 GB in use as of
2026-07). whisper `small` at int8 is <1 GB; Kokoro is a few hundred MB. Speaches'
model TTL offload keeps idle pressure near zero. If VRAM contention ever bites,
faster-whisper `small` on CPU is an acceptable fallback (int8, a few seconds per
utterance).
### 4. Tatlock — existing backend (`/mnt/media/Projects/tatlock`)
Untouched by this project. DeskLock consumes its OpenAI-compatible API over the internal Untouched by this project. DeskLock consumes its OpenAI-compatible API over the internal
LAN (port 8000, bypassing the Authentik-protected public route). If device auth is needed LAN (port 8000, bypassing the Authentik-protected public route). If device auth is needed
later, the gateway holds the credential — never the firmware. later, the gateway holds the credential — never the firmware.
## Latency budget & streaming
Measured/known numbers that shape the design:
| Stage | Cost |
|-------|------|
| STT (whisper `small`, GPU) | a few hundred ms for a ~5 s utterance |
| TTS (Piper/Kokoro) | faster than realtime |
| **Tatlock, full local flow (gemma4)** | **~35 s Steward analysis warm; ~2 min end-to-end** (per tatlock CLAUDE.md) |
Speech is not the bottleneck — **Tatlock is**, by two orders of magnitude. Constraints
this imposes:
1. **The gateway must consume Tatlock's streaming response and synthesize
sentence-by-sentence**, forwarding audio as each sentence is ready. The device starts
speaking after the first sentence instead of waiting for the full reply. The WS
protocol already supports this: one `audio_start` … PCM … `audio_end` envelope with
chunks arriving as they're synthesized — the device just plays a continuous stream.
2. **The `thinking` face state is a first-class feature**, not decoration — it's what
makes a long Tatlock turn feel intentional instead of broken. Consider progress cues
(e.g. surface Tatlock's reasoning summaries on-screen) later.
3. A **fast lane** may eventually be needed: MultiNet on-device commands for instant
home-automation phrases, and/or a low-latency intent path in Tatlock itself. Out of
scope for now, but don't design it out.
## WebSocket protocol (device ↔ gateway) ## WebSocket protocol (device ↔ gateway)
Binary frames: raw 16 kHz s16le mono PCM (mic upstream, TTS downstream). Binary frames: raw 16 kHz s16le mono PCM (mic upstream, TTS downstream).
@@ -65,10 +161,16 @@ gateway → device: {"type": "state", "value": "thinking"}
gateway → device: {"type": "transcript", "text": "..."} gateway → device: {"type": "transcript", "text": "..."}
gateway → device: {"type": "reply_text", "text": "..."} gateway → device: {"type": "reply_text", "text": "..."}
gateway → device: {"type": "audio_start", "sample_rate": 16000} gateway → device: {"type": "audio_start", "sample_rate": 16000}
gateway → device: <binary PCM frames> gateway → device: <binary PCM frames> (may arrive sentence-by-sentence; play as a stream)
gateway → device: {"type": "audio_end"} gateway → device: {"type": "audio_end"}
``` ```
Planned additions (documented before implemented, here first):
- `reply_delta` (gateway → device): incremental reply text for on-screen streaming while
audio is synthesized.
- An interrupt event (device → gateway) for barge-in during playback (phase 4).
Keep this protocol documented here and mirrored in `firmware/` and `gateway/` constants — Keep this protocol documented here and mirrored in `firmware/` and `gateway/` constants —
it is the one contract between the two halves of the repo. it is the one contract between the two halves of the repo.
@@ -76,11 +178,22 @@ it is the one contract between the two halves of the repo.
- **ESP-IDF native (not Arduino/ESPHome)**: the P4 + MIPI-DSI + esp-sr stack is only - **ESP-IDF native (not Arduino/ESPHome)**: the P4 + MIPI-DSI + esp-sr stack is only
first-class in ESP-IDF; Waveshare recommends it, and the BSP targets it. first-class in ESP-IDF; Waveshare recommends it, and the BSP targets it.
- **Gateway owns STT/TTS (not the device)**: the P4 could run small STT models, but - **Server-side STT/TTS, device does wake word + VAD + AEC only**: server whisper is
server-side whisper is dramatically better, and Piper voices beat embedded TTS. The dramatically better than anything embeddable, the GPU is already there, and the P4
GPU is already there. Wake word is the only speech task that must be on-device. physically can't run open-vocabulary STT. Wake word must be on-device (privacy: no
audio leaves the device until it fires).
- **STT/TTS as a shared Speaches service (not embedded in the gateway)**: one model
instance for the whole homelab (DeskLock, Open WebUI, potentially HA), slim gateway
image, models upgradable independently. `embedded` backend retained as a dev/fallback
mode.
- *Rejected — Wyoming protocol containers* (`wyoming-faster-whisper`/`wyoming-piper`):
native to Home Assistant's ecosystem, but Tatlock and Open WebUI already speak
OpenAI format, so Speaches fits the lab better. Revisit only if HA Assist becomes a
first-class consumer.
- *Rejected — cloud STT/TTS*: violates the local-first premise; also adds WAN latency
and per-minute cost.
- **Separate gateway (not extending Tatlock)**: keeps Tatlock's API text-only and clean; - **Separate gateway (not extending Tatlock)**: keeps Tatlock's API text-only and clean;
audio concerns (codecs, VAD, streaming) stay at the edge. The gateway is also where a audio concerns (codecs, VAD, streaming, sentence segmentation) stay at the edge. The
future second endpoint (kitchen, office) would connect. gateway is also where a future second endpoint (kitchen, office) would connect.
- **Monorepo**: the WS protocol couples firmware and gateway; versioning them together - **Monorepo**: the WS protocol couples firmware and gateway; versioning them together
avoids contract drift. avoids contract drift.