jpmschweitzerandClaude Fable 5
0886c84d5e
- STT/TTS move to a shared Speaches container (OpenAI-format, GPU, port 8601 proposed); gateway becomes a thin orchestrator with pluggable speech backends (speaches default, embedded fallback) - Record the on-device ceiling: WakeNet wake word, VAD, ES7210 AEC, optional MultiNet fixed commands; open-vocabulary STT permanently out - Record the real latency bottleneck (Tatlock ~2 min full local flow): gateway must stream chat tokens and synthesize sentence-by-sentence - Plan reply_delta + barge-in protocol additions Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
200 lines
11 KiB
Markdown
200 lines
11 KiB
Markdown
# DeskLock Architecture
|
||
|
||
## Goal
|
||
|
||
An always-on, glanceable butler face in the living room. You speak to it; it relays your
|
||
words to Tatlock and speaks the reply back, with a face that reflects what it's doing
|
||
(idle, listening, thinking, speaking). It is deliberately a *thin* endpoint: all
|
||
intelligence lives in Tatlock, all heavy audio processing lives server-side on
|
||
tower-of-joy. Everything is local — no audio, transcript, or reply ever leaves the LAN.
|
||
|
||
## System overview
|
||
|
||
```
|
||
┌──────────────────────┐ WebSocket: PCM audio + JSON events
|
||
│ DeskLock device │◄───────────────────────────────────┐
|
||
│ (ESP32-P4) │ │
|
||
│ • LVGL face │ ┌──────────────────────────────┴───────────┐
|
||
│ • touch / wake word │ │ DeskLock Gateway (container, :8600) │
|
||
│ • mic capture + AEC │ │ thin orchestrator — no ML dependencies │
|
||
│ • TTS playback │ └───────┬──────────────────┬───────────────┘
|
||
└──────────────────────┘ │ │ OpenAI-format HTTP
|
||
│ ▼
|
||
HTTP (LAN) │ ┌─────────────────────────────┐
|
||
▼ │ Speaches (container, GPU) │
|
||
┌────────────────────┐ │ • STT: faster-whisper │
|
||
│ Tatlock (butler) │ │ • TTS: Kokoro / Piper │
|
||
│ tatlock.schweitz. │ │ also usable by Open WebUI, │
|
||
│ internal :8000 │ │ Home Assistant, … │
|
||
└────────────────────┘ └─────────────────────────────┘
|
||
```
|
||
|
||
Tatlock stays a text-only brain. The **gateway** orchestrates Tatlock's ears and mouth;
|
||
the **speech layer** (Speaches) owns the actual STT/TTS models on the GPU.
|
||
|
||
## Components
|
||
|
||
### 1. Firmware (`firmware/`) — ESP32-P4
|
||
|
||
Responsibilities:
|
||
|
||
- **Face rendering** (LVGL 9 on the 800×800 round MIPI-DSI panel via the
|
||
`waveshare/esp32_p4_wifi6_touch_lcd_xc` BSP). Face states:
|
||
- `idle` — subtle animation + clock (it's a desk clock when nobody's talking to it)
|
||
- `listening` — visual feedback that the mic is hot
|
||
- `thinking` — Tatlock is working on a reply (this state earns its keep; see
|
||
[Latency budget](#latency-budget--streaming))
|
||
- `speaking` — mouth/waveform animation synced to TTS playback
|
||
- **Audio capture**: dual mics through the ES7210 (hardware echo cancellation reference
|
||
from the playback path), 16 kHz 16-bit mono PCM.
|
||
- **Audio playback**: ES8311 codec → speaker. Plays PCM streamed from the gateway.
|
||
- **Transport**: a single WebSocket to the gateway carrying binary PCM frames plus JSON
|
||
control events (`state`, `transcript`, `reply_text`, errors). Device reconnects with
|
||
backoff; face shows a disconnected state when the gateway is unreachable.
|
||
|
||
**On-device speech processing — what runs on the P4 and what deliberately doesn't.**
|
||
The P4 (dual RISC-V @ 400 MHz, 32 MB PSRAM) has a hard ceiling; the split is:
|
||
|
||
| On-device (planned) | Why |
|
||
|---------------------|-----|
|
||
| Wake word — esp-sr WakeNet (phase 2) | Must be local: always-listening audio should never leave the device until the wake word fires |
|
||
| Voice-activity detection (end-of-utterance) | Removes tap-to-stop; cheap on-device |
|
||
| Echo cancellation — ES7210 hardware | Enables barge-in while TTS is playing |
|
||
| esp-sr MultiNet fixed commands (optional, later) | ~200-phrase closed vocabulary recognized entirely on-device — instant "lights off"-style commands with zero round trip |
|
||
|
||
Full open-vocabulary STT on-device is **out of scope permanently**: even whisper-tiny
|
||
needs hundreds of MB and orders of magnitude more compute than the P4 offers. Anything
|
||
open-ended goes to the speech layer.
|
||
|
||
Non-responsibilities: no STT, no TTS, no conversation state. If it's not rendering,
|
||
recording, or playing, it doesn't belong in firmware.
|
||
|
||
### 2. Gateway (`gateway/`) — container on tower-of-joy, port 8600
|
||
|
||
A FastAPI service bridging device audio to Tatlock text. It owns *orchestration*, not
|
||
models — the container stays a slim pure-Python image with no CUDA/ML dependencies:
|
||
|
||
1. Accepts the device WebSocket (`/ws/voice`).
|
||
2. Buffers inbound PCM until end-of-utterance (client-signalled in phase 1; VAD later).
|
||
3. **STT**: POST to Speaches `/v1/audio/transcriptions`.
|
||
4. **Chat**: POST the transcript to Tatlock `/v1/chat/completions`
|
||
(`http://tatlock.schweitz.internal:8000`, OpenAI-compatible, **streaming**),
|
||
maintaining the conversation history so follow-ups have context.
|
||
5. **TTS**: as Tatlock's token stream completes each sentence, POST it to Speaches
|
||
`/v1/audio/speech` and forward the PCM immediately — see
|
||
[Latency budget](#latency-budget--streaming).
|
||
|
||
`stt.py` / `tts.py` are pluggable backends selected by config
|
||
(`DESKLOCK_STT_BACKEND` / `DESKLOCK_TTS_BACKEND`):
|
||
|
||
- `speaches` (default) — OpenAI-format HTTP to the shared speech container.
|
||
- `embedded` — in-process faster-whisper / Piper. Kept as a fallback so the gateway can
|
||
run standalone (dev on a laptop, speech container down), at the cost of a fat image.
|
||
|
||
> **Status note:** the initial scaffold implements only the `embedded` path; the
|
||
> backend switch and Speaches client are the next gateway task.
|
||
|
||
The gateway is stateless apart from in-flight conversations; it can restart freely.
|
||
|
||
### 3. Speech layer — Speaches (container, GPU)
|
||
|
||
[Speaches](https://github.com/speaches-ai/speaches) (successor to faster-whisper-server)
|
||
is a self-hosted, OpenAI-API-compatible speech server: STT via faster-whisper, TTS via
|
||
Kokoro/Piper, dynamic model load/offload with a TTL, and a `/v1/realtime` WebSocket API
|
||
we may adopt later for streaming transcription.
|
||
|
||
- **Deployment**: its own stack in `system-admin-toj/containers/stacks/`, GPU-enabled.
|
||
Proposed host port **8601** (verified free; register in `CONTAINERS.md` at deploy).
|
||
LAN-only like the Tatlock internal route — do not expose through NPM without auth.
|
||
- **Why a shared layer instead of models inside the gateway**: one GPU-resident model
|
||
instance serves the whole homelab. Open WebUI is currently configured with
|
||
`AUDIO_STT_ENGINE=openai` / `AUDIO_TTS_ENGINE=openai` (OpenAI *cloud*) — pointing its
|
||
audio base URL at Speaches makes it fully local with a config change. Home Assistant
|
||
can share it too. Meanwhile the gateway image needs no CUDA and rebuilds in seconds.
|
||
- **VRAM budget**: RTX 2080 Ti, 11 GB, shared with Ollama (~3.6 GB in use as of
|
||
2026-07). whisper `small` at int8 is <1 GB; Kokoro is a few hundred MB. Speaches'
|
||
model TTL offload keeps idle pressure near zero. If VRAM contention ever bites,
|
||
faster-whisper `small` on CPU is an acceptable fallback (int8, a few seconds per
|
||
utterance).
|
||
|
||
### 4. Tatlock — existing backend (`/mnt/media/Projects/tatlock`)
|
||
|
||
Untouched by this project. DeskLock consumes its OpenAI-compatible API over the internal
|
||
LAN (port 8000, bypassing the Authentik-protected public route). If device auth is needed
|
||
later, the gateway holds the credential — never the firmware.
|
||
|
||
## Latency budget & streaming
|
||
|
||
Measured/known numbers that shape the design:
|
||
|
||
| Stage | Cost |
|
||
|-------|------|
|
||
| STT (whisper `small`, GPU) | a few hundred ms for a ~5 s utterance |
|
||
| TTS (Piper/Kokoro) | faster than realtime |
|
||
| **Tatlock, full local flow (gemma4)** | **~35 s Steward analysis warm; ~2 min end-to-end** (per tatlock CLAUDE.md) |
|
||
|
||
Speech is not the bottleneck — **Tatlock is**, by two orders of magnitude. Constraints
|
||
this imposes:
|
||
|
||
1. **The gateway must consume Tatlock's streaming response and synthesize
|
||
sentence-by-sentence**, forwarding audio as each sentence is ready. The device starts
|
||
speaking after the first sentence instead of waiting for the full reply. The WS
|
||
protocol already supports this: one `audio_start` … PCM … `audio_end` envelope with
|
||
chunks arriving as they're synthesized — the device just plays a continuous stream.
|
||
2. **The `thinking` face state is a first-class feature**, not decoration — it's what
|
||
makes a long Tatlock turn feel intentional instead of broken. Consider progress cues
|
||
(e.g. surface Tatlock's reasoning summaries on-screen) later.
|
||
3. A **fast lane** may eventually be needed: MultiNet on-device commands for instant
|
||
home-automation phrases, and/or a low-latency intent path in Tatlock itself. Out of
|
||
scope for now, but don't design it out.
|
||
|
||
## WebSocket protocol (device ↔ gateway)
|
||
|
||
Binary frames: raw 16 kHz s16le mono PCM (mic upstream, TTS downstream).
|
||
Text frames: JSON control messages.
|
||
|
||
```
|
||
device → gateway: {"type": "utterance_start"}
|
||
device → gateway: <binary PCM frames>
|
||
device → gateway: {"type": "utterance_end"}
|
||
gateway → device: {"type": "state", "value": "thinking"}
|
||
gateway → device: {"type": "transcript", "text": "..."}
|
||
gateway → device: {"type": "reply_text", "text": "..."}
|
||
gateway → device: {"type": "audio_start", "sample_rate": 16000}
|
||
gateway → device: <binary PCM frames> (may arrive sentence-by-sentence; play as a stream)
|
||
gateway → device: {"type": "audio_end"}
|
||
```
|
||
|
||
Planned additions (documented before implemented, here first):
|
||
|
||
- `reply_delta` (gateway → device): incremental reply text for on-screen streaming while
|
||
audio is synthesized.
|
||
- An interrupt event (device → gateway) for barge-in during playback (phase 4).
|
||
|
||
Keep this protocol documented here and mirrored in `firmware/` and `gateway/` constants —
|
||
it is the one contract between the two halves of the repo.
|
||
|
||
## Key decisions & rationale
|
||
|
||
- **ESP-IDF native (not Arduino/ESPHome)**: the P4 + MIPI-DSI + esp-sr stack is only
|
||
first-class in ESP-IDF; Waveshare recommends it, and the BSP targets it.
|
||
- **Server-side STT/TTS, device does wake word + VAD + AEC only**: server whisper is
|
||
dramatically better than anything embeddable, the GPU is already there, and the P4
|
||
physically can't run open-vocabulary STT. Wake word must be on-device (privacy: no
|
||
audio leaves the device until it fires).
|
||
- **STT/TTS as a shared Speaches service (not embedded in the gateway)**: one model
|
||
instance for the whole homelab (DeskLock, Open WebUI, potentially HA), slim gateway
|
||
image, models upgradable independently. `embedded` backend retained as a dev/fallback
|
||
mode.
|
||
- *Rejected — Wyoming protocol containers* (`wyoming-faster-whisper`/`wyoming-piper`):
|
||
native to Home Assistant's ecosystem, but Tatlock and Open WebUI already speak
|
||
OpenAI format, so Speaches fits the lab better. Revisit only if HA Assist becomes a
|
||
first-class consumer.
|
||
- *Rejected — cloud STT/TTS*: violates the local-first premise; also adds WAN latency
|
||
and per-minute cost.
|
||
- **Separate gateway (not extending Tatlock)**: keeps Tatlock's API text-only and clean;
|
||
audio concerns (codecs, VAD, streaming, sentence segmentation) stay at the edge. The
|
||
gateway is also where a future second endpoint (kitchen, office) would connect.
|
||
- **Monorepo**: the WS protocol couples firmware and gateway; versioning them together
|
||
avoids contract drift.
|