jpmschweitzerandClaude Fable 5
576fd7d237
DeskLock gives the Tatlock butler a face and voice on a Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C round display in the living room. - firmware/: ESP-IDF project targeting esp32p4 with the Waveshare XC BSP - gateway/: FastAPI voice bridge (faster-whisper STT, Tatlock chat, Piper TTS) - docs/architecture.md: component design and device<->gateway WS protocol Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
87 lines
4.1 KiB
Markdown
87 lines
4.1 KiB
Markdown
# DeskLock Architecture
|
||
|
||
## Goal
|
||
|
||
A always-on, glanceable butler face in the living room. You speak to it; it relays your
|
||
words to Tatlock and speaks the reply back, with a face that reflects what it's doing
|
||
(idle, listening, thinking, speaking). It is deliberately a *thin* endpoint: all
|
||
intelligence lives in Tatlock, all heavy audio processing lives in the gateway.
|
||
|
||
## Components
|
||
|
||
### 1. Firmware (`firmware/`) — ESP32-P4
|
||
|
||
Responsibilities:
|
||
|
||
- **Face rendering** (LVGL 9 on the 800×800 round MIPI-DSI panel via the
|
||
`waveshare/esp32_p4_wifi6_touch_lcd_xc` BSP). Face states:
|
||
- `idle` — subtle animation + clock (it's a desk clock when nobody's talking to it)
|
||
- `listening` — visual feedback that the mic is hot
|
||
- `thinking` — Tatlock is working on a reply
|
||
- `speaking` — mouth/waveform animation synced to TTS playback
|
||
- **Audio capture**: dual mics through the ES7210 (hardware echo cancellation reference
|
||
from the playback path), 16 kHz 16-bit mono PCM.
|
||
- **Audio playback**: ES8311 codec → speaker. Plays PCM streamed from the gateway.
|
||
- **Transport**: a single WebSocket to the gateway carrying binary PCM frames plus JSON
|
||
control events (`state`, `transcript`, `reply_text`, errors). Device reconnects with
|
||
backoff; face shows a disconnected state when the gateway is unreachable.
|
||
- **Interaction**: phase 1 is touch-to-talk (tap the face). Phase 2 adds esp-sr WakeNet
|
||
wake word on the P4 so the interaction is hands-free.
|
||
|
||
Non-responsibilities: no STT, no TTS, no conversation state. If it's not rendering,
|
||
recording, or playing, it doesn't belong in firmware.
|
||
|
||
### 2. Gateway (`gateway/`) — container on tower-of-joy
|
||
|
||
A FastAPI service bridging device audio to Tatlock text:
|
||
|
||
1. Accepts the device WebSocket (`/ws/voice`).
|
||
2. Buffers inbound PCM until end-of-utterance (client-signalled in phase 1; VAD later).
|
||
3. **STT**: faster-whisper on the RTX 2080 Ti.
|
||
4. **Chat**: POSTs the transcript to Tatlock `/v1/chat/completions`
|
||
(`http://tatlock.schweitz.internal:8000`, OpenAI-compatible, streaming). Maintains the
|
||
conversation id so follow-ups have context.
|
||
5. **TTS**: Piper (fast, CPU-friendly, local) synthesizes the reply.
|
||
6. Streams reply PCM back to the device along with `reply_text` for on-screen display.
|
||
|
||
The gateway is stateless apart from in-flight conversations; it can restart freely.
|
||
|
||
### 3. Tatlock — existing backend (`/mnt/media/Projects/tatlock`)
|
||
|
||
Untouched by this project. DeskLock consumes its OpenAI-compatible API over the internal
|
||
LAN (port 8000, bypassing the Authentik-protected public route). If device auth is needed
|
||
later, the gateway holds the credential — never the firmware.
|
||
|
||
## WebSocket protocol (device ↔ gateway)
|
||
|
||
Binary frames: raw 16 kHz s16le mono PCM (mic upstream, TTS downstream).
|
||
Text frames: JSON control messages.
|
||
|
||
```
|
||
device → gateway: {"type": "utterance_start"}
|
||
device → gateway: <binary PCM frames>
|
||
device → gateway: {"type": "utterance_end"}
|
||
gateway → device: {"type": "state", "value": "thinking"}
|
||
gateway → device: {"type": "transcript", "text": "..."}
|
||
gateway → device: {"type": "reply_text", "text": "..."}
|
||
gateway → device: {"type": "audio_start", "sample_rate": 16000}
|
||
gateway → device: <binary PCM frames>
|
||
gateway → device: {"type": "audio_end"}
|
||
```
|
||
|
||
Keep this protocol documented here and mirrored in `firmware/` and `gateway/` constants —
|
||
it is the one contract between the two halves of the repo.
|
||
|
||
## Key decisions & rationale
|
||
|
||
- **ESP-IDF native (not Arduino/ESPHome)**: the P4 + MIPI-DSI + esp-sr stack is only
|
||
first-class in ESP-IDF; Waveshare recommends it, and the BSP targets it.
|
||
- **Gateway owns STT/TTS (not the device)**: the P4 could run small STT models, but
|
||
server-side whisper is dramatically better, and Piper voices beat embedded TTS. The
|
||
GPU is already there. Wake word is the only speech task that must be on-device.
|
||
- **Separate gateway (not extending Tatlock)**: keeps Tatlock's API text-only and clean;
|
||
audio concerns (codecs, VAD, streaming) stay at the edge. The gateway is also where a
|
||
future second endpoint (kitchen, office) would connect.
|
||
- **Monorepo**: the WS protocol couples firmware and gateway; versioning them together
|
||
avoids contract drift.
|