Initial scaffold: ESP32-P4 firmware + voice gateway for Tatlock endpoint
DeskLock gives the Tatlock butler a face and voice on a Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C round display in the living room. - firmware/: ESP-IDF project targeting esp32p4 with the Waveshare XC BSP - gateway/: FastAPI voice bridge (faster-whisper STT, Tatlock chat, Piper TTS) - docs/architecture.md: component design and device<->gateway WS protocol Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Claude Fable 5
@@ -0,0 +1,86 @@
|
||||
# DeskLock Architecture
|
||||
|
||||
## Goal
|
||||
|
||||
A always-on, glanceable butler face in the living room. You speak to it; it relays your
|
||||
words to Tatlock and speaks the reply back, with a face that reflects what it's doing
|
||||
(idle, listening, thinking, speaking). It is deliberately a *thin* endpoint: all
|
||||
intelligence lives in Tatlock, all heavy audio processing lives in the gateway.
|
||||
|
||||
## Components
|
||||
|
||||
### 1. Firmware (`firmware/`) — ESP32-P4
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- **Face rendering** (LVGL 9 on the 800×800 round MIPI-DSI panel via the
|
||||
`waveshare/esp32_p4_wifi6_touch_lcd_xc` BSP). Face states:
|
||||
- `idle` — subtle animation + clock (it's a desk clock when nobody's talking to it)
|
||||
- `listening` — visual feedback that the mic is hot
|
||||
- `thinking` — Tatlock is working on a reply
|
||||
- `speaking` — mouth/waveform animation synced to TTS playback
|
||||
- **Audio capture**: dual mics through the ES7210 (hardware echo cancellation reference
|
||||
from the playback path), 16 kHz 16-bit mono PCM.
|
||||
- **Audio playback**: ES8311 codec → speaker. Plays PCM streamed from the gateway.
|
||||
- **Transport**: a single WebSocket to the gateway carrying binary PCM frames plus JSON
|
||||
control events (`state`, `transcript`, `reply_text`, errors). Device reconnects with
|
||||
backoff; face shows a disconnected state when the gateway is unreachable.
|
||||
- **Interaction**: phase 1 is touch-to-talk (tap the face). Phase 2 adds esp-sr WakeNet
|
||||
wake word on the P4 so the interaction is hands-free.
|
||||
|
||||
Non-responsibilities: no STT, no TTS, no conversation state. If it's not rendering,
|
||||
recording, or playing, it doesn't belong in firmware.
|
||||
|
||||
### 2. Gateway (`gateway/`) — container on tower-of-joy
|
||||
|
||||
A FastAPI service bridging device audio to Tatlock text:
|
||||
|
||||
1. Accepts the device WebSocket (`/ws/voice`).
|
||||
2. Buffers inbound PCM until end-of-utterance (client-signalled in phase 1; VAD later).
|
||||
3. **STT**: faster-whisper on the RTX 2080 Ti.
|
||||
4. **Chat**: POSTs the transcript to Tatlock `/v1/chat/completions`
|
||||
(`http://tatlock.schweitz.internal:8000`, OpenAI-compatible, streaming). Maintains the
|
||||
conversation id so follow-ups have context.
|
||||
5. **TTS**: Piper (fast, CPU-friendly, local) synthesizes the reply.
|
||||
6. Streams reply PCM back to the device along with `reply_text` for on-screen display.
|
||||
|
||||
The gateway is stateless apart from in-flight conversations; it can restart freely.
|
||||
|
||||
### 3. Tatlock — existing backend (`/mnt/media/Projects/tatlock`)
|
||||
|
||||
Untouched by this project. DeskLock consumes its OpenAI-compatible API over the internal
|
||||
LAN (port 8000, bypassing the Authentik-protected public route). If device auth is needed
|
||||
later, the gateway holds the credential — never the firmware.
|
||||
|
||||
## WebSocket protocol (device ↔ gateway)
|
||||
|
||||
Binary frames: raw 16 kHz s16le mono PCM (mic upstream, TTS downstream).
|
||||
Text frames: JSON control messages.
|
||||
|
||||
```
|
||||
device → gateway: {"type": "utterance_start"}
|
||||
device → gateway: <binary PCM frames>
|
||||
device → gateway: {"type": "utterance_end"}
|
||||
gateway → device: {"type": "state", "value": "thinking"}
|
||||
gateway → device: {"type": "transcript", "text": "..."}
|
||||
gateway → device: {"type": "reply_text", "text": "..."}
|
||||
gateway → device: {"type": "audio_start", "sample_rate": 16000}
|
||||
gateway → device: <binary PCM frames>
|
||||
gateway → device: {"type": "audio_end"}
|
||||
```
|
||||
|
||||
Keep this protocol documented here and mirrored in `firmware/` and `gateway/` constants —
|
||||
it is the one contract between the two halves of the repo.
|
||||
|
||||
## Key decisions & rationale
|
||||
|
||||
- **ESP-IDF native (not Arduino/ESPHome)**: the P4 + MIPI-DSI + esp-sr stack is only
|
||||
first-class in ESP-IDF; Waveshare recommends it, and the BSP targets it.
|
||||
- **Gateway owns STT/TTS (not the device)**: the P4 could run small STT models, but
|
||||
server-side whisper is dramatically better, and Piper voices beat embedded TTS. The
|
||||
GPU is already there. Wake word is the only speech task that must be on-device.
|
||||
- **Separate gateway (not extending Tatlock)**: keeps Tatlock's API text-only and clean;
|
||||
audio concerns (codecs, VAD, streaming) stay at the edge. The gateway is also where a
|
||||
future second endpoint (kitchen, office) would connect.
|
||||
- **Monorepo**: the WS protocol couples firmware and gateway; versioning them together
|
||||
avoids contract drift.
|
||||
Reference in New Issue
Block a user