desklock/docs/architecture.md

DeskLock Architecture

Goal

A always-on, glanceable butler face in the living room. You speak to it; it relays your words to Tatlock and speaks the reply back, with a face that reflects what it's doing (idle, listening, thinking, speaking). It is deliberately a thin endpoint: all intelligence lives in Tatlock, all heavy audio processing lives in the gateway.

Components

1. Firmware (firmware/) — ESP32-P4

Responsibilities:

• Face rendering (LVGL 9 on the 800×800 round MIPI-DSI panel via the waveshare/esp32_p4_wifi6_touch_lcd_xc BSP). Face states:
  • idle — subtle animation + clock (it's a desk clock when nobody's talking to it)
  • listening — visual feedback that the mic is hot
  • thinking — Tatlock is working on a reply
  • speaking — mouth/waveform animation synced to TTS playback
• Audio capture: dual mics through the ES7210 (hardware echo cancellation reference from the playback path), 16 kHz 16-bit mono PCM.
• Audio playback: ES8311 codec → speaker. Plays PCM streamed from the gateway.
• Transport: a single WebSocket to the gateway carrying binary PCM frames plus JSON control events (state, transcript, reply_text, errors). Device reconnects with backoff; face shows a disconnected state when the gateway is unreachable.
• Interaction: phase 1 is touch-to-talk (tap the face). Phase 2 adds esp-sr WakeNet wake word on the P4 so the interaction is hands-free.

Non-responsibilities: no STT, no TTS, no conversation state. If it's not rendering, recording, or playing, it doesn't belong in firmware.

2. Gateway (gateway/) — container on tower-of-joy

A FastAPI service bridging device audio to Tatlock text:

1. Accepts the device WebSocket (/ws/voice).
2. Buffers inbound PCM until end-of-utterance (client-signalled in phase 1; VAD later).
3. STT: faster-whisper on the RTX 2080 Ti.
4. Chat: POSTs the transcript to Tatlock /v1/chat/completions (http://tatlock.schweitz.internal:8000, OpenAI-compatible, streaming). Maintains the conversation id so follow-ups have context.
5. TTS: Piper (fast, CPU-friendly, local) synthesizes the reply.
6. Streams reply PCM back to the device along with reply_text for on-screen display.

The gateway is stateless apart from in-flight conversations; it can restart freely.

3. Tatlock — existing backend (/mnt/media/Projects/tatlock)

Untouched by this project. DeskLock consumes its OpenAI-compatible API over the internal LAN (port 8000, bypassing the Authentik-protected public route). If device auth is needed later, the gateway holds the credential — never the firmware.

WebSocket protocol (device ↔ gateway)

Binary frames: raw 16 kHz s16le mono PCM (mic upstream, TTS downstream). Text frames: JSON control messages.

device → gateway:  {"type": "utterance_start"}
device → gateway:  <binary PCM frames>
device → gateway:  {"type": "utterance_end"}
gateway → device:  {"type": "state", "value": "thinking"}
gateway → device:  {"type": "transcript", "text": "..."}
gateway → device:  {"type": "reply_text", "text": "..."}
gateway → device:  {"type": "audio_start", "sample_rate": 16000}
gateway → device:  <binary PCM frames>
gateway → device:  {"type": "audio_end"}

Keep this protocol documented here and mirrored in firmware/ and gateway/ constants — it is the one contract between the two halves of the repo.

Key decisions & rationale

• ESP-IDF native (not Arduino/ESPHome): the P4 + MIPI-DSI + esp-sr stack is only first-class in ESP-IDF; Waveshare recommends it, and the BSP targets it.
• Gateway owns STT/TTS (not the device): the P4 could run small STT models, but server-side whisper is dramatically better, and Piper voices beat embedded TTS. The GPU is already there. Wake word is the only speech task that must be on-device.
• Separate gateway (not extending Tatlock): keeps Tatlock's API text-only and clean; audio concerns (codecs, VAD, streaming) stay at the edge. The gateway is also where a future second endpoint (kitchen, office) would connect.
• Monorepo: the WS protocol couples firmware and gateway; versioning them together avoids contract drift.