# DeskLock Architecture

## Goal

A always-on, glanceable butler face in the living room. You speak to it; it relays your
words to Tatlock and speaks the reply back, with a face that reflects what it's doing
(idle, listening, thinking, speaking). It is deliberately a *thin* endpoint: all
intelligence lives in Tatlock, all heavy audio processing lives in the gateway.

## Components

### 1. Firmware (`firmware/`) — ESP32-P4

Responsibilities:

- **Face rendering** (LVGL 9 on the 800×800 round MIPI-DSI panel via the
  `waveshare/esp32_p4_wifi6_touch_lcd_xc` BSP). Face states:
  - `idle` — subtle animation + clock (it's a desk clock when nobody's talking to it)
  - `listening` — visual feedback that the mic is hot
  - `thinking` — Tatlock is working on a reply
  - `speaking` — mouth/waveform animation synced to TTS playback
- **Audio capture**: dual mics through the ES7210 (hardware echo cancellation reference
  from the playback path), 16 kHz 16-bit mono PCM.
- **Audio playback**: ES8311 codec → speaker. Plays PCM streamed from the gateway.
- **Transport**: a single WebSocket to the gateway carrying binary PCM frames plus JSON
  control events (`state`, `transcript`, `reply_text`, errors). Device reconnects with
  backoff; face shows a disconnected state when the gateway is unreachable.
- **Interaction**: phase 1 is touch-to-talk (tap the face). Phase 2 adds esp-sr WakeNet
  wake word on the P4 so the interaction is hands-free.

Non-responsibilities: no STT, no TTS, no conversation state. If it's not rendering,
recording, or playing, it doesn't belong in firmware.

### 2. Gateway (`gateway/`) — container on tower-of-joy

A FastAPI service bridging device audio to Tatlock text:

1. Accepts the device WebSocket (`/ws/voice`).
2. Buffers inbound PCM until end-of-utterance (client-signalled in phase 1; VAD later).
3. **STT**: faster-whisper on the RTX 2080 Ti.
4. **Chat**: POSTs the transcript to Tatlock `/v1/chat/completions`
   (`http://tatlock.schweitz.internal:8000`, OpenAI-compatible, streaming). Maintains the
   conversation id so follow-ups have context.
5. **TTS**: Piper (fast, CPU-friendly, local) synthesizes the reply.
6. Streams reply PCM back to the device along with `reply_text` for on-screen display.

The gateway is stateless apart from in-flight conversations; it can restart freely.

### 3. Tatlock — existing backend (`/mnt/media/Projects/tatlock`)

Untouched by this project. DeskLock consumes its OpenAI-compatible API over the internal
LAN (port 8000, bypassing the Authentik-protected public route). If device auth is needed
later, the gateway holds the credential — never the firmware.

## WebSocket protocol (device ↔ gateway)

Binary frames: raw 16 kHz s16le mono PCM (mic upstream, TTS downstream).
Text frames: JSON control messages.

```
device → gateway:  {"type": "utterance_start"}
device → gateway:  <binary PCM frames>
device → gateway:  {"type": "utterance_end"}
gateway → device:  {"type": "state", "value": "thinking"}
gateway → device:  {"type": "transcript", "text": "..."}
gateway → device:  {"type": "reply_text", "text": "..."}
gateway → device:  {"type": "audio_start", "sample_rate": 16000}
gateway → device:  <binary PCM frames>
gateway → device:  {"type": "audio_end"}
```

Keep this protocol documented here and mirrored in `firmware/` and `gateway/` constants —
it is the one contract between the two halves of the repo.

## Key decisions & rationale

- **ESP-IDF native (not Arduino/ESPHome)**: the P4 + MIPI-DSI + esp-sr stack is only
  first-class in ESP-IDF; Waveshare recommends it, and the BSP targets it.
- **Gateway owns STT/TTS (not the device)**: the P4 could run small STT models, but
  server-side whisper is dramatically better, and Piper voices beat embedded TTS. The
  GPU is already there. Wake word is the only speech task that must be on-device.
- **Separate gateway (not extending Tatlock)**: keeps Tatlock's API text-only and clean;
  audio concerns (codecs, VAD, streaming) stay at the edge. The gateway is also where a
  future second endpoint (kitchen, office) would connect.
- **Monorepo**: the WS protocol couples firmware and gateway; versioning them together
  avoids contract drift.