desklock/docs/architecture.md

# DeskLock Architecture

## Goal

An always-on, glanceable butler face in the living room. You speak to it; it relays your
words to Tatlock and speaks the reply back, with a face that reflects what it's doing
(idle, listening, thinking, speaking). It is deliberately a *thin* endpoint: all
intelligence lives in Tatlock, all heavy audio processing lives server-side on
tower-of-joy. Everything is local — no audio, transcript, or reply ever leaves the LAN.

## System overview

```
┌──────────────────────┐  WebSocket: PCM audio + JSON events
│  DeskLock device     │◄───────────────────────────────────┐
│  (ESP32-P4)          │                                    │
│  • LVGL face         │     ┌──────────────────────────────┴───────────┐
│  • touch / wake word │     │  DeskLock Gateway  (container, :8600)    │
│  • mic capture + AEC │     │  thin orchestrator — no ML dependencies  │
│  • TTS playback      │     └───────┬──────────────────┬───────────────┘
└──────────────────────┘             │                  │  OpenAI-format HTTP
                                     │                  ▼
                          HTTP (LAN) │       ┌─────────────────────────────┐
                                     ▼       │  Speaches  (container, GPU) │
                     ┌────────────────────┐  │  • STT: faster-whisper      │
                     │  Tatlock (butler)  │  │  • TTS: Kokoro / Piper      │
                     │  container name:   │  │  also usable by Open WebUI, │
                     │  tatlock:8000      │  │  Home Assistant, …          │
                     └────────────────────┘  └─────────────────────────────┘
```

Tatlock stays a text-only brain. The **gateway** orchestrates Tatlock's ears and mouth;
the **speech layer** (Speaches) owns the actual STT/TTS models on the GPU.

## Components

### 1. Firmware (`firmware/`) — ESP32-P4

Responsibilities:

- **Face rendering** (LVGL 9 on the 800×800 round MIPI-DSI panel via the
  `waveshare/esp32_p4_wifi6_touch_lcd_xc` BSP). Six expression states — see
  [Face design](#face-design) for the visual contract.
- **Audio capture**: dual mics through the ES7210 (hardware echo cancellation reference
  from the playback path), 16 kHz 16-bit mono PCM.
- **Audio playback**: ES8311 codec → speaker. Plays PCM streamed from the gateway.
- **Transport**: a single WebSocket to the gateway carrying binary PCM frames plus JSON
  control events (`state`, `transcript`, `reply_text`, errors). Device reconnects with
  backoff; face shows a disconnected state when the gateway is unreachable.

**On-device speech processing — what runs on the P4 and what deliberately doesn't.**
The P4 (dual RISC-V @ 400 MHz, 32 MB PSRAM) has a hard ceiling; the split is:

| On-device (planned) | Why |
|---------------------|-----|
| Wake word — esp-sr WakeNet (phase 2) | Must be local: always-listening audio should never leave the device until the wake word fires |
| Voice-activity detection (end-of-utterance) | Removes tap-to-stop; cheap on-device |
| Echo cancellation — ES7210 hardware | Enables barge-in while TTS is playing |
| esp-sr MultiNet fixed commands (optional, later) | ~200-phrase closed vocabulary recognized entirely on-device — instant "lights off"-style commands with zero round trip |

Full open-vocabulary STT on-device is **out of scope permanently**: even whisper-tiny
needs hundreds of MB and orders of magnitude more compute than the P4 offers. Anything
open-ended goes to the speech layer.

Non-responsibilities: no STT, no TTS, no conversation state. If it's not rendering,
recording, or playing, it doesn't belong in firmware.

### 2. Gateway (`gateway/`) — container on tower-of-joy, port 8600

A FastAPI service bridging device audio to Tatlock text. It owns *orchestration*, not
models — the container stays a slim pure-Python image with no CUDA/ML dependencies:

1. Accepts the device WebSocket (`/ws/voice`).
2. Buffers inbound PCM until end-of-utterance (client-signalled in phase 1; VAD later).
3. **STT**: POST to Speaches `/v1/audio/transcriptions`.
4. **Chat**: POST the transcript to Tatlock `/v1/chat/completions`
   (`http://tatlock:8000`, OpenAI-compatible, **streaming**),
   maintaining the conversation history so follow-ups have context.
5. **TTS**: as Tatlock's token stream completes each sentence, POST it to Speaches
   `/v1/audio/speech` and forward the PCM immediately — see
   [Latency budget](#latency-budget--streaming).

`stt.py` / `tts.py` are pluggable backends selected by config
(`DESKLOCK_STT_BACKEND` / `DESKLOCK_TTS_BACKEND`):

- `speaches` (default) — OpenAI-format HTTP to the shared speech container.
- `embedded` — in-process faster-whisper / Piper. Kept as a fallback so the gateway can
  run standalone (dev on a laptop, speech container down), at the cost of a fat image.

The gateway is stateless apart from in-flight conversations; it can restart freely.

### 3. Speech layer — Speaches (container, GPU)

[Speaches](https://github.com/speaches-ai/speaches) (successor to faster-whisper-server)
is a self-hosted, OpenAI-API-compatible speech server: STT via faster-whisper, TTS via
Kokoro/Piper, dynamic model load/offload with a TTL, and a `/v1/realtime` WebSocket API
we may adopt later for streaming transcription.

- **Deployed 2026-07-14**: `ghcr.io/speaches-ai/speaches:latest-cuda` on host port
  **8601**, with `Systran/faster-whisper-small` (STT) and
  `speaches-ai/Kokoro-82M-v1.0-ONNX` (TTS — Kokoro is natively 24 kHz, but the
  gateway requests the 16 kHz device contract directly via Speaches' `sample_rate`
  extension, verified live; default voice `bm_george`, en-GB male). LAN-only like the
  Tatlock internal route — do not expose through NPM without auth. Register in
  `CONTAINERS.md`.
- **Measured** (live round trip, warm, 2026-08-07): STT ~0.30 s for a ~4.8 s utterance;
  TTS ~0.24 s for a ~4.5 s sentence (real-time factor ~0.05). The first call after an
  idle gap costs ~1.2 s; a full cold start after model TTL offload adds ~4 s.
- **Why a shared layer instead of models inside the gateway**: one GPU-resident model
  instance serves the whole homelab. Open WebUI is currently configured with
  `AUDIO_STT_ENGINE=openai` / `AUDIO_TTS_ENGINE=openai` (OpenAI *cloud*) — pointing its
  audio base URL at Speaches makes it fully local with a config change. Home Assistant
  can share it too. Meanwhile the gateway image needs no CUDA and rebuilds in seconds.
- **VRAM budget**: RTX 2080 Ti, 11,264 MiB, shared with Ollama. As of 2026-08-07 the
  steady state is ~4.9 GB used / ~5.9 GB free with everything resident: `gemma4:e2b`
  1.9 GB and `nomic-embed-text` 0.3 GB (both pinned), whisper `small` int8 <1 GB,
  Kokoro a few hundred MB. Speaches' model TTL offload keeps idle pressure near zero.
  **This budget is not slack — it is the constraint.** On 2026-08-07 Tatlock was
  deployed against `mistral-nemo:latest` (9.3 GB, 2 h keep-alive), which left 7 MiB
  free and made every transcription fail with `CUDA failed with error out of memory`
  while the Speaches container still reported healthy. Keep Tatlock's model at or below
  ~4 GB resident, and check `nvidia-smi` free VRAM before changing it. If contention
  ever bites anyway, faster-whisper `small` on CPU is an acceptable fallback (int8, a
  few seconds per utterance).

### 4. Tatlock — existing backend (`/mnt/media/Projects/tatlock`)

Untouched by this project. DeskLock consumes its OpenAI-compatible API over the internal
LAN (port 8000, bypassing the Authentik-protected public route). If device auth is needed
later, the gateway holds the credential — never the firmware.

## Face design

**Aesthetic**: pure black screen; a face drawn from ASCII/terminal glyphs in green
phosphor (`#adffc8` face, dimmer greens for secondary info); Matrix-style digital rain
whose **density encodes activity** — barely-there drips when idle, a downpour while
Tatlock works. No bitmaps, no skeuomorphism: glyphs only.

**Source of truth**: `sim/face/index.html` — a self-contained browser simulator of the
800×800 round panel. Design changes land there first, get approved visually, then get
ported to LVGL. The `STATES` table in the sim defines the contract:

| State | Eyes | Mouth | Rain | Extra cues |
|-------|------|-------|------|------------|
| `idle` | `-   -` | `\_/` | 2 slow streams | clock (HH:MM), breathing bob, blinks |
| `listening` | `O   O` | `o` | 16 streams | blinks |
| `pensive` | `·   ·` | `~` | 7 streams | cycling `...` thought dots |
| `effort` | `>   <` | `~` | 40 fast streams | **orbit arc on bezel + `[ Ns ]` elapsed counter**, face jitter |
| `speaking` | `^   ^` | cycles `o O - O = o` | 14 streams | mouth animates ~150 ms/frame |
| `rage` | — | — | 34 fast streams | 3-frame kaomoji loop through the eyes slot: `(°□°) ┬─┬` → `(╯°□°)╯︵ ┻━┻` → `┬─┬ ノ( º_º ノ)` — flips the table, then composes itself and puts it back |
| `error` | `x   x` | `-` | none (rain dies) | face dims to 45% |

**Sound signature**: the cathedral gong (`play_boot_gong` in firmware — 220 Hz
inharmonic partial stack, feedback-comb reflections, ~-7 dBFS) is the approved house
sound: "audible and butler-non-intrusive." Plays at boot; planned as the
wake-from-`dormant` sound. Tune by adjective: tail = `tau`s, cathedral size = echo
delays, depth = fundamental, presence = `GONG_PEAK`/`GONG_VOLUME`.

**Wait cues are a hard requirement** (user-stated): Tatlock turns take 10–25 s, so
`effort` must always show *alive-and-working* signals — the orbiting bezel arc, the
elapsed-seconds counter, and max rain. Never a bare static face during a wait, and no
fake progress bars — only honest cues.

**Protocol → face mapping**: gateway `state: thinking` → `effort`; transcription and
other short local waits → `pensive`; `listening`/`speaking` map 1:1; an in-flight
request failure (STT/Tatlock/TTS error) → `rage` for a few loops, then `idle`;
WebSocket disconnected → `error` (quiet, persistent); otherwise `idle`.

**LVGL port notes** (for phase 2):

- Drive everything from fixed-step `lv_timer`s (~30 fps rain tick) — the sim
  deliberately uses `setInterval`, not `requestAnimationFrame`, to mirror this.
- Rain: `lv_canvas` (or a pooled label grid) with per-frame fade; orbit arc = `lv_arc`.
- Fonts: generate a large monospace glyph font including the katakana subset used in
  `GLYPHS` via `lv_font_conv`; the built-in `unscii` fonts are too small for 800 px.
  The `rage` frames additionally need `╯ ︵ ┻ ━ ┬ ─ ノ ° □ º` in the subset.
- The sim's text glow (`text-shadow`) is browser flair — the device renders flat glyphs.

## Power management (prime concern)

User requirement: the device idles on a wall 95%+ of its life — low power when nothing
is happening is a first-class design goal, not a phase-5 nicety. The face state machine
is therefore built around a **power ladder** from day one:

| Power state | Backlight | Rendering | CPU | Entered when |
|-------------|-----------|-----------|-----|--------------|
| `active` | 100% | full animation | full clock | conversation in progress (listening→speaking) |
| `ambient` | ~35% | idle face, sparse rain | DFS enabled | idle, but activity in the last few minutes |
| `dormant` | off (or ≤5%) | **no redraws** — render loop parked | min clock via DFS | no voice/touch for N min (default 10) |
| `night` | off, panel sleep | none | min clock | schedule or "goodnight" command |

Levers, in order of impact:

1. **Backlight** — this is an IPS LCD: black pixels still burn backlight (unlike OLED),
   so brightness is the dominant lever. `bsp_display_brightness_set()` drives it.
2. **Render idleness** — rain off and animations parked means LVGL stops producing
   frames, which is what lets DFS actually reach its floor.
3. **DFS / power management** (`CONFIG_PM_ENABLE`) — automatic frequency scaling when
   tasks are quiet. Note the MIPI-DSI constraint below.
4. **Radio** — the C6 runs Wi-Fi modem power-save; the gateway WebSocket widens its
   ping interval when the device reports `dormant`.

Wake triggers (any → `ambient`/`active`): wake word (phase 5), touch (from phase 2),
local VAD "someone is speaking" pre-warm, a gateway-initiated event (butler wants to
say something), scheduled morning end of `night`.

**Hard edges — what limits how low we can go:**

- **The hands-free promise sets the power floor.** Wake word requires mics + the AFE
  pipeline running continuously; deep sleep is permanently off the table while the
  device promises to answer its name. The floor is "CPU lightly loaded at min clock,
  radios in power-save, backlight off."
- **DSI needs clocks while the panel is active** — the deepest CPU savings only unlock
  in `dormant`/`night` when the panel stops being refreshed (panel sleep / blank).
- **Wake latency budget: ≤ ~300 ms** from trigger to visible face (backlight ramp +
  first render). Anything slower reads as "it's off," which kills the butler illusion.
- **Touch stays powered** in all states except possibly `night` — its idle draw is
  negligible and tap-to-wake must always work.
- **No invented numbers**: actual draw gets measured with a USB power meter at each
  phase; working target is `dormant` ≤ ⅓ of `active`. (Always-on device: every watt
  saved ≈ 9 kWh/year.)

Implementation order: backlight dimming + `dormant` timeout + touch wake land in
**phase 2** with the face state machine (timeout-driven); voice-linked triggers upgrade
it in phase 5.

## Latency budget & streaming

Measured 2026-08-07 against the deployed stack (`gemma4:e2b` at ~95 tok/s, GPU-resident):

| Stage | Cost |
|-------|------|
| STT (Speaches whisper `small`) | ~0.30 s warm, for ~4.8 s of audio |
| TTS (Speaches Kokoro) | ~0.24 s warm, for ~4.5 s of audio (RTF ~0.05) |
| **Tatlock, full local flow** | **~10–13 s end-to-end** for simple turns |
| Tatlock cold model load | +~36 s — avoided while the model is pinned |

A Tatlock turn costs **3 sequential Ollama calls** (Steward routing → tool orchestration →
butler-tone synthesis) and ~710 generated tokens even for "what is 61 plus 12?". Most of
that is the model's own reasoning: gemma4 thinks by default, and the effort is spent three
times per turn.

(Older figures — "~35 s Steward / ~2 min flow" from the CPU-only era, and "11–25 s full
flow" from 2026-07-14 — are superseded. Do not plan against them.)

Speech is not the bottleneck — **Tatlock is**, by one to two orders of magnitude.
Constraints this imposes:

1. **The gateway must consume Tatlock's streaming response and synthesize
   sentence-by-sentence**, forwarding audio as each sentence is ready. The device starts
   speaking after the first sentence instead of waiting for the full reply — with
   streaming, first audio should land roughly at Steward-time + first-sentence-time,
   well under the ~10–13 s full-flow figure. The WS protocol already supports this: one
   `audio_start` … PCM … `audio_end` envelope with chunks arriving as they're
   synthesized — the device just plays a continuous stream.
2. **The `thinking` face state is a first-class feature**, not decoration — it's what
   makes a ~10 s Tatlock turn feel intentional instead of broken. Consider progress
   cues (e.g. surface Tatlock's reasoning summaries on-screen) later.
3. A **fast lane** may eventually be needed: MultiNet on-device commands for instant
   home-automation phrases, and/or a low-latency intent path in Tatlock itself. Out of
   scope for now, but don't design it out.

## WebSocket protocol (device ↔ gateway)

Binary frames: raw 16 kHz s16le mono PCM (mic upstream, TTS downstream).
Text frames: JSON control messages.

```
device → gateway:  {"type": "utterance_start"}
device → gateway:  <binary PCM frames>
device → gateway:  {"type": "utterance_end"}
gateway → device:  {"type": "state", "value": "thinking"}
gateway → device:  {"type": "transcript", "text": "..."}
gateway → device:  {"type": "reply_text", "text": "..."}
gateway → device:  {"type": "audio_start", "sample_rate": 16000}
gateway → device:  <binary PCM frames>   (may arrive sentence-by-sentence; play as a stream)
gateway → device:  {"type": "audio_end"}

gateway → device:  {"type": "command", "action": "volume_up"}   (LLM-bypass; see below)
```

`command` (gateway → device) is an **alternative to the reply path**: when the
gateway recognizes a simple device command in the transcript (volume/mute), it
sends a `command` instead of calling Tatlock — no `reply_text`/audio — then returns
to `idle`. Actions: `volume_up`, `volume_down`, `mute`, `unmute`, and `volume_set`
with an extra `"level"` field (0–11, the on-device volume scale). Matched by the
gateway's `commands.py`; applied on the device in `gw_client.c` → `face.c`.

Planned additions (documented before implemented, here first):

- `reply_delta` (gateway → device): incremental reply text for on-screen streaming while
  audio is synthesized.
- An interrupt event (device → gateway) for barge-in during playback (phase 4).

Keep this protocol documented here and mirrored in `firmware/` and `gateway/` constants —
it is the one contract between the two halves of the repo.

## Key decisions & rationale

- **ESP-IDF native (not Arduino/ESPHome)**: the P4 + MIPI-DSI + esp-sr stack is only
  first-class in ESP-IDF; Waveshare recommends it, and the BSP targets it.
- **Server-side STT/TTS, device does wake word + VAD + AEC only**: server whisper is
  dramatically better than anything embeddable, the GPU is already there, and the P4
  physically can't run open-vocabulary STT. Wake word must be on-device (privacy: no
  audio leaves the device until it fires).
- **STT/TTS as a shared Speaches service (not embedded in the gateway)**: one model
  instance for the whole homelab (DeskLock, Open WebUI, potentially HA), slim gateway
  image, models upgradable independently. `embedded` backend retained as a dev/fallback
  mode.
  - *Rejected — Wyoming protocol containers* (`wyoming-faster-whisper`/`wyoming-piper`):
    native to Home Assistant's ecosystem, but Tatlock and Open WebUI already speak
    OpenAI format, so Speaches fits the lab better. Revisit only if HA Assist becomes a
    first-class consumer.
  - *Rejected — cloud STT/TTS*: violates the local-first premise; also adds WAN latency
    and per-minute cost.
- **Separate gateway (not extending Tatlock)**: keeps Tatlock's API text-only and clean;
  audio concerns (codecs, VAD, streaming, sentence segmentation) stay at the edge. The
  gateway is also where a future second endpoint (kitchen, office) would connect.
- **Monorepo**: the WS protocol couples firmware and gateway; versioning them together
  avoids contract drift.

## Additional endpoints — sauron (planned)

**sauron** is an old iMac (Linux, text-only console) that will run the same UX as a
second butler endpoint plus an ops console. Because the gateway protocol is
endpoint-agnostic and each WS connection gets its own conversation, extra endpoints
are architecturally free.

- **Client**: a terminal UI (`clients/sauron/`, Python + curses/textual) — the face
  design is already ASCII, so a TTY renders it natively: glyph face states,
  character-cell matrix rain, green-on-black. Audio via ALSA (arecord/aplay-level,
  16 kHz mono PCM), same WebSocket protocol, same state machine.
- **Screensaver model** (user-confirmed): the face+voice layer is the *idle mode* —
  full-screen butler when nobody's working. The *workspace mode* is an SSH ops console:
  live stats and remote control of tower-of-joy and forge. Any keypress drops from face
  to console; idle timeout (and wake word later) raises the face again. Voice stays
  available in both modes.
- **Not started** — planned after the device reaches phase 4/5. No browser/kiosk stack
  needed unless we later want the glow.

## CI & deployment

Gitea Actions (`.gitea/workflows/build.yml`), following the tatlock/tatlock-ui pattern:

- **Every push to `main`**: lint + tests for the gateway (Python 3.12).
- **Version tags (`v0.1.0`, …)**: tests, then build `gateway/` into
  `git.schweitz.net/jpmschweitzer/desklock-gateway:{latest,tag}`, push to the
  Gitea registry, create a release, and trigger Watchtower to roll the running
  container.
- Required repo/org secrets: `REGISTRY_USER`, `REGISTRY_PASSWORD`,
  `WATCHTOWER_HTTP_API_TOKEN` (same trio tatlock uses).
- The gateway is a service in the **`tatlock-ui` Portainer stack**
  (`system-admin-toj/containers/stacks/tatlock-ui.yml`, registered in
  `CONTAINERS.md`): it shares `docker-dataplane` with Speaches (service-name URL
  `http://speaches:8000`) and reaches the host-run Tatlock via LAN IP.

Firmware is not containerized: it's flashed over USB (`idf.py flash`), with OTA planned
for phase 5.

**Versioning**: `0.x` while interfaces are still moving — roughly one minor bump per
roadmap phase (`0.1.x` gateway server-side, `0.2.x` first device firmware, `0.3.x`
hands-free). **`v1.0.0` is reserved for the wall milestone**: the device mounted in the
living room, talking to Tatlock end to end.