From 42b988b7efd524403f4acf43112519fb7841b6e0 Mon Sep 17 00:00:00 2001 From: Jeroen Schweitzer Date: Tue, 14 Jul 2026 17:32:16 +0200 Subject: [PATCH] Add face design: ASCII expressions + matrix rain simulator, wait cues, fresh latency numbers - sim/face/index.html: browser simulator of the 800x800 round panel with six states (idle/listening/pensive/effort/speaking/error), state-driven rain density, blink/talk/thought animations, idle clock - effort state gets hard-required wait cues: orbiting bezel arc, elapsed counter, max rain (Tatlock turns run 10-25s) - rain driven by setInterval, not rAF: renders under the screenshot tool's --virtual-time-budget and mirrors LVGL lv_timer - architecture.md: Face design contract (state table, protocol mapping, LVGL port notes) + latency table updated to GPU-era benchmarks (Steward ~6s warm, full flow 11-25s; old CPU-era figures retired) Co-Authored-By: Claude Fable 5 --- AGENTS.md | 8 ++ README.md | 3 + docs/architecture.md | 72 +++++++++++--- sim/face/index.html | 231 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 299 insertions(+), 15 deletions(-) create mode 100644 sim/face/index.html diff --git a/AGENTS.md b/AGENTS.md index 14b77fd..9171c12 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -22,6 +22,14 @@ The device and gateway speak a WebSocket protocol defined in `docs/architecture. **That doc is the contract** — update it in the same change as any protocol edit on either side. +The face (black screen, ASCII glyph expressions, matrix rain as activity signal) is +designed in `sim/face/index.html` — the design source of truth — and specified in the +"Face design" section of `docs/architecture.md`. Change the sim and the doc together; +the LVGL implementation follows them. Verify sim changes visually with +`~/bin/claude-screenshot` (note: the tool uses `--virtual-time-budget`, which starves +`requestAnimationFrame` — drive sim animation with `setInterval`, which also mirrors +LVGL timers). + ## Hard rules - **Keep the firmware thin.** No STT, no TTS, no conversation logic on the device. diff --git a/README.md b/README.md index 9a64af5..7bc7388 100644 --- a/README.md +++ b/README.md @@ -54,6 +54,9 @@ See [docs/architecture.md](docs/architecture.md) for the full design. - `firmware/` — ESP-IDF (C, LVGL 9) application for the ESP32-P4 - `gateway/` — Python FastAPI voice gateway, deployed as a container on tower-of-joy +- `sim/face/` — browser simulator of the face (design source of truth; serve with + `python3 -m http.server` and open `index.html`, or use `?state=…&nochrome=1` for + screenshots) - `docs/` — architecture and design notes ## Roadmap diff --git a/docs/architecture.md b/docs/architecture.md index 7ee01a9..789097c 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -39,12 +39,8 @@ the **speech layer** (Speaches) owns the actual STT/TTS models on the GPU. Responsibilities: - **Face rendering** (LVGL 9 on the 800×800 round MIPI-DSI panel via the - `waveshare/esp32_p4_wifi6_touch_lcd_xc` BSP). Face states: - - `idle` — subtle animation + clock (it's a desk clock when nobody's talking to it) - - `listening` — visual feedback that the mic is hot - - `thinking` — Tatlock is working on a reply (this state earns its keep; see - [Latency budget](#latency-budget--streaming)) - - `speaking` — mouth/waveform animation synced to TTS playback + `waveshare/esp32_p4_wifi6_touch_lcd_xc` BSP). Six expression states — see + [Face design](#face-design) for the visual contract. - **Audio capture**: dual mics through the ES7210 (hardware echo cancellation reference from the playback path), 16 kHz 16-bit mono PCM. - **Audio playback**: ES8311 codec → speaker. Plays PCM streamed from the gateway. @@ -123,27 +119,73 @@ Untouched by this project. DeskLock consumes its OpenAI-compatible API over the LAN (port 8000, bypassing the Authentik-protected public route). If device auth is needed later, the gateway holds the credential — never the firmware. +## Face design + +**Aesthetic**: pure black screen; a face drawn from ASCII/terminal glyphs in green +phosphor (`#adffc8` face, dimmer greens for secondary info); Matrix-style digital rain +whose **density encodes activity** — barely-there drips when idle, a downpour while +Tatlock works. No bitmaps, no skeuomorphism: glyphs only. + +**Source of truth**: `sim/face/index.html` — a self-contained browser simulator of the +800×800 round panel. Design changes land there first, get approved visually, then get +ported to LVGL. The `STATES` table in the sim defines the contract: + +| State | Eyes | Mouth | Rain | Extra cues | +|-------|------|-------|------|------------| +| `idle` | `- -` | `\_/` | 2 slow streams | clock (HH:MM), breathing bob, blinks | +| `listening` | `O O` | `o` | 16 streams | blinks | +| `pensive` | `· ·` | `~` | 7 streams | cycling `...` thought dots | +| `effort` | `> <` | `~` | 40 fast streams | **orbit arc on bezel + `[ Ns ]` elapsed counter**, face jitter | +| `speaking` | `^ ^` | cycles `o O - O = o` | 14 streams | mouth animates ~150 ms/frame | +| `error` | `x x` | `-` | none (rain dies) | face dims to 45% | + +**Wait cues are a hard requirement** (user-stated): Tatlock turns take 10–25 s, so +`effort` must always show *alive-and-working* signals — the orbiting bezel arc, the +elapsed-seconds counter, and max rain. Never a bare static face during a wait, and no +fake progress bars — only honest cues. + +**Protocol → face mapping**: gateway `state: thinking` → `effort`; transcription and +other short local waits → `pensive`; `listening`/`speaking` map 1:1; WebSocket +disconnected → `error`; otherwise `idle`. + +**LVGL port notes** (for phase 2): + +- Drive everything from fixed-step `lv_timer`s (~30 fps rain tick) — the sim + deliberately uses `setInterval`, not `requestAnimationFrame`, to mirror this. +- Rain: `lv_canvas` (or a pooled label grid) with per-frame fade; orbit arc = `lv_arc`. +- Fonts: generate a large monospace glyph font including the katakana subset used in + `GLYPHS` via `lv_font_conv`; the built-in `unscii` fonts are too small for 800 px. +- The sim's text glow (`text-shadow`) is browser flair — the device renders flat glyphs. + ## Latency budget & streaming -Measured/known numbers that shape the design: +Measured/known numbers that shape the design (Tatlock figures per tatlock CLAUDE.md, +GPU-resident benchmarks of 2026-07-14, gemma4:e2b at ~100 tok/s): | Stage | Cost | |-------|------| | STT (whisper `small`, GPU) | a few hundred ms for a ~5 s utterance | | TTS (Piper/Kokoro) | faster than realtime | -| **Tatlock, full local flow (gemma4)** | **~35 s Steward analysis warm; ~2 min end-to-end** (per tatlock CLAUDE.md) | +| Tatlock Steward analysis | ~6 s warm | +| **Tatlock, full local flow** | **11–25 s end-to-end** (librarian-routed ~20–25 s) | +| Tatlock cold start (>2 h idle) | +~8 s (`OLLAMA_KEEP_ALIVE=2h`) | -Speech is not the bottleneck — **Tatlock is**, by two orders of magnitude. Constraints -this imposes: +(Older "~35 s Steward / ~2 min flow" figures were from a CPU-only driver-mismatch era — +do not plan against them.) + +Speech is not the bottleneck — **Tatlock is**, by one to two orders of magnitude. +Constraints this imposes: 1. **The gateway must consume Tatlock's streaming response and synthesize sentence-by-sentence**, forwarding audio as each sentence is ready. The device starts - speaking after the first sentence instead of waiting for the full reply. The WS - protocol already supports this: one `audio_start` … PCM … `audio_end` envelope with - chunks arriving as they're synthesized — the device just plays a continuous stream. + speaking after the first sentence instead of waiting for the full reply — with + streaming, first audio should land roughly at Steward-time + first-sentence-time, + well under the 11–25 s full-flow figure. The WS protocol already supports this: one + `audio_start` … PCM … `audio_end` envelope with chunks arriving as they're + synthesized — the device just plays a continuous stream. 2. **The `thinking` face state is a first-class feature**, not decoration — it's what - makes a long Tatlock turn feel intentional instead of broken. Consider progress cues - (e.g. surface Tatlock's reasoning summaries on-screen) later. + makes a 10–25 s Tatlock turn feel intentional instead of broken. Consider progress + cues (e.g. surface Tatlock's reasoning summaries on-screen) later. 3. A **fast lane** may eventually be needed: MultiNet on-device commands for instant home-automation phrases, and/or a low-latency intent path in Tatlock itself. Out of scope for now, but don't design it out. diff --git a/sim/face/index.html b/sim/face/index.html new file mode 100644 index 0000000..18d09a6 --- /dev/null +++ b/sim/face/index.html @@ -0,0 +1,231 @@ + + + + + +DeskLock Face — design simulator + + + + +
DESKLOCK · FACE SIMULATOR · 800×800
+
+
+ +
+
+ 

+        

+        

+      

+      

+      

+      

+    

+    

+  

+
+
+
+