Add face design: ASCII expressions + matrix rain simulator, wait cues, fresh latency numbers
- sim/face/index.html: browser simulator of the 800x800 round panel with six states (idle/listening/pensive/effort/speaking/error), state-driven rain density, blink/talk/thought animations, idle clock - effort state gets hard-required wait cues: orbiting bezel arc, elapsed counter, max rain (Tatlock turns run 10-25s) - rain driven by setInterval, not rAF: renders under the screenshot tool's --virtual-time-budget and mirrors LVGL lv_timer - architecture.md: Face design contract (state table, protocol mapping, LVGL port notes) + latency table updated to GPU-era benchmarks (Steward ~6s warm, full flow 11-25s; old CPU-era figures retired) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Claude Fable 5
+57
-15
@@ -39,12 +39,8 @@ the **speech layer** (Speaches) owns the actual STT/TTS models on the GPU.
|
||||
Responsibilities:
|
||||
|
||||
- **Face rendering** (LVGL 9 on the 800×800 round MIPI-DSI panel via the
|
||||
`waveshare/esp32_p4_wifi6_touch_lcd_xc` BSP). Face states:
|
||||
- `idle` — subtle animation + clock (it's a desk clock when nobody's talking to it)
|
||||
- `listening` — visual feedback that the mic is hot
|
||||
- `thinking` — Tatlock is working on a reply (this state earns its keep; see
|
||||
[Latency budget](#latency-budget--streaming))
|
||||
- `speaking` — mouth/waveform animation synced to TTS playback
|
||||
`waveshare/esp32_p4_wifi6_touch_lcd_xc` BSP). Six expression states — see
|
||||
[Face design](#face-design) for the visual contract.
|
||||
- **Audio capture**: dual mics through the ES7210 (hardware echo cancellation reference
|
||||
from the playback path), 16 kHz 16-bit mono PCM.
|
||||
- **Audio playback**: ES8311 codec → speaker. Plays PCM streamed from the gateway.
|
||||
@@ -123,27 +119,73 @@ Untouched by this project. DeskLock consumes its OpenAI-compatible API over the
|
||||
LAN (port 8000, bypassing the Authentik-protected public route). If device auth is needed
|
||||
later, the gateway holds the credential — never the firmware.
|
||||
|
||||
## Face design
|
||||
|
||||
**Aesthetic**: pure black screen; a face drawn from ASCII/terminal glyphs in green
|
||||
phosphor (`#adffc8` face, dimmer greens for secondary info); Matrix-style digital rain
|
||||
whose **density encodes activity** — barely-there drips when idle, a downpour while
|
||||
Tatlock works. No bitmaps, no skeuomorphism: glyphs only.
|
||||
|
||||
**Source of truth**: `sim/face/index.html` — a self-contained browser simulator of the
|
||||
800×800 round panel. Design changes land there first, get approved visually, then get
|
||||
ported to LVGL. The `STATES` table in the sim defines the contract:
|
||||
|
||||
| State | Eyes | Mouth | Rain | Extra cues |
|
||||
|-------|------|-------|------|------------|
|
||||
| `idle` | `- -` | `\_/` | 2 slow streams | clock (HH:MM), breathing bob, blinks |
|
||||
| `listening` | `O O` | `o` | 16 streams | blinks |
|
||||
| `pensive` | `· ·` | `~` | 7 streams | cycling `...` thought dots |
|
||||
| `effort` | `> <` | `~` | 40 fast streams | **orbit arc on bezel + `[ Ns ]` elapsed counter**, face jitter |
|
||||
| `speaking` | `^ ^` | cycles `o O - O = o` | 14 streams | mouth animates ~150 ms/frame |
|
||||
| `error` | `x x` | `-` | none (rain dies) | face dims to 45% |
|
||||
|
||||
**Wait cues are a hard requirement** (user-stated): Tatlock turns take 10–25 s, so
|
||||
`effort` must always show *alive-and-working* signals — the orbiting bezel arc, the
|
||||
elapsed-seconds counter, and max rain. Never a bare static face during a wait, and no
|
||||
fake progress bars — only honest cues.
|
||||
|
||||
**Protocol → face mapping**: gateway `state: thinking` → `effort`; transcription and
|
||||
other short local waits → `pensive`; `listening`/`speaking` map 1:1; WebSocket
|
||||
disconnected → `error`; otherwise `idle`.
|
||||
|
||||
**LVGL port notes** (for phase 2):
|
||||
|
||||
- Drive everything from fixed-step `lv_timer`s (~30 fps rain tick) — the sim
|
||||
deliberately uses `setInterval`, not `requestAnimationFrame`, to mirror this.
|
||||
- Rain: `lv_canvas` (or a pooled label grid) with per-frame fade; orbit arc = `lv_arc`.
|
||||
- Fonts: generate a large monospace glyph font including the katakana subset used in
|
||||
`GLYPHS` via `lv_font_conv`; the built-in `unscii` fonts are too small for 800 px.
|
||||
- The sim's text glow (`text-shadow`) is browser flair — the device renders flat glyphs.
|
||||
|
||||
## Latency budget & streaming
|
||||
|
||||
Measured/known numbers that shape the design:
|
||||
Measured/known numbers that shape the design (Tatlock figures per tatlock CLAUDE.md,
|
||||
GPU-resident benchmarks of 2026-07-14, gemma4:e2b at ~100 tok/s):
|
||||
|
||||
| Stage | Cost |
|
||||
|-------|------|
|
||||
| STT (whisper `small`, GPU) | a few hundred ms for a ~5 s utterance |
|
||||
| TTS (Piper/Kokoro) | faster than realtime |
|
||||
| **Tatlock, full local flow (gemma4)** | **~35 s Steward analysis warm; ~2 min end-to-end** (per tatlock CLAUDE.md) |
|
||||
| Tatlock Steward analysis | ~6 s warm |
|
||||
| **Tatlock, full local flow** | **11–25 s end-to-end** (librarian-routed ~20–25 s) |
|
||||
| Tatlock cold start (>2 h idle) | +~8 s (`OLLAMA_KEEP_ALIVE=2h`) |
|
||||
|
||||
Speech is not the bottleneck — **Tatlock is**, by two orders of magnitude. Constraints
|
||||
this imposes:
|
||||
(Older "~35 s Steward / ~2 min flow" figures were from a CPU-only driver-mismatch era —
|
||||
do not plan against them.)
|
||||
|
||||
Speech is not the bottleneck — **Tatlock is**, by one to two orders of magnitude.
|
||||
Constraints this imposes:
|
||||
|
||||
1. **The gateway must consume Tatlock's streaming response and synthesize
|
||||
sentence-by-sentence**, forwarding audio as each sentence is ready. The device starts
|
||||
speaking after the first sentence instead of waiting for the full reply. The WS
|
||||
protocol already supports this: one `audio_start` … PCM … `audio_end` envelope with
|
||||
chunks arriving as they're synthesized — the device just plays a continuous stream.
|
||||
speaking after the first sentence instead of waiting for the full reply — with
|
||||
streaming, first audio should land roughly at Steward-time + first-sentence-time,
|
||||
well under the 11–25 s full-flow figure. The WS protocol already supports this: one
|
||||
`audio_start` … PCM … `audio_end` envelope with chunks arriving as they're
|
||||
synthesized — the device just plays a continuous stream.
|
||||
2. **The `thinking` face state is a first-class feature**, not decoration — it's what
|
||||
makes a long Tatlock turn feel intentional instead of broken. Consider progress cues
|
||||
(e.g. surface Tatlock's reasoning summaries on-screen) later.
|
||||
makes a 10–25 s Tatlock turn feel intentional instead of broken. Consider progress
|
||||
cues (e.g. surface Tatlock's reasoning summaries on-screen) later.
|
||||
3. A **fast lane** may eventually be needed: MultiNet on-device commands for instant
|
||||
home-automation phrases, and/or a low-latency intent path in Tatlock itself. Out of
|
||||
scope for now, but don't design it out.
|
||||
|
||||
Reference in New Issue
Block a user