Add face design: ASCII expressions + matrix rain simulator, wait cues, fresh latency numbers
- sim/face/index.html: browser simulator of the 800x800 round panel with six states (idle/listening/pensive/effort/speaking/error), state-driven rain density, blink/talk/thought animations, idle clock - effort state gets hard-required wait cues: orbiting bezel arc, elapsed counter, max rain (Tatlock turns run 10-25s) - rain driven by setInterval, not rAF: renders under the screenshot tool's --virtual-time-budget and mirrors LVGL lv_timer - architecture.md: Face design contract (state table, protocol mapping, LVGL port notes) + latency table updated to GPU-era benchmarks (Steward ~6s warm, full flow 11-25s; old CPU-era figures retired) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Claude Fable 5
@@ -22,6 +22,14 @@ The device and gateway speak a WebSocket protocol defined in `docs/architecture.
|
||||
**That doc is the contract** — update it in the same change as any protocol edit on
|
||||
either side.
|
||||
|
||||
The face (black screen, ASCII glyph expressions, matrix rain as activity signal) is
|
||||
designed in `sim/face/index.html` — the design source of truth — and specified in the
|
||||
"Face design" section of `docs/architecture.md`. Change the sim and the doc together;
|
||||
the LVGL implementation follows them. Verify sim changes visually with
|
||||
`~/bin/claude-screenshot` (note: the tool uses `--virtual-time-budget`, which starves
|
||||
`requestAnimationFrame` — drive sim animation with `setInterval`, which also mirrors
|
||||
LVGL timers).
|
||||
|
||||
## Hard rules
|
||||
|
||||
- **Keep the firmware thin.** No STT, no TTS, no conversation logic on the device.
|
||||
|
||||
@@ -54,6 +54,9 @@ See [docs/architecture.md](docs/architecture.md) for the full design.
|
||||
|
||||
- `firmware/` — ESP-IDF (C, LVGL 9) application for the ESP32-P4
|
||||
- `gateway/` — Python FastAPI voice gateway, deployed as a container on tower-of-joy
|
||||
- `sim/face/` — browser simulator of the face (design source of truth; serve with
|
||||
`python3 -m http.server` and open `index.html`, or use `?state=…&nochrome=1` for
|
||||
screenshots)
|
||||
- `docs/` — architecture and design notes
|
||||
|
||||
## Roadmap
|
||||
|
||||
+57
-15
@@ -39,12 +39,8 @@ the **speech layer** (Speaches) owns the actual STT/TTS models on the GPU.
|
||||
Responsibilities:
|
||||
|
||||
- **Face rendering** (LVGL 9 on the 800×800 round MIPI-DSI panel via the
|
||||
`waveshare/esp32_p4_wifi6_touch_lcd_xc` BSP). Face states:
|
||||
- `idle` — subtle animation + clock (it's a desk clock when nobody's talking to it)
|
||||
- `listening` — visual feedback that the mic is hot
|
||||
- `thinking` — Tatlock is working on a reply (this state earns its keep; see
|
||||
[Latency budget](#latency-budget--streaming))
|
||||
- `speaking` — mouth/waveform animation synced to TTS playback
|
||||
`waveshare/esp32_p4_wifi6_touch_lcd_xc` BSP). Six expression states — see
|
||||
[Face design](#face-design) for the visual contract.
|
||||
- **Audio capture**: dual mics through the ES7210 (hardware echo cancellation reference
|
||||
from the playback path), 16 kHz 16-bit mono PCM.
|
||||
- **Audio playback**: ES8311 codec → speaker. Plays PCM streamed from the gateway.
|
||||
@@ -123,27 +119,73 @@ Untouched by this project. DeskLock consumes its OpenAI-compatible API over the
|
||||
LAN (port 8000, bypassing the Authentik-protected public route). If device auth is needed
|
||||
later, the gateway holds the credential — never the firmware.
|
||||
|
||||
## Face design
|
||||
|
||||
**Aesthetic**: pure black screen; a face drawn from ASCII/terminal glyphs in green
|
||||
phosphor (`#adffc8` face, dimmer greens for secondary info); Matrix-style digital rain
|
||||
whose **density encodes activity** — barely-there drips when idle, a downpour while
|
||||
Tatlock works. No bitmaps, no skeuomorphism: glyphs only.
|
||||
|
||||
**Source of truth**: `sim/face/index.html` — a self-contained browser simulator of the
|
||||
800×800 round panel. Design changes land there first, get approved visually, then get
|
||||
ported to LVGL. The `STATES` table in the sim defines the contract:
|
||||
|
||||
| State | Eyes | Mouth | Rain | Extra cues |
|
||||
|-------|------|-------|------|------------|
|
||||
| `idle` | `- -` | `\_/` | 2 slow streams | clock (HH:MM), breathing bob, blinks |
|
||||
| `listening` | `O O` | `o` | 16 streams | blinks |
|
||||
| `pensive` | `· ·` | `~` | 7 streams | cycling `...` thought dots |
|
||||
| `effort` | `> <` | `~` | 40 fast streams | **orbit arc on bezel + `[ Ns ]` elapsed counter**, face jitter |
|
||||
| `speaking` | `^ ^` | cycles `o O - O = o` | 14 streams | mouth animates ~150 ms/frame |
|
||||
| `error` | `x x` | `-` | none (rain dies) | face dims to 45% |
|
||||
|
||||
**Wait cues are a hard requirement** (user-stated): Tatlock turns take 10–25 s, so
|
||||
`effort` must always show *alive-and-working* signals — the orbiting bezel arc, the
|
||||
elapsed-seconds counter, and max rain. Never a bare static face during a wait, and no
|
||||
fake progress bars — only honest cues.
|
||||
|
||||
**Protocol → face mapping**: gateway `state: thinking` → `effort`; transcription and
|
||||
other short local waits → `pensive`; `listening`/`speaking` map 1:1; WebSocket
|
||||
disconnected → `error`; otherwise `idle`.
|
||||
|
||||
**LVGL port notes** (for phase 2):
|
||||
|
||||
- Drive everything from fixed-step `lv_timer`s (~30 fps rain tick) — the sim
|
||||
deliberately uses `setInterval`, not `requestAnimationFrame`, to mirror this.
|
||||
- Rain: `lv_canvas` (or a pooled label grid) with per-frame fade; orbit arc = `lv_arc`.
|
||||
- Fonts: generate a large monospace glyph font including the katakana subset used in
|
||||
`GLYPHS` via `lv_font_conv`; the built-in `unscii` fonts are too small for 800 px.
|
||||
- The sim's text glow (`text-shadow`) is browser flair — the device renders flat glyphs.
|
||||
|
||||
## Latency budget & streaming
|
||||
|
||||
Measured/known numbers that shape the design:
|
||||
Measured/known numbers that shape the design (Tatlock figures per tatlock CLAUDE.md,
|
||||
GPU-resident benchmarks of 2026-07-14, gemma4:e2b at ~100 tok/s):
|
||||
|
||||
| Stage | Cost |
|
||||
|-------|------|
|
||||
| STT (whisper `small`, GPU) | a few hundred ms for a ~5 s utterance |
|
||||
| TTS (Piper/Kokoro) | faster than realtime |
|
||||
| **Tatlock, full local flow (gemma4)** | **~35 s Steward analysis warm; ~2 min end-to-end** (per tatlock CLAUDE.md) |
|
||||
| Tatlock Steward analysis | ~6 s warm |
|
||||
| **Tatlock, full local flow** | **11–25 s end-to-end** (librarian-routed ~20–25 s) |
|
||||
| Tatlock cold start (>2 h idle) | +~8 s (`OLLAMA_KEEP_ALIVE=2h`) |
|
||||
|
||||
Speech is not the bottleneck — **Tatlock is**, by two orders of magnitude. Constraints
|
||||
this imposes:
|
||||
(Older "~35 s Steward / ~2 min flow" figures were from a CPU-only driver-mismatch era —
|
||||
do not plan against them.)
|
||||
|
||||
Speech is not the bottleneck — **Tatlock is**, by one to two orders of magnitude.
|
||||
Constraints this imposes:
|
||||
|
||||
1. **The gateway must consume Tatlock's streaming response and synthesize
|
||||
sentence-by-sentence**, forwarding audio as each sentence is ready. The device starts
|
||||
speaking after the first sentence instead of waiting for the full reply. The WS
|
||||
protocol already supports this: one `audio_start` … PCM … `audio_end` envelope with
|
||||
chunks arriving as they're synthesized — the device just plays a continuous stream.
|
||||
speaking after the first sentence instead of waiting for the full reply — with
|
||||
streaming, first audio should land roughly at Steward-time + first-sentence-time,
|
||||
well under the 11–25 s full-flow figure. The WS protocol already supports this: one
|
||||
`audio_start` … PCM … `audio_end` envelope with chunks arriving as they're
|
||||
synthesized — the device just plays a continuous stream.
|
||||
2. **The `thinking` face state is a first-class feature**, not decoration — it's what
|
||||
makes a long Tatlock turn feel intentional instead of broken. Consider progress cues
|
||||
(e.g. surface Tatlock's reasoning summaries on-screen) later.
|
||||
makes a 10–25 s Tatlock turn feel intentional instead of broken. Consider progress
|
||||
cues (e.g. surface Tatlock's reasoning summaries on-screen) later.
|
||||
3. A **fast lane** may eventually be needed: MultiNet on-device commands for instant
|
||||
home-automation phrases, and/or a low-latency intent path in Tatlock itself. Out of
|
||||
scope for now, but don't design it out.
|
||||
|
||||
@@ -0,0 +1,231 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>DeskLock Face — design simulator</title>
|
||||
<!--
|
||||
Design reference for the DeskLock face (docs/architecture.md "Face design").
|
||||
This file is the source of truth for the look: the LVGL firmware port must match
|
||||
the states, glyphs, and rain behaviour defined here.
|
||||
|
||||
URL params: ?state=idle|listening|pensive|effort|speaking|error
|
||||
&demo=1 (auto-cycle) &nochrome=1 (hide controls, device view only)
|
||||
-->
|
||||
<style>
|
||||
* { box-sizing: border-box; }
|
||||
body {
|
||||
margin: 0; padding: 24px;
|
||||
background: #050807; color: #7fae91;
|
||||
font-family: "Courier New", monospace;
|
||||
display: flex; flex-direction: column; align-items: center; gap: 20px;
|
||||
min-height: 100vh;
|
||||
}
|
||||
body.nochrome { padding: 0; }
|
||||
body.nochrome #controls, body.nochrome #title { display: none; }
|
||||
#title { font-size: 14px; letter-spacing: .3em; color: #2f7a4b; }
|
||||
#stage { transform-origin: top center; display: flex; flex-direction: column; align-items: center; gap: 20px; }
|
||||
|
||||
/* The physical screen: 800x800 round panel */
|
||||
#screen {
|
||||
width: 800px; height: 800px; border-radius: 50%;
|
||||
background: #000; overflow: hidden; position: relative;
|
||||
box-shadow: 0 0 0 12px #0d0d0d, 0 0 70px rgba(0, 255, 120, .10);
|
||||
}
|
||||
#rain { position: absolute; inset: 0; }
|
||||
#faceShade {
|
||||
position: absolute; inset: 0;
|
||||
background: radial-gradient(circle at 50% 47%,
|
||||
rgba(0,0,0,.94) 0 25%, rgba(0,0,0,.6) 37%, rgba(0,0,0,0) 54%);
|
||||
}
|
||||
#faceWrap {
|
||||
position: absolute; inset: 0;
|
||||
display: flex; flex-direction: column; align-items: center; justify-content: center;
|
||||
gap: 44px; color: #adffc8; text-shadow: 0 0 24px rgba(0,255,130,.4);
|
||||
animation: breathe 4.5s ease-in-out infinite;
|
||||
}
|
||||
#eyes { font-size: 170px; line-height: 1; margin: 0; }
|
||||
#mouth { font-size: 105px; line-height: 1; margin: 0; }
|
||||
#thought {
|
||||
position: absolute; top: 22%; right: 20%; width: 140px;
|
||||
font-size: 62px; color: #63cf8b; text-align: left;
|
||||
}
|
||||
#clock {
|
||||
font-size: 46px; color: #2f7a4b; text-shadow: none;
|
||||
letter-spacing: .18em; display: none;
|
||||
}
|
||||
body[data-state="idle"] #clock { display: block; }
|
||||
body[data-state="effort"] #faceWrap { animation: jitter .16s steps(2) infinite; }
|
||||
body[data-state="error"] #faceWrap { opacity: .45; animation: none; }
|
||||
|
||||
/* wait cues while Tatlock works (10-25 s): orbiting arc + elapsed counter */
|
||||
#orbit {
|
||||
position: absolute; inset: 12px; border-radius: 50%;
|
||||
border: 6px solid transparent; border-top-color: #37ff8b;
|
||||
filter: drop-shadow(0 0 9px rgba(55,255,139,.55));
|
||||
animation: spin 1.4s linear infinite; display: none;
|
||||
}
|
||||
#elapsed {
|
||||
position: absolute; bottom: 13%; left: 50%; transform: translateX(-50%);
|
||||
font-size: 42px; color: #2f7a4b; letter-spacing: .15em; display: none;
|
||||
background: rgba(0,0,0,.8); padding: 6px 20px; border-radius: 12px;
|
||||
}
|
||||
body[data-state="effort"] #orbit,
|
||||
body[data-state="effort"] #elapsed { display: block; }
|
||||
@keyframes spin { to { transform: rotate(360deg); } }
|
||||
|
||||
@keyframes breathe { 0%,100% { transform: translateY(0); } 50% { transform: translateY(9px); } }
|
||||
@keyframes jitter { 0% { transform: translate(1px,-1px); } 100% { transform: translate(-1px,1px); } }
|
||||
|
||||
#controls { display: flex; gap: 10px; flex-wrap: wrap; justify-content: center; }
|
||||
#controls button {
|
||||
background: #0c1410; border: 1px solid #1d3527; color: #7fae91;
|
||||
font-family: inherit; font-size: 14px; padding: 8px 16px; cursor: pointer;
|
||||
letter-spacing: .1em;
|
||||
}
|
||||
#controls button.active { border-color: #37ff8b; color: #b9ffd6; }
|
||||
</style>
|
||||
</head>
|
||||
<body data-state="idle">
|
||||
<div id="title">DESKLOCK · FACE SIMULATOR · 800×800</div>
|
||||
<div id="stage">
|
||||
<div id="screen">
|
||||
<canvas id="rain" width="800" height="800"></canvas>
|
||||
<div id="faceShade"></div>
|
||||
<div id="faceWrap">
|
||||
<pre id="eyes"></pre>
|
||||
<pre id="mouth"></pre>
|
||||
<div id="clock"></div>
|
||||
</div>
|
||||
<div id="thought"></div>
|
||||
<div id="orbit"></div>
|
||||
<div id="elapsed"></div>
|
||||
</div>
|
||||
<div id="controls"></div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
"use strict";
|
||||
|
||||
/* ---- expression + rain definitions (the contract for the LVGL port) ---- */
|
||||
const STATES = {
|
||||
idle: { eyes: "- -", mouth: "\\_/", blink: true, rain: { streams: 2, speed: 4 } },
|
||||
listening: { eyes: "O O", mouth: "o", blink: true, rain: { streams: 16, speed: 7 } },
|
||||
pensive: { eyes: "· ·", mouth: "~", blink: false, thought: true,
|
||||
rain: { streams: 7, speed: 5 } },
|
||||
effort: { eyes: "> <", mouth: "~", blink: false, rain: { streams: 40, speed: 16 } },
|
||||
speaking: { eyes: "^ ^", mouth: "o", blink: true, talk: true,
|
||||
rain: { streams: 14, speed: 9 } },
|
||||
error: { eyes: "x x", mouth: "-", blink: false, rain: { streams: 0, speed: 0 } },
|
||||
};
|
||||
const ORDER = ["idle", "listening", "pensive", "effort", "speaking", "error"];
|
||||
const TALK = ["o", "O", "-", "O", "=", "o"];
|
||||
const GLYPHS = "アイウエオカキクケコサシスセソタチツテトナニヌネノ0123456789ACEFHKZ$#%*+=<>";
|
||||
|
||||
const qs = new URLSearchParams(location.search);
|
||||
let cur = ORDER.includes(qs.get("state")) ? qs.get("state") : "idle";
|
||||
const st = () => STATES[cur];
|
||||
|
||||
const el = (id) => document.getElementById(id);
|
||||
const eyesEl = el("eyes"), mouthEl = el("mouth"), thoughtEl = el("thought"),
|
||||
clockEl = el("clock"), elapsedEl = el("elapsed");
|
||||
let stateSince = performance.now();
|
||||
|
||||
/* ---- matrix rain ---- */
|
||||
const R = 800, CELL = 22, COLS = Math.floor(R / CELL);
|
||||
const ctx = el("rain").getContext("2d");
|
||||
ctx.textAlign = "center";
|
||||
let streams = [];
|
||||
const spawn = () => ({
|
||||
col: Math.floor(Math.random() * COLS),
|
||||
y: -Math.random() * 14,
|
||||
speed: st().rain.speed * (0.7 + Math.random() * 0.6),
|
||||
});
|
||||
let last = performance.now();
|
||||
function frame(now) {
|
||||
const dt = Math.min(0.05, (now - last) / 1000); last = now;
|
||||
ctx.fillStyle = "rgba(0,0,0,0.08)";
|
||||
ctx.fillRect(0, 0, R, R);
|
||||
ctx.font = "bold " + CELL + "px monospace";
|
||||
for (const s of streams) {
|
||||
s.y += s.speed * dt;
|
||||
const px = s.col * CELL + CELL / 2, py = s.y * CELL;
|
||||
ctx.fillStyle = "rgba(195,255,215,0.95)";
|
||||
ctx.fillText(GLYPHS[(Math.random() * GLYPHS.length) | 0], px, py);
|
||||
ctx.fillStyle = "rgba(0,230,100,0.55)";
|
||||
ctx.fillText(GLYPHS[(Math.random() * GLYPHS.length) | 0], px, py - CELL);
|
||||
}
|
||||
streams = streams.filter((s) => s.y * CELL <= R + 2 * CELL);
|
||||
while (streams.length < st().rain.streams) streams.push(spawn());
|
||||
}
|
||||
|
||||
/* ---- face behaviour ---- */
|
||||
function applyFace() {
|
||||
eyesEl.textContent = st().eyes;
|
||||
mouthEl.textContent = st().mouth;
|
||||
thoughtEl.textContent = "";
|
||||
document.body.dataset.state = cur;
|
||||
document.querySelectorAll("#controls button[data-state]").forEach(
|
||||
(b) => b.classList.toggle("active", b.dataset.state === cur));
|
||||
}
|
||||
function setState(name) { cur = name; stateSince = performance.now(); applyFace(); }
|
||||
|
||||
setInterval(() => {
|
||||
elapsedEl.textContent = "[ " + Math.floor((performance.now() - stateSince) / 1000) + "s ]";
|
||||
}, 250);
|
||||
|
||||
(function blinkLoop() {
|
||||
setTimeout(() => {
|
||||
if (st().blink) {
|
||||
eyesEl.textContent = st().eyes.replace(/[^ ]/g, "_");
|
||||
setTimeout(() => { eyesEl.textContent = st().eyes; }, 130);
|
||||
}
|
||||
blinkLoop();
|
||||
}, 2600 + Math.random() * 3600);
|
||||
})();
|
||||
|
||||
let talkI = 0;
|
||||
setInterval(() => { if (st().talk) mouthEl.textContent = TALK[talkI++ % TALK.length]; }, 150);
|
||||
|
||||
let dotI = 0;
|
||||
setInterval(() => {
|
||||
thoughtEl.textContent = st().thought ? ".".repeat(1 + (dotI++ % 3)) : "";
|
||||
}, 480);
|
||||
|
||||
setInterval(() => {
|
||||
const d = new Date();
|
||||
clockEl.textContent =
|
||||
String(d.getHours()).padStart(2, "0") + ":" + String(d.getMinutes()).padStart(2, "0");
|
||||
}, 1000);
|
||||
|
||||
/* ---- sim chrome ---- */
|
||||
const controls = el("controls");
|
||||
for (const name of ORDER) {
|
||||
const b = document.createElement("button");
|
||||
b.textContent = name; b.dataset.state = name;
|
||||
b.onclick = () => { demo = false; demoBtn.textContent = "demo"; setState(name); };
|
||||
controls.appendChild(b);
|
||||
}
|
||||
let demo = qs.get("demo") === "1", demoI = 0;
|
||||
const demoBtn = document.createElement("button");
|
||||
demoBtn.textContent = demo ? "demo: on" : "demo";
|
||||
demoBtn.onclick = () => { demo = !demo; demoBtn.textContent = demo ? "demo: on" : "demo"; };
|
||||
controls.appendChild(demoBtn);
|
||||
setInterval(() => { if (demo) setState(ORDER[++demoI % ORDER.length]); }, 3000);
|
||||
|
||||
if (qs.get("nochrome") === "1") document.body.classList.add("nochrome");
|
||||
function rescale() {
|
||||
const pad = document.body.classList.contains("nochrome") ? 0 : 140;
|
||||
const s = Math.min(1, (innerWidth - 20) / 824, (innerHeight - pad) / 824);
|
||||
el("stage").style.transform = "scale(" + s + ")";
|
||||
}
|
||||
addEventListener("resize", rescale);
|
||||
|
||||
rescale();
|
||||
applyFace();
|
||||
/* fixed-step timer, not requestAnimationFrame: mirrors the LVGL port (lv_timer)
|
||||
and renders under headless/virtual-time captures where rAF is starved */
|
||||
setInterval(() => frame(performance.now()), 33);
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user