Document Speaches speech layer, on-device processing limits, latency constraints

- STT/TTS move to a shared Speaches container (OpenAI-format, GPU, port
  8601 proposed); gateway becomes a thin orchestrator with pluggable
  speech backends (speaches default, embedded fallback)
- Record the on-device ceiling: WakeNet wake word, VAD, ES7210 AEC,
  optional MultiNet fixed commands; open-vocabulary STT permanently out
- Record the real latency bottleneck (Tatlock ~2 min full local flow):
  gateway must stream chat tokens and synthesize sentence-by-sentence
- Plan reply_delta + barge-in protocol additions

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-14 17:14:44 +02:00
co-authored by Claude Fable 5
parent 576fd7d237
commit 0886c84d5e
3 changed files with 160 additions and 42 deletions
+6 -2
View File
@@ -11,8 +11,12 @@ one repo:
- `firmware/` — ESP-IDF (C, LVGL 9) app for the Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C
(3.4" round 800×800 touch display, dual mics + ES7210 AEC, ES8311 codec + speaker).
- `gateway/` — Python FastAPI container on tower-of-joy doing STT (faster-whisper, GPU),
chat (Tatlock `/v1/chat/completions`), and TTS (Piper). Listens on port **8600**.
- `gateway/` — Python FastAPI container on tower-of-joy orchestrating STT → chat
(Tatlock `/v1/chat/completions`) → TTS. Listens on port **8600**. STT/TTS models live
in a shared **Speaches** container (proposed port 8601, OpenAI-format API), not in the
gateway image; `stt.py`/`tts.py` are pluggable backends (`speaches` default,
`embedded` fallback for dev). See docs/architecture.md — the scaffold currently
implements only `embedded`.
The device and gateway speak a WebSocket protocol defined in `docs/architecture.md`.
**That doc is the contract** — update it in the same change as any protocol edit on