Document Speaches speech layer, on-device processing limits, latency constraints
- STT/TTS move to a shared Speaches container (OpenAI-format, GPU, port 8601 proposed); gateway becomes a thin orchestrator with pluggable speech backends (speaches default, embedded fallback) - Record the on-device ceiling: WakeNet wake word, VAD, ES7210 AEC, optional MultiNet fixed commands; open-vocabulary STT permanently out - Record the real latency bottleneck (Tatlock ~2 min full local flow): gateway must stream chat tokens and synthesize sentence-by-sentence - Plan reply_delta + barge-in protocol additions Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -11,8 +11,12 @@ one repo:
|
||||
|
||||
- `firmware/` — ESP-IDF (C, LVGL 9) app for the Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C
|
||||
(3.4" round 800×800 touch display, dual mics + ES7210 AEC, ES8311 codec + speaker).
|
||||
- `gateway/` — Python FastAPI container on tower-of-joy doing STT (faster-whisper, GPU),
|
||||
chat (Tatlock `/v1/chat/completions`), and TTS (Piper). Listens on port **8600**.
|
||||
- `gateway/` — Python FastAPI container on tower-of-joy orchestrating STT → chat
|
||||
(Tatlock `/v1/chat/completions`) → TTS. Listens on port **8600**. STT/TTS models live
|
||||
in a shared **Speaches** container (proposed port 8601, OpenAI-format API), not in the
|
||||
gateway image; `stt.py`/`tts.py` are pluggable backends (`speaches` default,
|
||||
`embedded` fallback for dev). See docs/architecture.md — the scaffold currently
|
||||
implements only `embedded`.
|
||||
|
||||
The device and gateway speak a WebSocket protocol defined in `docs/architecture.md`.
|
||||
**That doc is the contract** — update it in the same change as any protocol edit on
|
||||
|
||||
Reference in New Issue
Block a user