jpmschweitzerandClaude Fable 5
b8974ec84f
Root cause of the dead data path (assoc/scan/RPC fine, zero data frames): esp-hosted 1.4.x is formally incompatible with IDF 5.5 (esp-hosted-mcu#47) and the factory C6 slave firmware was ancient (couldn't even answer a version RPC). Waveshare's examples pin 1.4.* — do not follow them. Fix: - host: espressif/esp_hosted ^2.12 (+ esp_wifi_remote 1.6) - slave: 2.12.11 network_adapter.bin embedded in the app (c6_ota.c streams it to the C6 over the SDIO RPC channel at boot when the reported version is < 2.x; ~10s, no wires, idempotent) - RAM diet for hosted 2.x's footprint (MEMPOOL_PREFER_SPIRAM, reduced WIFI_RMT buffers) — without it internal SRAM famine boot-loops in xTaskCreateStaticPinnedToCore before app_main - conservative 20MHz SDIO clock for first verified data path Verified on hardware: DHCP lease (even that healed), 8/8 pings to router and tower-of-joy at 1-5ms, WebSocket to the gateway connected, device registered at /devices as desklock-p4 with fw version. Also: wifi_diag.c L1 SoftAP diagnostic mode (DESKLOCK-DIAG) with review fixes, boot ping ladder in net.c, static-IP fallback, face_status() line, docs for the whole failure taxonomy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
131 lines
7.4 KiB
Markdown
131 lines
7.4 KiB
Markdown
# AGENTS.md
|
||
|
||
> Operational protocols and architecture for AI assistants working on DeskLock.
|
||
> Read [docs/architecture.md](docs/architecture.md) before making design changes.
|
||
|
||
## What this project is
|
||
|
||
DeskLock is the living-room visual/audio endpoint for **Tatlock**, the homelab butler
|
||
(`/mnt/media/Projects/tatlock`, API at `http://tatlock.schweitz.internal:8000`). Two halves,
|
||
one repo:
|
||
|
||
- `firmware/` — ESP-IDF (C, LVGL 9) app for the Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C
|
||
(3.4" round 800×800 touch display, dual mics + ES7210 AEC, ES8311 codec + speaker).
|
||
- `gateway/` — Python FastAPI container on tower-of-joy orchestrating STT → chat
|
||
(Tatlock `/v1/chat/completions`) → TTS. Listens on port **8600**. STT/TTS models live
|
||
in the shared **Speaches** container (live on port 8601, OpenAI-format API), not in
|
||
the gateway image; `stt.py`/`tts.py` are pluggable backends (`speaches` default,
|
||
`embedded` fallback needing the `[speech]` extra). Gateway needs **Python ≥ 3.11**
|
||
(no ceiling; the container runs 3.13) — but system python3 on tower-of-joy is 3.8,
|
||
so `make setup` explicitly uses `python3.12`. No local audio resampling in the
|
||
default path: the gateway requests 16 kHz output via Speaches' `sample_rate`
|
||
extension (verified live).
|
||
|
||
The device and gateway speak a WebSocket protocol defined in `docs/architecture.md`.
|
||
**That doc is the contract** — update it in the same change as any protocol edit on
|
||
either side.
|
||
|
||
The face (black screen, ASCII glyph expressions, matrix rain as activity signal) is
|
||
designed in `sim/face/index.html` — the design source of truth — and specified in the
|
||
"Face design" section of `docs/architecture.md`. Change the sim and the doc together;
|
||
the LVGL implementation follows them. Verify sim changes visually with
|
||
`~/bin/claude-screenshot` (note: the tool uses `--virtual-time-budget`, which starves
|
||
`requestAnimationFrame` — drive sim animation with `setInterval`, which also mirrors
|
||
LVGL timers).
|
||
|
||
## Hard rules
|
||
|
||
- **Keep the firmware thin.** No STT, no TTS, no conversation logic on the device.
|
||
If a feature needs intelligence, it goes in the gateway or in Tatlock itself.
|
||
- **Never modify Tatlock from this repo.** It is a separate project with its own repo.
|
||
DeskLock consumes its public API only.
|
||
- **Secrets** (Wi-Fi credentials, any future API keys) never go in source. Firmware
|
||
gets them via a gitignored `firmware/secrets.h` (see AGENTS notes below) or NVS;
|
||
the gateway via environment variables (`DESKLOCK_*`).
|
||
|
||
## Firmware (`firmware/`)
|
||
|
||
- Toolchain: **ESP-IDF ≥ 5.4** (not Arduino, not PlatformIO). Target `esp32p4`.
|
||
- BSP: [`waveshare/esp32_p4_wifi6_touch_lcd_xc`](https://components.espressif.com/components/waveshare/esp32_p4_wifi6_touch_lcd_xc)
|
||
from the ESP Component Registry (pulled automatically via `main/idf_component.yml`).
|
||
- Reference implementations: [waveshareteam/ESP32-P4-WIFI6-Touch-LCD-XC](https://github.com/waveshareteam/ESP32-P4-WIFI6-Touch-LCD-XC)
|
||
`examples/esp-idf/` — notably `08_lvgl_demo_v9` (display), `06_I2SCodec` (audio),
|
||
`04_wifistation` (Wi-Fi via ESP-Hosted). When wiring a new peripheral, check the
|
||
official example first; do not guess pin mappings.
|
||
```bash
|
||
# ESP-IDF v5.5 is installed at ~/esp-idf. Every shell:
|
||
source ~/esp-idf/export.sh
|
||
|
||
# Build / flash (device on USB-C at /dev/ttyACM0, CH343 bridge)
|
||
cd firmware
|
||
idf.py build
|
||
sg dialout -c "bash -lc 'source ~/esp-idf/export.sh >/dev/null && idf.py -p /dev/ttyACM0 flash'"
|
||
```
|
||
|
||
- `sg dialout -c '…'` is needed because the login session predates the user's dialout
|
||
membership; a plain `idf.py flash` works after any re-login.
|
||
- **Radio stack: esp_hosted ≥ 2.x on BOTH chips, non-negotiable.** esp-hosted 1.x is
|
||
formally incompatible with IDF 5.5 (esp-hosted-mcu#47) — symptom: RPC/scan/connect
|
||
all work, but NO data frames ever flow (no DHCP, no ARP, no ping). Waveshare's
|
||
examples pin 1.4.* and the factory C6 slave firmware is ancient — both wrong. The
|
||
host manifest pins `espressif/esp_hosted: "^2.12"`; the matching slave image is
|
||
embedded as `main/c6_slave.bin` and `c6_ota.c` flashes the C6 **over SDIO** at boot
|
||
whenever the C6 reports a version < 2.x (build a new bin from the component's
|
||
`slave/` project for esp32c6 when bumping versions).
|
||
- **Internal-RAM famine assert**: `assert failed: xTaskCreateStaticPinnedToCore …
|
||
xPortcheckValidStackMem` in a pre-app_main boot loop means static+early allocations
|
||
starved internal SRAM (hosted 2.x is hungry). Keep
|
||
`CONFIG_ESP_HOSTED_MEMPOOL_PREFER_SPIRAM=y` and the reduced `WIFI_RMT_*` buffer
|
||
counts in sdkconfig.defaults; check `heap_init:` pool lines when the binary grows.
|
||
- SDIO clock is set conservatively (`CONFIG_ESP_HOSTED_SDIO_CLOCK_FREQ_KHZ=20000`),
|
||
ample for 16 kHz voice; raising to 40 MHz is untested on this board's data path.
|
||
- **L1 driver diagnosis mode**: set `WIFI_DIAG_MODE 1` in desklock_main.c — the device
|
||
becomes AP `DESKLOCK-DIAG` (pass `desklock123`, page at http://192.168.4.1/) proving
|
||
radio+SDIO+IP with zero external network variables. Ladder: L0 SDIO control → L1
|
||
softap data → L2 STA to any network → L3 STA to "Outside" → L4 gateway.
|
||
- **PSRAM must run at 200 MHz** or the 800×800 MIPI-DSI framebuffer underruns
|
||
(`lcd.dsi.dpi: can't fetch data…` spam, LVGL lock never frees, task watchdog).
|
||
`CONFIG_SPIRAM_SPEED_200M` only takes effect together with
|
||
`CONFIG_IDF_EXPERIMENTAL_FEATURES=y` — otherwise it is **silently dropped** and you
|
||
get 20 MHz. `sdkconfig.defaults` mirrors the official `08_lvgl_demo_v9` config.
|
||
- Non-interactive boot-log capture (avoid `idf.py monitor`, it's interactive): open
|
||
`/dev/ttyACM0` at 115200 with pyserial, pulse RTS to reset, read ~8 s. Verified boot
|
||
is ~1.6 s from reset to `desklock: DeskLock up`.
|
||
- If the device doesn't enumerate, hold BOOT while pressing RESET for download mode.
|
||
|
||
## Gateway (`gateway/`)
|
||
|
||
```bash
|
||
cd gateway
|
||
make setup # venv + dev deps (no ML models)
|
||
make setup-speech # additionally install faster-whisper + piper
|
||
make run # uvicorn on :8600 with reload
|
||
make test # pytest
|
||
make lint # ruff check + format check
|
||
make typecheck # mypy
|
||
```
|
||
|
||
- Config via `DESKLOCK_*` env vars — see `src/desklock_gateway/config.py` for the schema
|
||
and defaults.
|
||
- `stt.py` / `tts.py` defer their heavy imports so the app boots without the `speech`
|
||
extra — keep it that way so protocol tests stay fast.
|
||
- Deployment is CI-driven: pushing a `v*` tag makes Gitea Actions test, build, and push
|
||
`desklock-gateway:{latest,tag}` to the registry and trigger Watchtower
|
||
(`.gitea/workflows/build.yml`; needs `REGISTRY_USER`/`REGISTRY_PASSWORD`/
|
||
`WATCHTOWER_HTTP_API_TOKEN` secrets). Plain pushes to `main` run lint + tests only. The
|
||
gateway deploys as part of the **`tatlock-ui` Portainer stack** —
|
||
`system-admin-toj/containers/stacks/tatlock-ui.yml` (registered in `CONTAINERS.md`,
|
||
port 8600). Stack updates go through the Portainer API on :8001 (JWT auth; recipe in
|
||
`system-admin-toj/containers/setup-new-host.md`), not by editing files on disk.
|
||
- Verify speech changes against the live Speaches container with a real round trip
|
||
(TTS → STT of a known phrase, expect the transcript back); warm timings to expect:
|
||
STT ~0.3 s, TTS ~2 s per sentence.
|
||
|
||
## Homelab context
|
||
|
||
- This server **is** tower-of-joy; the device, gateway, and Tatlock all share the LAN.
|
||
- Use `tatlock.schweitz.internal:8000` (direct, no SSO) — the public
|
||
`tatlock.schweitz.net` route sits behind Authentik and is not for machine-to-machine
|
||
traffic.
|
||
- Git remote: `git.schweitz.net` (Gitea).
|