desklock/AGENTS.md

AGENTS.md

Operational protocols and architecture for AI assistants working on DeskLock. Read docs/architecture.md before making design changes.

What this project is

DeskLock is the living-room visual/audio endpoint for Tatlock, the homelab butler (/mnt/media/Projects/tatlock, API at http://tatlock:8000 on docker-dataplane). Two halves, one repo:

• firmware/ — ESP-IDF (C, LVGL 9) app for the Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C (3.4" round 800×800 touch display, dual mics + ES7210 AEC, ES8311 codec + speaker).
• gateway/ — Python FastAPI container on tower-of-joy orchestrating STT → chat (Tatlock /v1/chat/completions) → TTS. Listens on port 8600. STT/TTS models live in the shared Speaches container (live on port 8601, OpenAI-format API), not in the gateway image; stt.py/tts.py are pluggable backends (speaches default, embedded fallback needing the [speech] extra). Gateway needs Python ≥ 3.11 (no ceiling; the container runs 3.13) — but system python3 on tower-of-joy is 3.8, so make setup explicitly uses python3.12. No local audio resampling in the default path: the gateway requests 16 kHz output via Speaches' sample_rate extension (verified live).

The device and gateway speak a WebSocket protocol defined in docs/architecture.md. That doc is the contract — update it in the same change as any protocol edit on either side.

The face (black screen, ASCII glyph expressions, matrix rain as activity signal) is designed in sim/face/index.html — the design source of truth — and specified in the "Face design" section of docs/architecture.md. Change the sim and the doc together; the LVGL implementation follows them. Verify sim changes visually with ~/bin/claude-screenshot (note: the tool uses --virtual-time-budget, which starves requestAnimationFrame — drive sim animation with setInterval, which also mirrors LVGL timers).

Hard rules

• Keep the firmware thin. No STT, no TTS, no conversation logic on the device. If a feature needs intelligence, it goes in the gateway or in Tatlock itself.
• Never modify Tatlock from this repo. It is a separate project with its own repo. DeskLock consumes its public API only.
• Secrets (Wi-Fi credentials, any future API keys) never go in source. Firmware gets them via a gitignored firmware/secrets.h (see AGENTS notes below) or NVS; the gateway via environment variables (DESKLOCK_*).

Firmware (firmware/)

• Toolchain: ESP-IDF ≥ 5.4 (not Arduino, not PlatformIO). Target esp32p4.
• BSP: waveshare/esp32_p4_wifi6_touch_lcd_xc from the ESP Component Registry (pulled automatically via main/idf_component.yml).
• Reference implementations: waveshareteam/ESP32-P4-WIFI6-Touch-LCD-XC examples/esp-idf/ — notably 08_lvgl_demo_v9 (display), 06_I2SCodec (audio), 04_wifistation (Wi-Fi via ESP-Hosted). When wiring a new peripheral, check the official example first; do not guess pin mappings.

# ESP-IDF v5.5 is installed at ~/esp-idf. Every shell:
source ~/esp-idf/export.sh

# Build / flash (device on USB-C at /dev/ttyACM0, CH343 bridge)
cd firmware
idf.py build
sg dialout -c "bash -lc 'source ~/esp-idf/export.sh >/dev/null && idf.py -p /dev/ttyACM0 flash'"

• sg dialout -c '…' is needed because the login session predates the user's dialout membership; a plain idf.py flash works after any re-login.
• Radio stack: esp_hosted ≥ 2.x on BOTH chips, non-negotiable. esp-hosted 1.x is formally incompatible with IDF 5.5 (esp-hosted-mcu#47) — symptom: RPC/scan/connect all work, but NO data frames ever flow (no DHCP, no ARP, no ping). Waveshare's examples pin 1.4.* and the factory C6 slave firmware is ancient — both wrong. The host manifest pins espressif/esp_hosted: "^2.12"; the matching slave image is embedded as main/c6_slave.bin and c6_ota.c flashes the C6 over SDIO at boot whenever the C6 reports a version < 2.x (build a new bin from the component's slave/ project for esp32c6 when bumping versions).
• Internal-RAM famine assert: assert failed: xTaskCreateStaticPinnedToCore … xPortcheckValidStackMem in a pre-app_main boot loop means static+early allocations starved internal SRAM (hosted 2.x is hungry). Keep CONFIG_ESP_HOSTED_MEMPOOL_PREFER_SPIRAM=y and the reduced WIFI_RMT_* buffer counts in sdkconfig.defaults; check heap_init: pool lines when the binary grows.
• SDIO clock is set conservatively (CONFIG_ESP_HOSTED_SDIO_CLOCK_FREQ_KHZ=20000), ample for 16 kHz voice; raising to 40 MHz is untested on this board's data path.
• L1 driver diagnosis mode: set WIFI_DIAG_MODE 1 in desklock_main.c — the device becomes AP DESKLOCK-DIAG (pass desklock123, page at http://192.168.4.1/) proving radio+SDIO+IP with zero external network variables. Ladder: L0 SDIO control → L1 softap data → L2 STA to any network → L3 STA to "Outside" → L4 gateway.
• PSRAM must run at 200 MHz or the 800×800 MIPI-DSI framebuffer underruns (lcd.dsi.dpi: can't fetch data… spam, LVGL lock never frees, task watchdog). CONFIG_SPIRAM_SPEED_200M only takes effect together with CONFIG_IDF_EXPERIMENTAL_FEATURES=y — otherwise it is silently dropped and you get 20 MHz. sdkconfig.defaults mirrors the official 08_lvgl_demo_v9 config.
• Non-interactive boot-log capture (avoid idf.py monitor, it's interactive): open /dev/ttyACM0 at 115200 with pyserial, pulse RTS to reset, read ~8 s. Verified boot is ~1.6 s from reset to desklock: DeskLock up.
• If the device doesn't enumerate, hold BOOT while pressing RESET for download mode.

Gateway (gateway/)

cd gateway
make setup          # venv + dev deps (no ML models)
make setup-speech   # additionally install faster-whisper + piper
make run            # uvicorn on :8600 with reload
make test           # pytest
make lint           # ruff check + format check
make typecheck      # mypy

• Config via DESKLOCK_* env vars — see src/desklock_gateway/config.py for the schema and defaults.
• stt.py / tts.py defer their heavy imports so the app boots without the speech extra — keep it that way so protocol tests stay fast.
• Deployment is CI-driven: pushing a v* tag makes Gitea Actions test, build, and push desklock-gateway:{latest,tag} to the registry and trigger Watchtower (.gitea/workflows/build.yml; needs REGISTRY_USER/REGISTRY_PASSWORD/ WATCHTOWER_HTTP_API_TOKEN secrets). Plain pushes to main run lint + tests only. The gateway deploys as part of the tatlock-ui Portainer stack — system-admin-toj/containers/stacks/tatlock-ui.yml (registered in CONTAINERS.md, port 8600). Stack updates go through the Portainer API on :8001 (JWT auth; recipe in system-admin-toj/containers/setup-new-host.md), not by editing files on disk.
• Verify speech changes against the live Speaches container with a real round trip (TTS → STT of a known phrase, expect the transcript back); warm timings to expect: STT ~0.3 s, TTS ~2 s per sentence.

Homelab context

• This server is tower-of-joy; the device, gateway, and Tatlock all share the LAN.
• Machine-to-machine traffic uses container names on the docker-dataplane network — http://tatlock:8000 (the deployed stack overrides DESKLOCK_TATLOCK_BASE_URL to the host LAN IP while Tatlock runs on the host). Browser-facing URLs use https://<name>.schweitz.net (NPM, Authentik SSO) — never for M2M traffic.
• Git remote: git.schweitz.net (Gitea).