desklock/AGENTS.md

AGENTS.md

Operational protocols and architecture for AI assistants working on DeskLock. Read docs/architecture.md before making design changes.

What this project is

DeskLock is the living-room visual/audio endpoint for Tatlock, the homelab butler (/mnt/media/Projects/tatlock, API at http://tatlock.schweitz.internal:8000). Two halves, one repo:

• firmware/ — ESP-IDF (C, LVGL 9) app for the Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C (3.4" round 800×800 touch display, dual mics + ES7210 AEC, ES8311 codec + speaker).
• gateway/ — Python FastAPI container on tower-of-joy orchestrating STT → chat (Tatlock /v1/chat/completions) → TTS. Listens on port 8600. STT/TTS models live in the shared Speaches container (live on port 8601, OpenAI-format API), not in the gateway image; stt.py/tts.py are pluggable backends (speaches default, embedded fallback needing the [speech] extra). Gateway needs Python ≥ 3.11 (no ceiling; the container runs 3.13) — but system python3 on tower-of-joy is 3.8, so make setup explicitly uses python3.12. No local audio resampling in the default path: the gateway requests 16 kHz output via Speaches' sample_rate extension (verified live).

The device and gateway speak a WebSocket protocol defined in docs/architecture.md. That doc is the contract — update it in the same change as any protocol edit on either side.

The face (black screen, ASCII glyph expressions, matrix rain as activity signal) is designed in sim/face/index.html — the design source of truth — and specified in the "Face design" section of docs/architecture.md. Change the sim and the doc together; the LVGL implementation follows them. Verify sim changes visually with ~/bin/claude-screenshot (note: the tool uses --virtual-time-budget, which starves requestAnimationFrame — drive sim animation with setInterval, which also mirrors LVGL timers).

Hard rules

• Keep the firmware thin. No STT, no TTS, no conversation logic on the device. If a feature needs intelligence, it goes in the gateway or in Tatlock itself.
• Never modify Tatlock from this repo. It is a separate project with its own repo. DeskLock consumes its public API only.
• Secrets (Wi-Fi credentials, any future API keys) never go in source. Firmware gets them via a gitignored firmware/secrets.h (see AGENTS notes below) or NVS; the gateway via environment variables (DESKLOCK_*).

Firmware (firmware/)

• Toolchain: ESP-IDF ≥ 5.4 (not Arduino, not PlatformIO). Target esp32p4.
• BSP: waveshare/esp32_p4_wifi6_touch_lcd_xc from the ESP Component Registry (pulled automatically via main/idf_component.yml).
• Reference implementations: waveshareteam/ESP32-P4-WIFI6-Touch-LCD-XC examples/esp-idf/ — notably 08_lvgl_demo_v9 (display), 06_I2SCodec (audio), 04_wifistation (Wi-Fi via ESP-Hosted). When wiring a new peripheral, check the official example first; do not guess pin mappings.

# ESP-IDF v5.5 is installed at ~/esp-idf. Every shell:
source ~/esp-idf/export.sh

# Build / flash (device on USB-C at /dev/ttyACM0, CH343 bridge)
cd firmware
idf.py build
sg dialout -c "bash -lc 'source ~/esp-idf/export.sh >/dev/null && idf.py -p /dev/ttyACM0 flash'"

• sg dialout -c '…' is needed because the login session predates the user's dialout membership; a plain idf.py flash works after any re-login.
• PSRAM must run at 200 MHz or the 800×800 MIPI-DSI framebuffer underruns (lcd.dsi.dpi: can't fetch data… spam, LVGL lock never frees, task watchdog). CONFIG_SPIRAM_SPEED_200M only takes effect together with CONFIG_IDF_EXPERIMENTAL_FEATURES=y — otherwise it is silently dropped and you get 20 MHz. sdkconfig.defaults mirrors the official 08_lvgl_demo_v9 config.
• Non-interactive boot-log capture (avoid idf.py monitor, it's interactive): open /dev/ttyACM0 at 115200 with pyserial, pulse RTS to reset, read ~8 s. Verified boot is ~1.6 s from reset to desklock: DeskLock up.
• If the device doesn't enumerate, hold BOOT while pressing RESET for download mode.

Gateway (gateway/)

cd gateway
make setup          # venv + dev deps (no ML models)
make setup-speech   # additionally install faster-whisper + piper
make run            # uvicorn on :8600 with reload
make test           # pytest
make lint           # ruff check + format check
make typecheck      # mypy

• Config via DESKLOCK_* env vars — see src/desklock_gateway/config.py for the schema and defaults.
• stt.py / tts.py defer their heavy imports so the app boots without the speech extra — keep it that way so protocol tests stay fast.
• Deployment is CI-driven: pushing a v* tag makes Gitea Actions test, build, and push desklock-gateway:{latest,tag} to the registry and trigger Watchtower (.gitea/workflows/build.yml; needs REGISTRY_USER/REGISTRY_PASSWORD/ WATCHTOWER_HTTP_API_TOKEN secrets). Plain pushes to main run lint + tests only. The gateway deploys as part of the tatlock-ui Portainer stack — system-admin-toj/containers/stacks/tatlock-ui.yml (registered in CONTAINERS.md, port 8600). Stack updates go through the Portainer API on :8001 (JWT auth; recipe in system-admin-toj/containers/setup-new-host.md), not by editing files on disk.
• Verify speech changes against the live Speaches container with a real round trip (TTS → STT of a known phrase, expect the transcript back); warm timings to expect: STT ~0.3 s, TTS ~2 s per sentence.

Homelab context

• This server is tower-of-joy; the device, gateway, and Tatlock all share the LAN.
• Use tatlock.schweitz.internal:8000 (direct, no SSO) — the public tatlock.schweitz.net route sits behind Authentik and is not for machine-to-machine traffic.
• Git remote: git.schweitz.net (Gitea).