desklock/AGENTS.md

AGENTS.md

Operational protocols and architecture for AI assistants working on DeskLock. Read docs/architecture.md before making design changes.

What this project is

DeskLock is the living-room visual/audio endpoint for Tatlock, the homelab butler (/mnt/media/Projects/tatlock, API at http://tatlock.schweitz.internal:8000). Two halves, one repo:

• firmware/ — ESP-IDF (C, LVGL 9) app for the Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C (3.4" round 800×800 touch display, dual mics + ES7210 AEC, ES8311 codec + speaker).
• gateway/ — Python FastAPI container on tower-of-joy orchestrating STT → chat (Tatlock /v1/chat/completions) → TTS. Listens on port 8600. STT/TTS models live in the shared Speaches container (live on port 8601, OpenAI-format API), not in the gateway image; stt.py/tts.py are pluggable backends (speaches default, embedded fallback needing the [speech] extra). Gateway needs Python ≥ 3.11 (no ceiling; the container runs 3.13) — but system python3 on tower-of-joy is 3.8, so make setup explicitly uses python3.12. No local audio resampling in the default path: the gateway requests 16 kHz output via Speaches' sample_rate extension (verified live).

The device and gateway speak a WebSocket protocol defined in docs/architecture.md. That doc is the contract — update it in the same change as any protocol edit on either side.

The face (black screen, ASCII glyph expressions, matrix rain as activity signal) is designed in sim/face/index.html — the design source of truth — and specified in the "Face design" section of docs/architecture.md. Change the sim and the doc together; the LVGL implementation follows them. Verify sim changes visually with ~/bin/claude-screenshot (note: the tool uses --virtual-time-budget, which starves requestAnimationFrame — drive sim animation with setInterval, which also mirrors LVGL timers).

Hard rules

• Keep the firmware thin. No STT, no TTS, no conversation logic on the device. If a feature needs intelligence, it goes in the gateway or in Tatlock itself.
• Never modify Tatlock from this repo. It is a separate project with its own repo. DeskLock consumes its public API only.
• Secrets (Wi-Fi credentials, any future API keys) never go in source. Firmware gets them via a gitignored firmware/secrets.h (see AGENTS notes below) or NVS; the gateway via environment variables (DESKLOCK_*).

Firmware (firmware/)

• Toolchain: ESP-IDF ≥ 5.4 (not Arduino, not PlatformIO). Target esp32p4.
• BSP: waveshare/esp32_p4_wifi6_touch_lcd_xc from the ESP Component Registry (pulled automatically via main/idf_component.yml).
• Reference implementations: waveshareteam/ESP32-P4-WIFI6-Touch-LCD-XC examples/esp-idf/ — notably 08_lvgl_demo_v9 (display), 06_I2SCodec (audio), 04_wifistation (Wi-Fi via ESP-Hosted). When wiring a new peripheral, check the official example first; do not guess pin mappings.
• ⚠️ Unverified scaffold: the BSP API calls in desklock_main.c and the sdkconfig.defaults values were written before the first successful build. Validate against the official examples on first bring-up, then delete this warning.

# One-time: install ESP-IDF (not yet installed on tower-of-joy)
git clone -b v5.5 --recursive https://github.com/espressif/esp-idf.git ~/esp-idf
~/esp-idf/install.sh esp32p4

# Every shell:
source ~/esp-idf/export.sh

# Build / flash / monitor (device is on USB-C; check `ls /dev/ttyACM*`)
cd firmware
idf.py set-target esp32p4        # once
idf.py build
idf.py -p /dev/ttyACM0 flash monitor   # Ctrl+] exits monitor

Flashing requires the dialout group (or run with sudo once and fix the group). If the device doesn't enumerate, hold BOOT while pressing RESET to enter download mode.

Gateway (gateway/)

cd gateway
make setup          # venv + dev deps (no ML models)
make setup-speech   # additionally install faster-whisper + piper
make run            # uvicorn on :8600 with reload
make test           # pytest
make lint           # ruff check + format check
make typecheck      # mypy

• Config via DESKLOCK_* env vars — see src/desklock_gateway/config.py for the schema and defaults.
• stt.py / tts.py defer their heavy imports so the app boots without the speech extra — keep it that way so protocol tests stay fast.
• Deployment is CI-driven: pushing a v* tag makes Gitea Actions test, build, and push desklock-gateway:{latest,tag} to the registry and trigger Watchtower (.gitea/workflows/build.yml; needs REGISTRY_USER/REGISTRY_PASSWORD/ WATCHTOWER_HTTP_API_TOKEN secrets). Plain pushes to main run lint + tests only. The gateway deploys as part of the tatlock-ui Portainer stack — system-admin-toj/containers/stacks/tatlock-ui.yml (registered in CONTAINERS.md, port 8600). Stack updates go through the Portainer API on :8001 (JWT auth; recipe in system-admin-toj/containers/setup-new-host.md), not by editing files on disk.
• Verify speech changes against the live Speaches container with a real round trip (TTS → STT of a known phrase, expect the transcript back); warm timings to expect: STT ~0.3 s, TTS ~2 s per sentence.

Homelab context

• This server is tower-of-joy; the device, gateway, and Tatlock all share the LAN.
• Use tatlock.schweitz.internal:8000 (direct, no SSO) — the public tatlock.schweitz.net route sits behind Authentik and is not for machine-to-machine traffic.
• Git remote: git.schweitz.net (Gitea).