feat(dev): device screenshot over USB
Adds a way to capture the live LVGL screen off the device and rebuild it as a PNG on the host, so UI changes can be verified remotely without a camera. A watcher task polls the USB-serial-JTAG RX for a trigger byte and streams the current screen as raw RGB565 straight to the USB FIFO (framed by ###SHOT_BEGIN/END### with a CRC); firmware/tools/device_shot.py and the device-screenshot skill drive it from the host. Writing straight to the USB FIFO bypasses the primary UART console, which at 115200 baud would take ~37s per frame. The whole capability is behind DESKLOCK_DEVMODE (off by default, enable at deploy time with -DDESKLOCK_DEVMODE=ON) so production spends no internal RAM on the watcher and nothing extra runs on the render path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LKPbR6DY2JygHbyLjxm7Uu
This commit is contained in:
Claude Opus 4.8
@@ -0,0 +1,83 @@
|
||||
---
|
||||
name: device-screenshot
|
||||
description: >
|
||||
Capture a screenshot of the live DeskLock ESP32-P4 display over USB and view
|
||||
it as a PNG. Use whenever you need to SEE what's on the device screen — verify
|
||||
a face/UI change, check the tap controls overlay, confirm a state (idle,
|
||||
listening, effort), or debug layout remotely without the user's camera.
|
||||
Triggers: "screenshot the device", "what's on the screen", "capture the
|
||||
display", "show me the face", "did the UI change land".
|
||||
---
|
||||
|
||||
# DeskLock device screenshot over USB
|
||||
|
||||
Grabs the current LVGL screen off the device and rebuilds it as a PNG on the
|
||||
host. No camera, no gateway — it rides the USB serial that's already attached for
|
||||
flashing.
|
||||
|
||||
## How it works (so you can debug it)
|
||||
|
||||
The firmware (`main/face.c` `face_screenshot_dump`, gated by `DESKLOCK_DEVMODE`
|
||||
in `desklock_main.c`) runs a task that watches the **USB-serial-JTAG RX FIFO**
|
||||
for a trigger byte, renders the active screen with `lv_snapshot_take_to_draw_buf`
|
||||
into a PSRAM buffer, 2×-downscales to 400×400, and streams it as **raw binary**
|
||||
straight to the USB-serial-JTAG TX FIFO — framed by a text header
|
||||
`###SHOT_BEGIN … bytes=N crc=0x… bin=1###` + N bytes + `###SHOT_END###`. The host
|
||||
tool `firmware/tools/device_shot.py` sends the trigger, reads N bytes, checks the
|
||||
CRC (retrying on the rare dropped frame), and writes a PNG.
|
||||
|
||||
Two things this design is deliberately built around, learned the hard way:
|
||||
- **Don't use `printf`.** The primary console is UART at 115200 baud (~11 KB/s) —
|
||||
a frame would take ~37 s. Writing straight to the USB FIFO runs at USB speed
|
||||
(~1 s). That's why the dump uses `usb_serial_jtag_ll_write_txfifo`, not stdout.
|
||||
- **Opening the port does NOT reset the P4** (unlike esptool), and there's no USB
|
||||
stdin, so the trigger is a byte the firmware polls for — on-demand, no reboot,
|
||||
captures whatever state is currently on screen.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Firmware flashed with **dev mode ON** — it's OFF by default (production spends
|
||||
no internal RAM on the watcher and nothing extra runs on the render path). Flash
|
||||
a dev build with:
|
||||
```bash
|
||||
cd firmware && sg dialout -c "bash -c 'source ~/esp-idf/export.sh && idf.py -DDESKLOCK_DEVMODE=ON -p /dev/ttyACM0 flash'"
|
||||
```
|
||||
If screenshots return "no frame", the flashed build is production — reflash with
|
||||
`-DDESKLOCK_DEVMODE=ON`. Back to production: `-DDESKLOCK_DEVMODE=OFF` (the value
|
||||
sticks in the CMake cache until you flip it). Leave it ON for a UI-dev session
|
||||
(you're reflashing for UI changes anyway); flip OFF for the final/production flash.
|
||||
- Device on `/dev/ttyACM0`. The port needs the `dialout` group, so run the tool
|
||||
under `sg dialout -c '…'` (this login session predates dialout membership).
|
||||
- Nothing else holding the port (no `idf.py monitor` running) — one owner only.
|
||||
|
||||
## Take a shot
|
||||
|
||||
From the `desklock` repo root (`/mnt/media/Projects/desklock`):
|
||||
|
||||
```bash
|
||||
# current screen (whatever state the device is in right now)
|
||||
sg dialout -c "python3 firmware/tools/device_shot.py /tmp/shot.png"
|
||||
|
||||
# force the tap controls overlay in-frame (mic + volume) — for verifying it
|
||||
# remotely since you can't physically tap
|
||||
sg dialout -c "python3 firmware/tools/device_shot.py /tmp/shot.png --overlay"
|
||||
```
|
||||
|
||||
Then **Read `/tmp/shot.png`** to view it. A clean run prints
|
||||
`[try 1] 400x400 320000B crc … OK` and takes ~5 s.
|
||||
|
||||
Trigger bytes: `s` = current screen, `o` = force overlay. The tool retries up to
|
||||
4× on a CRC mismatch (occasional console contention), so a transient bad frame
|
||||
self-heals.
|
||||
|
||||
## Notes / caveats
|
||||
|
||||
- **RAM / render path:** the watcher costs one ~5 KB internal-RAM task, so it's
|
||||
behind `DESKLOCK_DEVMODE` (off by default). Internal RAM is the scarce resource
|
||||
on this board (it caps the rain sprite pool). Never ship a production build with
|
||||
it on.
|
||||
- 400×400 is plenty for layout/UI checks. To change resolution, adjust `SHOT_DS`
|
||||
in `face.c` (the downscale factor) — the host reads the size from the header.
|
||||
- If you just reflashed, wait ~7 s for boot before the first shot.
|
||||
- Only the USB console path is used; this does not touch the device↔gateway
|
||||
WebSocket protocol.
|
||||
Reference in New Issue
Block a user