add AudioPen ingest pipeline and sync architecture setup docs

Voice-note capture as the primary mobile input channel. AudioPen polishes on their cloud, POSTs to a self-hosted receiver on the desktop (exposed via Tailscale Funnel so no router ports open and nothing behind Authentik). The receiver writes to inbox/raw/; a bash normaliser wraps each drop in frontmatter and moves it to inbox/; /triage-inbox routes them from there. scripts/audiopen-ingest.sh — 80-line bash normaliser. Idempotent; safe to re-run. Extracts title, slugifies, computes capture timestamp from mtime, rewrites with fleeting-note frontmatter. scripts/audiopen-webhook/main.py — stdlib-only Python HTTP server. Zero pip deps; binds to 127.0.0.1 by default. Accepts flexible payload shapes (title/name + body/output/summary/polished/ orig_transcript) so AudioPen version drift is logged rather than silently dropped. scripts/audiopen-webhook/*.example — systemd user-unit templates for the receiver and the path/service pair that fires the ingest wrapper on inbox/raw/ changes. scripts/pyproject.toml + README.md — Python venv convention. Zero deps today; venv location reserved at scripts/.venv/ (gitignored), manifest at scripts/pyproject.toml, bootstrap documented for both plain pip and uv. Optional-dependencies groups let individual tools pull what they need without bloating the whole env. docs/setup/audiopen.md — full setup guide: generating the shared secret, installing the systemd units, pairing with Tailscale Funnel, configuring AudioPen's webhook, and the IMAP-fallback path for setups that can't run a public-reachable receiver. docs/setup/sync.md — companion guide: Gitea SSH remote (works even with Authentik gating HTTPS), obsidian-git plugin configuration, Syncthing desktop↔phone pairing with the critical .stignore patterns, and the three phone-role tiers so the user can pick Tier 1 / 2 / 3 at their own pace. .gitignore gains **/.venv/, **/venv/, **/__pycache__/, and *.pyc so nobody accidentally commits a materialised environment. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 20:17:48 +02:00
parent 2d5615b1ba
commit 2c8df9425b
10 changed files with 850 additions and 0 deletions
@@ -0,0 +1,62 @@
+# scripts/
+
+Operational scripts for the PKB. Everything here is side-effectful — writing files,
+running webhooks, ingesting voice notes. Keep it small; anything analytical belongs
+in `.claude/commands/` or in the `mql` CLI.
+
+## Current contents
+
+| Path | Language | Purpose |
+|---|---|---|
+| `audiopen-ingest.sh` | bash | Normalises raw AudioPen drops (`inbox/raw/*.md`) into properly-frontmattered fleeting notes in `inbox/`. Idempotent. |
+| `audiopen-webhook/main.py` | Python (stdlib) | Tiny HTTP server that receives AudioPen webhook POSTs and writes to `inbox/raw/`. Designed to sit behind Tailscale Funnel. |
+| `audiopen-webhook/*.example` | systemd | User-unit examples for the receiver + the path/service that runs `audiopen-ingest.sh` on file change. |
+
+Setup guide for the full pipeline: [`docs/setup/audiopen.md`](../docs/setup/audiopen.md).
+
+## Python venv convention
+
+Python code in this repo is **stdlib-only as of now**. No pip-installable dependencies.
+If/when a real dependency arrives (e.g. `watchdog`, `imap-tools`, anything heavier
+than stdlib), the convention is:
+
+- Venv lives at **`scripts/.venv/`** (gitignored — every user materialises locally).
+- Dependencies tracked in **`scripts/pyproject.toml`** (committed).
+- Bootstrap on a fresh machine:
+
+  ```sh
+  cd scripts
+  python3 -m venv .venv
+  .venv/bin/pip install -e .
+  ```
+
+  Or with `uv` (recommended if available):
+
+  ```sh
+  cd scripts
+  uv venv
+  uv pip install -e .
+  ```
+
+- Scripts that need the venv should either:
+  - Invoke `scripts/.venv/bin/python3` explicitly in their shebang or systemd `ExecStart`.
+  - Or document activation (`source scripts/.venv/bin/activate`) before running.
+
+Keeping the venv under `scripts/` rather than the repo root keeps the Obsidian file
+tree uncluttered — `scripts/` is already excluded from Obsidian's index
+(`userIgnoreFilters` in `.obsidian/app.json`) so nothing Python-related will ever
+show in the graph or search.
+
+**Don't create the venv preemptively.** When the first dependency lands, that's when
+you create it. Keeps the repo lighter and avoids stale venv directories.
+
+The `pyproject.toml` uses `[project.optional-dependencies]` to split deps by tool
+— e.g. `pip install -e .[imap]` installs only the IMAP fetcher's deps. This lets
+the webhook receiver stay dep-free while heavier tools can pull in what they need.
+
+## Shell scripts
+
+Kept in plain bash. Portable to any Linux running GNU coreutils (which Bazzite does).
+No zsh-isms, no bashisms beyond `set -euo pipefail` and arrays. If a script grows
+past ~200 lines or needs real argument parsing, rewrite it in Python (and you're now
+back in the venv convention above).
@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+# audiopen-ingest.sh
+#
+# Normalises raw AudioPen drops in inbox/raw/ into proper fleeting notes in inbox/.
+# Adds frontmatter, renames to timestamp-slug format, moves to inbox/.
+# Idempotent: safe to re-run. Destructive on inbox/raw/ (files are moved, not copied).
+#
+# Environment:
+#   COUNCIL_REPO  — vault root. Default: /var/mnt/data/projects/council
+#
+# Exit status:
+#   0 = success (even if zero files were processed)
+#   1 = unexpected error
+
+set -euo pipefail
+
+REPO="${COUNCIL_REPO:-/var/mnt/data/projects/council}"
+RAW="$REPO/inbox/raw"
+OUT="$REPO/inbox"
+
+if [ ! -d "$RAW" ]; then
+    echo "audiopen-ingest: raw dir $RAW does not exist; nothing to do." >&2
+    exit 0
+fi
+mkdir -p "$OUT"
+
+shopt -s nullglob
+count=0
+for f in "$RAW"/*.md; do
+    # mtime as ISO-8601 (date -r works on GNU coreutils; Bazzite ships it)
+    captured=$(date -r "$f" -Iseconds)
+    ts=$(date -r "$f" +%Y-%m-%d-%H%M)
+
+    # Title: first `# ` line, else filename (without extension)
+    title=$(grep -m1 '^# ' "$f" 2>/dev/null | sed 's/^# *//' || true)
+    if [ -z "$title" ]; then
+        title=$(basename "$f" .md)
+    fi
+
+    # Slugify: lowercase, non-alnum → dash, collapse dashes, trim, cap at 60
+    slug=$(printf '%s' "$title" | tr '[:upper:]' '[:lower:]' \
+        | sed 's/[^a-z0-9]/-/g' \
+        | sed 's/--*/-/g' \
+        | sed 's/^-*//;s/-*$//' \
+        | cut -c1-60)
+    [ -z "$slug" ] && slug="untitled"
+
+    out="$OUT/${ts}_${slug}.md"
+
+    # If target exists (same timestamp + slug), append a numeric suffix
+    i=2
+    while [ -e "$out" ]; do
+        out="$OUT/${ts}_${slug}_${i}.md"
+        i=$((i+1))
+    done
+
+    # Body: strip the first `# title` line so we regenerate it in the normalised form
+    content=$(awk 'BEGIN{stripped=0} /^# / && !stripped { stripped=1; next } { print }' "$f")
+
+    {
+        echo "---"
+        echo "type: fleeting"
+        echo "source: audiopen"
+        echo "captured: $captured"
+        echo "slug: $slug"
+        echo "---"
+        echo
+        echo "# $title"
+        echo
+        printf '%s\n' "$content"
+    } > "$out"
+
+    rm "$f"
+    echo "ingested: $(basename "$f") → $(basename "$out")"
+    count=$((count+1))
+done
+
+echo "audiopen-ingest: $count file(s) normalised."
@@ -0,0 +1,69 @@
+# audiopen-webhook
+
+Zero-dependency Python HTTP server that receives AudioPen webhook POSTs and
+writes bare markdown files to `inbox/raw/`. The separate `scripts/audiopen-ingest.sh`
+wrapper normalises those files into proper fleeting notes under `inbox/`.
+
+## Why this shape
+
+- **Receiver is minimal (stdlib-only http.server).** No Flask, no runtime to keep
+  updated, ~150 LOC. Reads AudioPen's JSON payload, writes one file, logs to
+  stderr. Anything fancier belongs in the ingest wrapper.
+- **Two-step raw → normalised.** Keeps the receiver stupid; all schema decisions
+  (frontmatter, slug format, filename) live in one shell script you can read
+  in 30 seconds.
+- **Designed for Tailscale Funnel.** Binds to 127.0.0.1 by default. Tailscale
+  Funnel (or Cloudflare Tunnel, or an SSH remote forward, or ngrok) terminates
+  the public endpoint and proxies to 127.0.0.1:8765. No ports opened on your
+  router; nothing behind Authentik.
+
+## Setup
+
+Full step-by-step lives in [`docs/setup/audiopen.md`](../../docs/setup/audiopen.md).
+Quick version:
+
+1. **Generate a secret**: `head -c 24 /dev/urandom | base64 | tr -d '/+='`.
+2. **Configure AudioPen** to POST JSON to `https://<your-funnel>.ts.net/audiopen/<secret>`.
+3. **Install the systemd user units**:
+   ```sh
+   mkdir -p ~/.config/audiopen-webhook ~/.config/systemd/user
+   echo "AUDIOPEN_WEBHOOK_SECRET=<secret>" > ~/.config/audiopen-webhook/env
+   chmod 600 ~/.config/audiopen-webhook/env
+   cp audiopen-webhook.service.example ~/.config/systemd/user/audiopen-webhook.service
+   # create two more systemd files for the ingest path/service per
+   # audiopen-ingest.path.example
+   systemctl --user daemon-reload
+   systemctl --user enable --now audiopen-webhook.service audiopen-ingest.path
+   ```
+4. **Expose via Tailscale Funnel**:
+   ```sh
+   tailscale funnel --bg 8765
+   ```
+5. **Test**:
+   ```sh
+   curl -X POST -H 'Content-Type: application/json' \
+     -d '{"title":"Test","body":"Hello from curl"}' \
+     https://<your-funnel>.ts.net/audiopen/<secret>
+   # → OK
+   ls ~/path/to/vault/inbox/raw   # should see the test file
+   # wait a beat for the inotify path unit to fire
+   ls ~/path/to/vault/inbox       # should see the normalised file
+   ```
+
+## Payload shape
+
+The receiver accepts any of these JSON keys for the title: `title`, `name`.
+For the body: `body`, `output`, `summary`, `polished`, `orig_transcript`. Unknown
+keys are ignored; the raw payload is logged to stderr so you can see what
+AudioPen actually sent if shape drifts.
+
+If AudioPen's webhook format changes in a way the current extractor misses,
+check `journalctl --user -u audiopen-webhook.service` for the logged shape and
+update the key list in `extract_content()` at the top of `main.py`.
+
+## Alternative: email + IMAP fetch
+
+If running a public-reachable receiver isn't viable, the plan has an email-path
+fallback documented in `docs/setup/audiopen.md`. It's entirely outbound: AudioPen
+emails each note to a dedicated address, and a cron on the desktop pulls via
+IMAP and drops files in `inbox/raw/`. Same ingest wrapper handles the rest.
@@ -0,0 +1,34 @@
+# audiopen-ingest path + service units — example
+#
+# These two units let the ingest wrapper fire automatically whenever
+# inbox/raw/ gains a file. Install:
+#
+#   cp audiopen-ingest.path.example   ~/.config/systemd/user/audiopen-ingest.path
+#   cp audiopen-ingest.service.example ~/.config/systemd/user/audiopen-ingest.service
+#   systemctl --user daemon-reload
+#   systemctl --user enable --now audiopen-ingest.path
+#
+# (Adjust %h paths or vault location as needed.)
+
+# ---- audiopen-ingest.path ----
+[Unit]
+Description=Watch council/inbox/raw for new AudioPen drops
+
+[Path]
+PathChanged=/var/mnt/data/projects/council/inbox/raw
+Unit=audiopen-ingest.service
+
+[Install]
+WantedBy=default.target
+
+# ---- audiopen-ingest.service ----
+# (Put this in a separate file — audiopen-ingest.service — systemd does not
+# allow two units in one file.)
+#
+# [Unit]
+# Description=Normalise AudioPen drops into council/inbox/
+#
+# [Service]
+# Type=oneshot
+# ExecStart=/var/mnt/data/projects/council/scripts/audiopen-ingest.sh
+# Environment=COUNCIL_REPO=/var/mnt/data/projects/council
@@ -0,0 +1,36 @@
+# audiopen-webhook user systemd unit — example
+#
+# Install to ~/.config/systemd/user/audiopen-webhook.service, then:
+#   systemctl --user daemon-reload
+#   systemctl --user enable --now audiopen-webhook.service
+#
+# Check logs:
+#   journalctl --user -u audiopen-webhook.service -f
+#
+# The shared secret lives in ~/.config/audiopen-webhook/env (chmod 600).
+# Example env file contents:
+#   AUDIOPEN_WEBHOOK_SECRET=<random-32-char-string>
+#   # Optional overrides:
+#   # AUDIOPEN_WEBHOOK_HOST=127.0.0.1
+#   # AUDIOPEN_WEBHOOK_PORT=8765
+#   # COUNCIL_INBOX_RAW=/var/mnt/data/projects/council/inbox/raw
+
+[Unit]
+Description=AudioPen webhook receiver (writes to council/inbox/raw)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+EnvironmentFile=%h/.config/audiopen-webhook/env
+ExecStart=/usr/bin/python3 %h/projects/council/scripts/audiopen-webhook/main.py
+Restart=on-failure
+RestartSec=5
+# Narrow permissions:
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=read-only
+ReadWritePaths=/var/mnt/data/projects/council/inbox/raw
+
+[Install]
+WantedBy=default.target
@@ -0,0 +1,139 @@
+#!/usr/bin/env python3
+"""AudioPen webhook receiver.
+
+Listens for POST requests at /audiopen/<shared-secret> and writes the polished-prose
+payload to COUNCIL_INBOX_RAW as a bare markdown file. A separate cron/systemd job
+(or inotify) then runs audiopen-ingest.sh to normalise into the PKB inbox.
+
+Zero external dependencies (stdlib only). Designed to sit behind Tailscale Funnel
+or similar outbound-initiated tunnel — binds to 127.0.0.1 by default.
+
+Environment:
+    AUDIOPEN_WEBHOOK_SECRET   required. Shared secret in the URL path.
+    AUDIOPEN_WEBHOOK_HOST     default 127.0.0.1
+    AUDIOPEN_WEBHOOK_PORT     default 8765
+    COUNCIL_INBOX_RAW         default /var/mnt/data/projects/council/inbox/raw
+
+AudioPen's webhook payload shape varies by version; we accept any of these JSON keys
+for the body: `body`, `output`, `summary`, `polished`. For the title: `title`,
+`name`. Unknown fields are ignored. The raw payload is kept in an HTTP access log
+on stderr so you can inspect it if AudioPen changes shape.
+"""
+
+from __future__ import annotations
+import datetime as _dt
+import json
+import os
+import pathlib
+import re
+import sys
+from http.server import BaseHTTPRequestHandler, HTTPServer
+
+
+INBOX_RAW = pathlib.Path(
+    os.environ.get("COUNCIL_INBOX_RAW", "/var/mnt/data/projects/council/inbox/raw")
+)
+SECRET = os.environ.get("AUDIOPEN_WEBHOOK_SECRET")
+HOST = os.environ.get("AUDIOPEN_WEBHOOK_HOST", "127.0.0.1")
+PORT = int(os.environ.get("AUDIOPEN_WEBHOOK_PORT", "8765"))
+
+_SLUG_RE = re.compile(r"[^a-z0-9]+")
+
+
+def slugify(text: str, max_len: int = 60) -> str:
+    s = _SLUG_RE.sub("-", text.strip().lower()).strip("-")
+    return (s or "untitled")[:max_len]
+
+
+def extract_content(payload: dict) -> tuple[str, str]:
+    title = str(
+        payload.get("title")
+        or payload.get("name")
+        or ""
+    ).strip()
+    body = str(
+        payload.get("body")
+        or payload.get("output")
+        or payload.get("summary")
+        or payload.get("polished")
+        or payload.get("orig_transcript")
+        or ""
+    ).strip()
+    if not title and body:
+        title = body.splitlines()[0][:60]
+    return title or "Untitled", body
+
+
+class Handler(BaseHTTPRequestHandler):
+    server_version = "audiopen-webhook/1"
+
+    def _reply(self, status: int, msg: str = "") -> None:
+        self.send_response(status)
+        self.send_header("Content-Type", "text/plain; charset=utf-8")
+        self.send_header("Content-Length", str(len(msg)))
+        self.end_headers()
+        if msg:
+            self.wfile.write(msg.encode())
+
+    def do_POST(self) -> None:
+        parts = [p for p in self.path.split("/") if p]
+        if len(parts) < 2 or parts[0] != "audiopen" or parts[1] != SECRET:
+            self._reply(404, "not found")
+            return
+
+        length = int(self.headers.get("Content-Length", "0") or 0)
+        raw = self.rfile.read(length) if length > 0 else b""
+        try:
+            payload = json.loads(raw) if raw else {}
+        except json.JSONDecodeError:
+            self._reply(400, "invalid JSON")
+            return
+        if not isinstance(payload, dict):
+            self._reply(400, "expected JSON object")
+            return
+
+        title, body = extract_content(payload)
+        if not body:
+            self._reply(400, "empty body")
+            return
+
+        slug = slugify(title)
+        ts = _dt.datetime.now().strftime("%Y-%m-%d-%H%M")
+        filename = f"{ts}_{slug}.md"
+
+        INBOX_RAW.mkdir(parents=True, exist_ok=True)
+        target = INBOX_RAW / filename
+        # Avoid overwriting on same-minute collision
+        suffix = 2
+        while target.exists():
+            target = INBOX_RAW / f"{ts}_{slug}_{suffix}.md"
+            suffix += 1
+
+        target.write_text(f"# {title}\n\n{body}\n", encoding="utf-8")
+        sys.stderr.write(f"[audiopen-webhook] wrote {target}\n")
+        self._reply(200, "ok\n")
+
+    def do_GET(self) -> None:
+        # Health check at /health; everything else returns 404.
+        if self.path == "/health":
+            self._reply(200, "ok\n")
+        else:
+            self._reply(404, "not found")
+
+    def log_message(self, fmt: str, *args) -> None:
+        sys.stderr.write(f"[audiopen-webhook] {self.address_string()} {fmt % args}\n")
+
+
+def main() -> int:
+    if not SECRET:
+        sys.stderr.write("AUDIOPEN_WEBHOOK_SECRET env var is required.\n")
+        return 1
+    sys.stderr.write(
+        f"[audiopen-webhook] listening on http://{HOST}:{PORT}, inbox_raw={INBOX_RAW}\n"
+    )
+    HTTPServer((HOST, PORT), Handler).serve_forever()
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
@@ -0,0 +1,42 @@
+[project]
+name = "council-scripts"
+version = "0.1.0"
+description = "Operational scripts for the Council PKB — AudioPen ingest, webhook receiver, inbox normalisation."
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = []
+
+# Empty today. Stdlib-only code in this directory. Add entries here when a real
+# pip-installable dependency arrives (e.g. watchdog, imap-tools). Bootstrap:
+#
+#   cd scripts
+#   python3 -m venv .venv
+#   .venv/bin/pip install -e .
+#
+# Or with uv:
+#
+#   cd scripts
+#   uv venv
+#   uv pip install -e .
+
+[project.optional-dependencies]
+# Example shape for when deps arrive — split by tool so you can install only what
+# you run. Uncomment and populate as needed.
+#
+# imap = [
+#     "imap-tools>=1.4.0",
+# ]
+# watch = [
+#     "watchdog>=4.0.0",
+# ]
+
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools]
+# There's no package to build — these are standalone scripts. Point setuptools
+# at an empty package list so `pip install -e .` succeeds as a no-op installer
+# for the dependency manifest, without trying to import a module named
+# `council-scripts`.
+packages = []