add AudioPen ingest pipeline and sync architecture setup docs
Voice-note capture as the primary mobile input channel. AudioPen polishes on their cloud, POSTs to a self-hosted receiver on the desktop (exposed via Tailscale Funnel so no router ports open and nothing behind Authentik). The receiver writes to inbox/raw/; a bash normaliser wraps each drop in frontmatter and moves it to inbox/; /triage-inbox routes them from there. scripts/audiopen-ingest.sh — 80-line bash normaliser. Idempotent; safe to re-run. Extracts title, slugifies, computes capture timestamp from mtime, rewrites with fleeting-note frontmatter. scripts/audiopen-webhook/main.py — stdlib-only Python HTTP server. Zero pip deps; binds to 127.0.0.1 by default. Accepts flexible payload shapes (title/name + body/output/summary/polished/ orig_transcript) so AudioPen version drift is logged rather than silently dropped. scripts/audiopen-webhook/*.example — systemd user-unit templates for the receiver and the path/service pair that fires the ingest wrapper on inbox/raw/ changes. scripts/pyproject.toml + README.md — Python venv convention. Zero deps today; venv location reserved at scripts/.venv/ (gitignored), manifest at scripts/pyproject.toml, bootstrap documented for both plain pip and uv. Optional-dependencies groups let individual tools pull what they need without bloating the whole env. docs/setup/audiopen.md — full setup guide: generating the shared secret, installing the systemd units, pairing with Tailscale Funnel, configuring AudioPen's webhook, and the IMAP-fallback path for setups that can't run a public-reachable receiver. docs/setup/sync.md — companion guide: Gitea SSH remote (works even with Authentik gating HTTPS), obsidian-git plugin configuration, Syncthing desktop↔phone pairing with the critical .stignore patterns, and the three phone-role tiers so the user can pick Tier 1 / 2 / 3 at their own pace. .gitignore gains **/.venv/, **/venv/, **/__pycache__/, and *.pyc so nobody accidentally commits a materialised environment. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.7
@@ -0,0 +1,210 @@
|
||||
# AudioPen → PKB setup
|
||||
|
||||
Voice-note capture pipeline. AudioPen records on phone, polishes on their cloud, POSTs
|
||||
the polished prose to your own infra. This vault's `inbox/raw/` receives the drop;
|
||||
`scripts/audiopen-ingest.sh` normalises frontmatter and moves it to `inbox/`;
|
||||
`/triage-inbox` routes it into `daily/`, `notes/`, `projects/`, or `reading/`.
|
||||
|
||||
Two ingest paths are supported. Pick one. Either works with the same
|
||||
`/triage-inbox` downstream.
|
||||
|
||||
---
|
||||
|
||||
## Path A — Webhook via Tailscale Funnel (recommended)
|
||||
|
||||
**Requires.** Tailscale installed on the desktop with Funnel enabled on your tailnet.
|
||||
Tailscale Funnel provides an outbound-initiated HTTPS endpoint (`*.ts.net`) with no
|
||||
router configuration and no exposed ports. Nothing sits behind Authentik; AudioPen
|
||||
hits the Funnel URL directly.
|
||||
|
||||
### 1. Generate a shared secret
|
||||
|
||||
```sh
|
||||
head -c 24 /dev/urandom | base64 | tr -d '/+=' > /tmp/secret
|
||||
cat /tmp/secret
|
||||
```
|
||||
|
||||
Store it privately; it becomes part of the webhook URL.
|
||||
|
||||
### 2. Install the receiver systemd unit
|
||||
|
||||
```sh
|
||||
mkdir -p ~/.config/audiopen-webhook ~/.config/systemd/user
|
||||
printf 'AUDIOPEN_WEBHOOK_SECRET=%s\n' "$(cat /tmp/secret)" > ~/.config/audiopen-webhook/env
|
||||
chmod 600 ~/.config/audiopen-webhook/env
|
||||
|
||||
cp /var/mnt/data/projects/council/scripts/audiopen-webhook/audiopen-webhook.service.example \
|
||||
~/.config/systemd/user/audiopen-webhook.service
|
||||
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now audiopen-webhook.service
|
||||
systemctl --user status audiopen-webhook.service # should show active (running)
|
||||
```
|
||||
|
||||
Tail logs:
|
||||
|
||||
```sh
|
||||
journalctl --user -u audiopen-webhook.service -f
|
||||
```
|
||||
|
||||
### 3. Install the ingest path + service units
|
||||
|
||||
```sh
|
||||
# audiopen-ingest.service — normalises raw drops
|
||||
cat > ~/.config/systemd/user/audiopen-ingest.service <<'EOF'
|
||||
[Unit]
|
||||
Description=Normalise AudioPen drops into council/inbox/
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/var/mnt/data/projects/council/scripts/audiopen-ingest.sh
|
||||
Environment=COUNCIL_REPO=/var/mnt/data/projects/council
|
||||
EOF
|
||||
|
||||
# audiopen-ingest.path — fires the service whenever inbox/raw/ gains a file
|
||||
cat > ~/.config/systemd/user/audiopen-ingest.path <<'EOF'
|
||||
[Unit]
|
||||
Description=Watch council/inbox/raw for new AudioPen drops
|
||||
|
||||
[Path]
|
||||
PathChanged=/var/mnt/data/projects/council/inbox/raw
|
||||
Unit=audiopen-ingest.service
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
EOF
|
||||
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now audiopen-ingest.path
|
||||
```
|
||||
|
||||
### 4. Expose the receiver via Tailscale Funnel
|
||||
|
||||
```sh
|
||||
tailscale funnel --bg 8765
|
||||
tailscale funnel status # show the public URL
|
||||
```
|
||||
|
||||
Your endpoint is now `https://<your-host>.<tailnet>.ts.net:443/`. AudioPen will
|
||||
POST to `https://<your-host>.<tailnet>.ts.net/audiopen/<secret>`.
|
||||
|
||||
### 5. Configure AudioPen
|
||||
|
||||
In AudioPen (web or mobile app): settings → integrations → add a generic webhook.
|
||||
|
||||
- **URL:** `https://<your-host>.<tailnet>.ts.net/audiopen/<secret>`
|
||||
- **Method:** POST
|
||||
- **Content-Type:** application/json
|
||||
- **Trigger:** after each recording is polished
|
||||
|
||||
AudioPen typically sends JSON with keys like `title`, `body`, `orig_transcript`.
|
||||
The receiver accepts any of these; if AudioPen changes shape, check
|
||||
`journalctl --user -u audiopen-webhook.service` for the logged payload and
|
||||
update `scripts/audiopen-webhook/main.py::extract_content()` if needed.
|
||||
|
||||
### 6. Test end-to-end
|
||||
|
||||
```sh
|
||||
SECRET=$(awk -F= '/AUDIOPEN_WEBHOOK_SECRET/ {print $2}' ~/.config/audiopen-webhook/env)
|
||||
FUNNEL=$(tailscale funnel status | awk '/https:/ {print $1; exit}')
|
||||
|
||||
curl -X POST -H 'Content-Type: application/json' \
|
||||
-d '{"title":"Test","body":"Hello from curl. This is a test drop."}' \
|
||||
"${FUNNEL%/}/audiopen/${SECRET}"
|
||||
# → ok
|
||||
|
||||
ls /var/mnt/data/projects/council/inbox/raw # should contain the test file briefly
|
||||
sleep 2
|
||||
ls /var/mnt/data/projects/council/inbox # should now contain the normalised file
|
||||
```
|
||||
|
||||
Then record an actual voice note in AudioPen, wait for polishing, confirm the same
|
||||
flow. Run `/triage-inbox` in Claude Code when you're ready to route the contents.
|
||||
|
||||
---
|
||||
|
||||
## Path B — Email → IMAP fetch (no-open-ports fallback)
|
||||
|
||||
Use this if you don't run Tailscale / Cloudflare Tunnel and don't want to stand one
|
||||
up. Fully outbound: AudioPen emails each note; a cron on the desktop pulls via IMAP
|
||||
and drops files in `inbox/raw/`.
|
||||
|
||||
### 1. Dedicated email address
|
||||
|
||||
Create or repurpose a mailbox for AudioPen's output. Options:
|
||||
|
||||
- Fastmail alias → Fastmail IMAP on desktop (works).
|
||||
- Self-hosted mail server → pull via IMAP from localhost (works).
|
||||
- Gmail "+audiopen" alias → fine but adds Google to the dependency chain.
|
||||
|
||||
### 2. IMAP fetch script (stub — you'll customise for your mail server)
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""Minimal IMAP fetcher: pulls unread messages from a mailbox, writes them as
|
||||
bare markdown to inbox/raw/. Uses stdlib imaplib. Keep credentials in
|
||||
~/.config/audiopen-mail/env or similar."""
|
||||
import imaplib, email, os, re, pathlib, datetime
|
||||
|
||||
HOST = os.environ["IMAP_HOST"]
|
||||
USER = os.environ["IMAP_USER"]
|
||||
PASS = os.environ["IMAP_PASS"]
|
||||
MBOX = os.environ.get("IMAP_MAILBOX", "INBOX")
|
||||
INBOX = pathlib.Path(os.environ.get("COUNCIL_INBOX_RAW",
|
||||
"/var/mnt/data/projects/council/inbox/raw"))
|
||||
INBOX.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
m = imaplib.IMAP4_SSL(HOST)
|
||||
m.login(USER, PASS)
|
||||
m.select(MBOX)
|
||||
typ, data = m.search(None, 'UNSEEN')
|
||||
for num in data[0].split():
|
||||
typ, msg_data = m.fetch(num, '(RFC822)')
|
||||
msg = email.message_from_bytes(msg_data[0][1])
|
||||
title = (msg.get("Subject") or "Untitled").strip()
|
||||
body = ""
|
||||
if msg.is_multipart():
|
||||
for p in msg.walk():
|
||||
if p.get_content_type() == "text/plain":
|
||||
body = p.get_payload(decode=True).decode('utf-8', errors='replace')
|
||||
break
|
||||
else:
|
||||
body = msg.get_payload(decode=True).decode('utf-8', errors='replace')
|
||||
slug = re.sub(r"[^a-z0-9]+", "-", title.lower()).strip("-")[:60] or "untitled"
|
||||
ts = datetime.datetime.now().strftime("%Y-%m-%d-%H%M")
|
||||
(INBOX / f"{ts}_{slug}.md").write_text(f"# {title}\n\n{body}\n")
|
||||
m.store(num, '+FLAGS', '\\Seen')
|
||||
m.logout()
|
||||
```
|
||||
|
||||
Save as `scripts/audiopen-imap-fetch.py`. Run it every N minutes via a systemd
|
||||
user-timer or cron. The same `audiopen-ingest.path` unit (from Path A, step 3)
|
||||
picks up the drops from there.
|
||||
|
||||
### 3. Configure AudioPen
|
||||
|
||||
Settings → integrations → email output → recipient = your dedicated address.
|
||||
Each recording → one email. Done.
|
||||
|
||||
---
|
||||
|
||||
## Routine
|
||||
|
||||
Once everything is wired:
|
||||
|
||||
1. **Capture**: voice note → AudioPen → (webhook or email) → desktop.
|
||||
2. **Ingest**: automatic — you don't run anything; systemd fires on file change.
|
||||
3. **Route**: when you want, open a Claude Code session in the vault and run
|
||||
`/triage-inbox`. Koskela proposes destinations; you approve in batch.
|
||||
4. **Weekly/monthly**: the promoted notes are already in their homes; `/reflect`,
|
||||
`/review-week`, and `/review-month` will see them naturally.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Check |
|
||||
|---|---|
|
||||
| Files pile up in `inbox/raw/` but never normalise | `systemctl --user status audiopen-ingest.path audiopen-ingest.service` — path unit enabled? service errored? Run the script manually once: `bash scripts/audiopen-ingest.sh`. |
|
||||
| Webhook POSTs 200 but no file lands | Check the systemd logs for the receiver: `journalctl --user -u audiopen-webhook.service -n 50`. The logged path should be writable and under `inbox/raw/`. |
|
||||
| Webhook POSTs 404 | Secret mismatch between AudioPen URL and `~/.config/audiopen-webhook/env`. Regenerate and re-set both sides. |
|
||||
| Funnel URL unreachable | `tailscale status`; `tailscale funnel status`. Re-run `tailscale funnel --bg 8765`. Confirm Funnel is enabled on your tailnet (admin → DNS → HTTPS Certificates; admin → Access controls → Funnel). |
|
||||
| Normalised file missing frontmatter | Bug in the ingest wrapper; open the raw file, run the script with `bash -x scripts/audiopen-ingest.sh` to trace. |
|
||||
Reference in New Issue
Block a user