add AudioPen ingest pipeline and sync architecture setup docs
Voice-note capture as the primary mobile input channel. AudioPen polishes on their cloud, POSTs to a self-hosted receiver on the desktop (exposed via Tailscale Funnel so no router ports open and nothing behind Authentik). The receiver writes to inbox/raw/; a bash normaliser wraps each drop in frontmatter and moves it to inbox/; /triage-inbox routes them from there. scripts/audiopen-ingest.sh — 80-line bash normaliser. Idempotent; safe to re-run. Extracts title, slugifies, computes capture timestamp from mtime, rewrites with fleeting-note frontmatter. scripts/audiopen-webhook/main.py — stdlib-only Python HTTP server. Zero pip deps; binds to 127.0.0.1 by default. Accepts flexible payload shapes (title/name + body/output/summary/polished/ orig_transcript) so AudioPen version drift is logged rather than silently dropped. scripts/audiopen-webhook/*.example — systemd user-unit templates for the receiver and the path/service pair that fires the ingest wrapper on inbox/raw/ changes. scripts/pyproject.toml + README.md — Python venv convention. Zero deps today; venv location reserved at scripts/.venv/ (gitignored), manifest at scripts/pyproject.toml, bootstrap documented for both plain pip and uv. Optional-dependencies groups let individual tools pull what they need without bloating the whole env. docs/setup/audiopen.md — full setup guide: generating the shared secret, installing the systemd units, pairing with Tailscale Funnel, configuring AudioPen's webhook, and the IMAP-fallback path for setups that can't run a public-reachable receiver. docs/setup/sync.md — companion guide: Gitea SSH remote (works even with Authentik gating HTTPS), obsidian-git plugin configuration, Syncthing desktop↔phone pairing with the critical .stignore patterns, and the three phone-role tiers so the user can pick Tier 1 / 2 / 3 at their own pace. .gitignore gains **/.venv/, **/venv/, **/__pycache__/, and *.pyc so nobody accidentally commits a materialised environment. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.7
@@ -14,3 +14,9 @@ Thumbs.db
|
||||
.obsidian/workspace.json
|
||||
.obsidian/workspace-mobile.json
|
||||
.obsidian/cache
|
||||
|
||||
# Python — virtualenvs (location convention: scripts/.venv/) and bytecode
|
||||
**/.venv/
|
||||
**/venv/
|
||||
**/__pycache__/
|
||||
*.pyc
|
||||
|
||||
@@ -0,0 +1,210 @@
|
||||
# AudioPen → PKB setup
|
||||
|
||||
Voice-note capture pipeline. AudioPen records on phone, polishes on their cloud, POSTs
|
||||
the polished prose to your own infra. This vault's `inbox/raw/` receives the drop;
|
||||
`scripts/audiopen-ingest.sh` normalises frontmatter and moves it to `inbox/`;
|
||||
`/triage-inbox` routes it into `daily/`, `notes/`, `projects/`, or `reading/`.
|
||||
|
||||
Two ingest paths are supported. Pick one. Either works with the same
|
||||
`/triage-inbox` downstream.
|
||||
|
||||
---
|
||||
|
||||
## Path A — Webhook via Tailscale Funnel (recommended)
|
||||
|
||||
**Requires.** Tailscale installed on the desktop with Funnel enabled on your tailnet.
|
||||
Tailscale Funnel provides an outbound-initiated HTTPS endpoint (`*.ts.net`) with no
|
||||
router configuration and no exposed ports. Nothing sits behind Authentik; AudioPen
|
||||
hits the Funnel URL directly.
|
||||
|
||||
### 1. Generate a shared secret
|
||||
|
||||
```sh
|
||||
head -c 24 /dev/urandom | base64 | tr -d '/+=' > /tmp/secret
|
||||
cat /tmp/secret
|
||||
```
|
||||
|
||||
Store it privately; it becomes part of the webhook URL.
|
||||
|
||||
### 2. Install the receiver systemd unit
|
||||
|
||||
```sh
|
||||
mkdir -p ~/.config/audiopen-webhook ~/.config/systemd/user
|
||||
printf 'AUDIOPEN_WEBHOOK_SECRET=%s\n' "$(cat /tmp/secret)" > ~/.config/audiopen-webhook/env
|
||||
chmod 600 ~/.config/audiopen-webhook/env
|
||||
|
||||
cp /var/mnt/data/projects/council/scripts/audiopen-webhook/audiopen-webhook.service.example \
|
||||
~/.config/systemd/user/audiopen-webhook.service
|
||||
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now audiopen-webhook.service
|
||||
systemctl --user status audiopen-webhook.service # should show active (running)
|
||||
```
|
||||
|
||||
Tail logs:
|
||||
|
||||
```sh
|
||||
journalctl --user -u audiopen-webhook.service -f
|
||||
```
|
||||
|
||||
### 3. Install the ingest path + service units
|
||||
|
||||
```sh
|
||||
# audiopen-ingest.service — normalises raw drops
|
||||
cat > ~/.config/systemd/user/audiopen-ingest.service <<'EOF'
|
||||
[Unit]
|
||||
Description=Normalise AudioPen drops into council/inbox/
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/var/mnt/data/projects/council/scripts/audiopen-ingest.sh
|
||||
Environment=COUNCIL_REPO=/var/mnt/data/projects/council
|
||||
EOF
|
||||
|
||||
# audiopen-ingest.path — fires the service whenever inbox/raw/ gains a file
|
||||
cat > ~/.config/systemd/user/audiopen-ingest.path <<'EOF'
|
||||
[Unit]
|
||||
Description=Watch council/inbox/raw for new AudioPen drops
|
||||
|
||||
[Path]
|
||||
PathChanged=/var/mnt/data/projects/council/inbox/raw
|
||||
Unit=audiopen-ingest.service
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
EOF
|
||||
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now audiopen-ingest.path
|
||||
```
|
||||
|
||||
### 4. Expose the receiver via Tailscale Funnel
|
||||
|
||||
```sh
|
||||
tailscale funnel --bg 8765
|
||||
tailscale funnel status # show the public URL
|
||||
```
|
||||
|
||||
Your endpoint is now `https://<your-host>.<tailnet>.ts.net:443/`. AudioPen will
|
||||
POST to `https://<your-host>.<tailnet>.ts.net/audiopen/<secret>`.
|
||||
|
||||
### 5. Configure AudioPen
|
||||
|
||||
In AudioPen (web or mobile app): settings → integrations → add a generic webhook.
|
||||
|
||||
- **URL:** `https://<your-host>.<tailnet>.ts.net/audiopen/<secret>`
|
||||
- **Method:** POST
|
||||
- **Content-Type:** application/json
|
||||
- **Trigger:** after each recording is polished
|
||||
|
||||
AudioPen typically sends JSON with keys like `title`, `body`, `orig_transcript`.
|
||||
The receiver accepts any of these; if AudioPen changes shape, check
|
||||
`journalctl --user -u audiopen-webhook.service` for the logged payload and
|
||||
update `scripts/audiopen-webhook/main.py::extract_content()` if needed.
|
||||
|
||||
### 6. Test end-to-end
|
||||
|
||||
```sh
|
||||
SECRET=$(awk -F= '/AUDIOPEN_WEBHOOK_SECRET/ {print $2}' ~/.config/audiopen-webhook/env)
|
||||
FUNNEL=$(tailscale funnel status | awk '/https:/ {print $1; exit}')
|
||||
|
||||
curl -X POST -H 'Content-Type: application/json' \
|
||||
-d '{"title":"Test","body":"Hello from curl. This is a test drop."}' \
|
||||
"${FUNNEL%/}/audiopen/${SECRET}"
|
||||
# → ok
|
||||
|
||||
ls /var/mnt/data/projects/council/inbox/raw # should contain the test file briefly
|
||||
sleep 2
|
||||
ls /var/mnt/data/projects/council/inbox # should now contain the normalised file
|
||||
```
|
||||
|
||||
Then record an actual voice note in AudioPen, wait for polishing, confirm the same
|
||||
flow. Run `/triage-inbox` in Claude Code when you're ready to route the contents.
|
||||
|
||||
---
|
||||
|
||||
## Path B — Email → IMAP fetch (no-open-ports fallback)
|
||||
|
||||
Use this if you don't run Tailscale / Cloudflare Tunnel and don't want to stand one
|
||||
up. Fully outbound: AudioPen emails each note; a cron on the desktop pulls via IMAP
|
||||
and drops files in `inbox/raw/`.
|
||||
|
||||
### 1. Dedicated email address
|
||||
|
||||
Create or repurpose a mailbox for AudioPen's output. Options:
|
||||
|
||||
- Fastmail alias → Fastmail IMAP on desktop (works).
|
||||
- Self-hosted mail server → pull via IMAP from localhost (works).
|
||||
- Gmail "+audiopen" alias → fine but adds Google to the dependency chain.
|
||||
|
||||
### 2. IMAP fetch script (stub — you'll customise for your mail server)
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""Minimal IMAP fetcher: pulls unread messages from a mailbox, writes them as
|
||||
bare markdown to inbox/raw/. Uses stdlib imaplib. Keep credentials in
|
||||
~/.config/audiopen-mail/env or similar."""
|
||||
import imaplib, email, os, re, pathlib, datetime
|
||||
|
||||
HOST = os.environ["IMAP_HOST"]
|
||||
USER = os.environ["IMAP_USER"]
|
||||
PASS = os.environ["IMAP_PASS"]
|
||||
MBOX = os.environ.get("IMAP_MAILBOX", "INBOX")
|
||||
INBOX = pathlib.Path(os.environ.get("COUNCIL_INBOX_RAW",
|
||||
"/var/mnt/data/projects/council/inbox/raw"))
|
||||
INBOX.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
m = imaplib.IMAP4_SSL(HOST)
|
||||
m.login(USER, PASS)
|
||||
m.select(MBOX)
|
||||
typ, data = m.search(None, 'UNSEEN')
|
||||
for num in data[0].split():
|
||||
typ, msg_data = m.fetch(num, '(RFC822)')
|
||||
msg = email.message_from_bytes(msg_data[0][1])
|
||||
title = (msg.get("Subject") or "Untitled").strip()
|
||||
body = ""
|
||||
if msg.is_multipart():
|
||||
for p in msg.walk():
|
||||
if p.get_content_type() == "text/plain":
|
||||
body = p.get_payload(decode=True).decode('utf-8', errors='replace')
|
||||
break
|
||||
else:
|
||||
body = msg.get_payload(decode=True).decode('utf-8', errors='replace')
|
||||
slug = re.sub(r"[^a-z0-9]+", "-", title.lower()).strip("-")[:60] or "untitled"
|
||||
ts = datetime.datetime.now().strftime("%Y-%m-%d-%H%M")
|
||||
(INBOX / f"{ts}_{slug}.md").write_text(f"# {title}\n\n{body}\n")
|
||||
m.store(num, '+FLAGS', '\\Seen')
|
||||
m.logout()
|
||||
```
|
||||
|
||||
Save as `scripts/audiopen-imap-fetch.py`. Run it every N minutes via a systemd
|
||||
user-timer or cron. The same `audiopen-ingest.path` unit (from Path A, step 3)
|
||||
picks up the drops from there.
|
||||
|
||||
### 3. Configure AudioPen
|
||||
|
||||
Settings → integrations → email output → recipient = your dedicated address.
|
||||
Each recording → one email. Done.
|
||||
|
||||
---
|
||||
|
||||
## Routine
|
||||
|
||||
Once everything is wired:
|
||||
|
||||
1. **Capture**: voice note → AudioPen → (webhook or email) → desktop.
|
||||
2. **Ingest**: automatic — you don't run anything; systemd fires on file change.
|
||||
3. **Route**: when you want, open a Claude Code session in the vault and run
|
||||
`/triage-inbox`. Koskela proposes destinations; you approve in batch.
|
||||
4. **Weekly/monthly**: the promoted notes are already in their homes; `/reflect`,
|
||||
`/review-week`, and `/review-month` will see them naturally.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Check |
|
||||
|---|---|
|
||||
| Files pile up in `inbox/raw/` but never normalise | `systemctl --user status audiopen-ingest.path audiopen-ingest.service` — path unit enabled? service errored? Run the script manually once: `bash scripts/audiopen-ingest.sh`. |
|
||||
| Webhook POSTs 200 but no file lands | Check the systemd logs for the receiver: `journalctl --user -u audiopen-webhook.service -n 50`. The logged path should be writable and under `inbox/raw/`. |
|
||||
| Webhook POSTs 404 | Secret mismatch between AudioPen URL and `~/.config/audiopen-webhook/env`. Regenerate and re-set both sides. |
|
||||
| Funnel URL unreachable | `tailscale status`; `tailscale funnel status`. Re-run `tailscale funnel --bg 8765`. Confirm Funnel is enabled on your tailnet (admin → DNS → HTTPS Certificates; admin → Access controls → Funnel). |
|
||||
| Normalised file missing frontmatter | Bug in the ingest wrapper; open the raw file, run the script with `bash -x scripts/audiopen-ingest.sh` to trace. |
|
||||
@@ -0,0 +1,174 @@
|
||||
# Sync architecture — setup guide
|
||||
|
||||
Three independent sync channels. Setting each up on a fresh machine.
|
||||
|
||||
## 1. Gitea over SSH — version history + off-site backup
|
||||
|
||||
**Already configured** on the desktop for this repo:
|
||||
|
||||
```sh
|
||||
$ git remote -v
|
||||
origin ssh://git@git.schweitz.net:2222/jpmschweitzer/council.git (fetch)
|
||||
origin ssh://git@git.schweitz.net:2222/jpmschweitzer/council.git (push)
|
||||
```
|
||||
|
||||
On a fresh machine:
|
||||
|
||||
```sh
|
||||
git clone ssh://git@git.schweitz.net:2222/jpmschweitzer/council.git
|
||||
cd council
|
||||
```
|
||||
|
||||
Requires an SSH key registered in Gitea (Gitea settings → SSH keys → add).
|
||||
Gitea is behind Authentik for HTTPS, so HTTPS clones won't work without the browser
|
||||
auth dance. SSH bypasses that entirely — the ssh daemon on Gitea handles key auth
|
||||
independently.
|
||||
|
||||
### obsidian-git plugin (desktop)
|
||||
|
||||
The `obsidian-git` plugin is already installed in `.obsidian/plugins/`. Configure it
|
||||
to auto-commit and push on your chosen cadence:
|
||||
|
||||
1. Obsidian settings → Community plugins → Obsidian Git.
|
||||
2. "Remote URL" — leave blank; it uses the repo's existing `origin`.
|
||||
3. "Auto commit-and-sync interval" — e.g. 10 minutes. Set to 0 to disable auto-push.
|
||||
4. "Commit message on auto commit-and-sync" — use the default template.
|
||||
5. Confirm it can push: Command Palette → "Obsidian Git: Push".
|
||||
|
||||
### What NOT to commit
|
||||
|
||||
Already set in `.gitignore`:
|
||||
- `.env`, `.env.local`, `.env.*.local`
|
||||
- `.obsidian/workspace.json`, `.obsidian/workspace-mobile.json`, `.obsidian/cache`
|
||||
- `.claude/settings.local.json`
|
||||
- `**/.venv/`, `**/venv/`, `**/__pycache__/`, `*.pyc`
|
||||
|
||||
---
|
||||
|
||||
## 2. Syncthing — desktop ↔ phone peer-to-peer sync
|
||||
|
||||
Optional. Skip if you're picking **Tier 1** from the phone-role decision (AudioPen
|
||||
only, no mobile vault).
|
||||
|
||||
### Desktop setup
|
||||
|
||||
Syncthing is available on Bazzite via rpm-ostree (layered) or Flatpak. Flatpak is
|
||||
simpler:
|
||||
|
||||
```sh
|
||||
flatpak install flathub me.kozec.syncthingtk
|
||||
# or
|
||||
rpm-ostree install syncthing
|
||||
```
|
||||
|
||||
Start it:
|
||||
|
||||
```sh
|
||||
systemctl --user enable --now syncthing.service
|
||||
# Web UI:
|
||||
xdg-open http://127.0.0.1:8384
|
||||
```
|
||||
|
||||
In the Syncthing web UI:
|
||||
1. **Add folder**: path = `/var/mnt/data/projects/council/`, folder ID = `council`.
|
||||
2. **Ignore patterns** (crucial — click "Edit" → "Ignore Patterns" on the folder):
|
||||
|
||||
```
|
||||
.git
|
||||
.obsidian/workspace.json
|
||||
.obsidian/workspace-mobile.json
|
||||
.obsidian/cache
|
||||
.trash
|
||||
**.sync-conflict-*
|
||||
scripts/.venv
|
||||
```
|
||||
|
||||
This list matters. Syncthing must never propagate `.git/` — partial packs mid-commit
|
||||
would corrupt the mirror.
|
||||
|
||||
### Phone setup (Android)
|
||||
|
||||
1. Install **Syncthing-Fork** from F-Droid (more robust than upstream Syncthing on
|
||||
Android).
|
||||
2. Pair with desktop: desktop shows its device ID in the web UI → enter on phone
|
||||
→ desktop accepts the pairing.
|
||||
3. On the phone, accept the `council` folder share. Set its target directory to
|
||||
e.g. `/sdcard/Documents/council/`.
|
||||
4. Install **Obsidian Mobile**. Open as vault → pick the Syncthing-watched path.
|
||||
|
||||
### Conflict discipline
|
||||
|
||||
Both devices edit the same file while disconnected → Syncthing creates
|
||||
`<filename>.sync-conflict-YYYYMMDD-HHMMSS-deviceID.md`. Rare if you edit one device
|
||||
at a time. To resolve: read both, copy the good bits into the canonical file,
|
||||
delete the conflict.
|
||||
|
||||
### How the Syncthing/git channels interact
|
||||
|
||||
- **Git lives on desktop only.** The phone never sees `.git/` (it's in `.stignore`).
|
||||
- **Syncthing replicates content only.** The phone has the markdown; it doesn't have
|
||||
history.
|
||||
- **To push from desktop**: just `git push` as usual. obsidian-git automates it.
|
||||
- **To pull new content on desktop from phone**: Syncthing already did it; there's
|
||||
nothing to pull. Review the changes in `git status` and commit them like any
|
||||
other working-tree edit.
|
||||
|
||||
---
|
||||
|
||||
## 3. AudioPen webhook via Tailscale Funnel — voice-note ingestion
|
||||
|
||||
See [`docs/setup/audiopen.md`](./audiopen.md). Independent of the two above.
|
||||
|
||||
---
|
||||
|
||||
## Phone-role decision
|
||||
|
||||
When you set up Obsidian Mobile (if at all), pick a tier:
|
||||
|
||||
| Tier | Phone runs | Trade-off |
|
||||
|---|---|---|
|
||||
| 1 | AudioPen only | Simplest. No vault on phone. Read/edit on desktop. |
|
||||
| 2 | AudioPen + browser → Gitea markdown view | Occasional phone read via Authentik-gated Gitea. |
|
||||
| 3 | AudioPen + Syncthing-Fork + Obsidian Mobile | Full vault on phone. Maximum capability, moderate battery + ~few hundred MB storage for the vault. |
|
||||
|
||||
All three are compatible with the rest of the stack. The AudioPen pipeline
|
||||
(path A or B) is identical across tiers.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
After everything is set up:
|
||||
|
||||
```sh
|
||||
# Git remote reachable
|
||||
git ls-remote origin | head -3 # should list refs from Gitea
|
||||
|
||||
# Syncthing running (if Tier 2/3)
|
||||
systemctl --user is-active syncthing.service
|
||||
# → active
|
||||
|
||||
# Tailscale Funnel for AudioPen webhook
|
||||
tailscale funnel status
|
||||
# should list the https endpoint for :8765
|
||||
|
||||
# AudioPen receiver running
|
||||
systemctl --user is-active audiopen-webhook.service
|
||||
# → active
|
||||
|
||||
# Push a test commit
|
||||
cd /var/mnt/data/projects/council
|
||||
echo "# test" > /tmp/sync-test.md
|
||||
mv /tmp/sync-test.md daily/2099-01-01.md
|
||||
git add daily/2099-01-01.md
|
||||
git commit -m "test sync pipeline" -m "Co-Authored-By: me"
|
||||
git push
|
||||
# should succeed
|
||||
|
||||
# On phone (Tier 3): wait a moment, open Obsidian Mobile, verify daily/2099-01-01.md appears.
|
||||
|
||||
# Clean up
|
||||
git rm daily/2099-01-01.md
|
||||
git commit -m "drop sync test"
|
||||
git push
|
||||
```
|
||||
@@ -0,0 +1,62 @@
|
||||
# scripts/
|
||||
|
||||
Operational scripts for the PKB. Everything here is side-effectful — writing files,
|
||||
running webhooks, ingesting voice notes. Keep it small; anything analytical belongs
|
||||
in `.claude/commands/` or in the `mql` CLI.
|
||||
|
||||
## Current contents
|
||||
|
||||
| Path | Language | Purpose |
|
||||
|---|---|---|
|
||||
| `audiopen-ingest.sh` | bash | Normalises raw AudioPen drops (`inbox/raw/*.md`) into properly-frontmattered fleeting notes in `inbox/`. Idempotent. |
|
||||
| `audiopen-webhook/main.py` | Python (stdlib) | Tiny HTTP server that receives AudioPen webhook POSTs and writes to `inbox/raw/`. Designed to sit behind Tailscale Funnel. |
|
||||
| `audiopen-webhook/*.example` | systemd | User-unit examples for the receiver + the path/service that runs `audiopen-ingest.sh` on file change. |
|
||||
|
||||
Setup guide for the full pipeline: [`docs/setup/audiopen.md`](../docs/setup/audiopen.md).
|
||||
|
||||
## Python venv convention
|
||||
|
||||
Python code in this repo is **stdlib-only as of now**. No pip-installable dependencies.
|
||||
If/when a real dependency arrives (e.g. `watchdog`, `imap-tools`, anything heavier
|
||||
than stdlib), the convention is:
|
||||
|
||||
- Venv lives at **`scripts/.venv/`** (gitignored — every user materialises locally).
|
||||
- Dependencies tracked in **`scripts/pyproject.toml`** (committed).
|
||||
- Bootstrap on a fresh machine:
|
||||
|
||||
```sh
|
||||
cd scripts
|
||||
python3 -m venv .venv
|
||||
.venv/bin/pip install -e .
|
||||
```
|
||||
|
||||
Or with `uv` (recommended if available):
|
||||
|
||||
```sh
|
||||
cd scripts
|
||||
uv venv
|
||||
uv pip install -e .
|
||||
```
|
||||
|
||||
- Scripts that need the venv should either:
|
||||
- Invoke `scripts/.venv/bin/python3` explicitly in their shebang or systemd `ExecStart`.
|
||||
- Or document activation (`source scripts/.venv/bin/activate`) before running.
|
||||
|
||||
Keeping the venv under `scripts/` rather than the repo root keeps the Obsidian file
|
||||
tree uncluttered — `scripts/` is already excluded from Obsidian's index
|
||||
(`userIgnoreFilters` in `.obsidian/app.json`) so nothing Python-related will ever
|
||||
show in the graph or search.
|
||||
|
||||
**Don't create the venv preemptively.** When the first dependency lands, that's when
|
||||
you create it. Keeps the repo lighter and avoids stale venv directories.
|
||||
|
||||
The `pyproject.toml` uses `[project.optional-dependencies]` to split deps by tool
|
||||
— e.g. `pip install -e .[imap]` installs only the IMAP fetcher's deps. This lets
|
||||
the webhook receiver stay dep-free while heavier tools can pull in what they need.
|
||||
|
||||
## Shell scripts
|
||||
|
||||
Kept in plain bash. Portable to any Linux running GNU coreutils (which Bazzite does).
|
||||
No zsh-isms, no bashisms beyond `set -euo pipefail` and arrays. If a script grows
|
||||
past ~200 lines or needs real argument parsing, rewrite it in Python (and you're now
|
||||
back in the venv convention above).
|
||||
Executable
+78
@@ -0,0 +1,78 @@
|
||||
#!/usr/bin/env bash
|
||||
# audiopen-ingest.sh
|
||||
#
|
||||
# Normalises raw AudioPen drops in inbox/raw/ into proper fleeting notes in inbox/.
|
||||
# Adds frontmatter, renames to timestamp-slug format, moves to inbox/.
|
||||
# Idempotent: safe to re-run. Destructive on inbox/raw/ (files are moved, not copied).
|
||||
#
|
||||
# Environment:
|
||||
# COUNCIL_REPO — vault root. Default: /var/mnt/data/projects/council
|
||||
#
|
||||
# Exit status:
|
||||
# 0 = success (even if zero files were processed)
|
||||
# 1 = unexpected error
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
REPO="${COUNCIL_REPO:-/var/mnt/data/projects/council}"
|
||||
RAW="$REPO/inbox/raw"
|
||||
OUT="$REPO/inbox"
|
||||
|
||||
if [ ! -d "$RAW" ]; then
|
||||
echo "audiopen-ingest: raw dir $RAW does not exist; nothing to do." >&2
|
||||
exit 0
|
||||
fi
|
||||
mkdir -p "$OUT"
|
||||
|
||||
shopt -s nullglob
|
||||
count=0
|
||||
for f in "$RAW"/*.md; do
|
||||
# mtime as ISO-8601 (date -r works on GNU coreutils; Bazzite ships it)
|
||||
captured=$(date -r "$f" -Iseconds)
|
||||
ts=$(date -r "$f" +%Y-%m-%d-%H%M)
|
||||
|
||||
# Title: first `# ` line, else filename (without extension)
|
||||
title=$(grep -m1 '^# ' "$f" 2>/dev/null | sed 's/^# *//' || true)
|
||||
if [ -z "$title" ]; then
|
||||
title=$(basename "$f" .md)
|
||||
fi
|
||||
|
||||
# Slugify: lowercase, non-alnum → dash, collapse dashes, trim, cap at 60
|
||||
slug=$(printf '%s' "$title" | tr '[:upper:]' '[:lower:]' \
|
||||
| sed 's/[^a-z0-9]/-/g' \
|
||||
| sed 's/--*/-/g' \
|
||||
| sed 's/^-*//;s/-*$//' \
|
||||
| cut -c1-60)
|
||||
[ -z "$slug" ] && slug="untitled"
|
||||
|
||||
out="$OUT/${ts}_${slug}.md"
|
||||
|
||||
# If target exists (same timestamp + slug), append a numeric suffix
|
||||
i=2
|
||||
while [ -e "$out" ]; do
|
||||
out="$OUT/${ts}_${slug}_${i}.md"
|
||||
i=$((i+1))
|
||||
done
|
||||
|
||||
# Body: strip the first `# title` line so we regenerate it in the normalised form
|
||||
content=$(awk 'BEGIN{stripped=0} /^# / && !stripped { stripped=1; next } { print }' "$f")
|
||||
|
||||
{
|
||||
echo "---"
|
||||
echo "type: fleeting"
|
||||
echo "source: audiopen"
|
||||
echo "captured: $captured"
|
||||
echo "slug: $slug"
|
||||
echo "---"
|
||||
echo
|
||||
echo "# $title"
|
||||
echo
|
||||
printf '%s\n' "$content"
|
||||
} > "$out"
|
||||
|
||||
rm "$f"
|
||||
echo "ingested: $(basename "$f") → $(basename "$out")"
|
||||
count=$((count+1))
|
||||
done
|
||||
|
||||
echo "audiopen-ingest: $count file(s) normalised."
|
||||
@@ -0,0 +1,69 @@
|
||||
# audiopen-webhook
|
||||
|
||||
Zero-dependency Python HTTP server that receives AudioPen webhook POSTs and
|
||||
writes bare markdown files to `inbox/raw/`. The separate `scripts/audiopen-ingest.sh`
|
||||
wrapper normalises those files into proper fleeting notes under `inbox/`.
|
||||
|
||||
## Why this shape
|
||||
|
||||
- **Receiver is minimal (stdlib-only http.server).** No Flask, no runtime to keep
|
||||
updated, ~150 LOC. Reads AudioPen's JSON payload, writes one file, logs to
|
||||
stderr. Anything fancier belongs in the ingest wrapper.
|
||||
- **Two-step raw → normalised.** Keeps the receiver stupid; all schema decisions
|
||||
(frontmatter, slug format, filename) live in one shell script you can read
|
||||
in 30 seconds.
|
||||
- **Designed for Tailscale Funnel.** Binds to 127.0.0.1 by default. Tailscale
|
||||
Funnel (or Cloudflare Tunnel, or an SSH remote forward, or ngrok) terminates
|
||||
the public endpoint and proxies to 127.0.0.1:8765. No ports opened on your
|
||||
router; nothing behind Authentik.
|
||||
|
||||
## Setup
|
||||
|
||||
Full step-by-step lives in [`docs/setup/audiopen.md`](../../docs/setup/audiopen.md).
|
||||
Quick version:
|
||||
|
||||
1. **Generate a secret**: `head -c 24 /dev/urandom | base64 | tr -d '/+='`.
|
||||
2. **Configure AudioPen** to POST JSON to `https://<your-funnel>.ts.net/audiopen/<secret>`.
|
||||
3. **Install the systemd user units**:
|
||||
```sh
|
||||
mkdir -p ~/.config/audiopen-webhook ~/.config/systemd/user
|
||||
echo "AUDIOPEN_WEBHOOK_SECRET=<secret>" > ~/.config/audiopen-webhook/env
|
||||
chmod 600 ~/.config/audiopen-webhook/env
|
||||
cp audiopen-webhook.service.example ~/.config/systemd/user/audiopen-webhook.service
|
||||
# create two more systemd files for the ingest path/service per
|
||||
# audiopen-ingest.path.example
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now audiopen-webhook.service audiopen-ingest.path
|
||||
```
|
||||
4. **Expose via Tailscale Funnel**:
|
||||
```sh
|
||||
tailscale funnel --bg 8765
|
||||
```
|
||||
5. **Test**:
|
||||
```sh
|
||||
curl -X POST -H 'Content-Type: application/json' \
|
||||
-d '{"title":"Test","body":"Hello from curl"}' \
|
||||
https://<your-funnel>.ts.net/audiopen/<secret>
|
||||
# → OK
|
||||
ls ~/path/to/vault/inbox/raw # should see the test file
|
||||
# wait a beat for the inotify path unit to fire
|
||||
ls ~/path/to/vault/inbox # should see the normalised file
|
||||
```
|
||||
|
||||
## Payload shape
|
||||
|
||||
The receiver accepts any of these JSON keys for the title: `title`, `name`.
|
||||
For the body: `body`, `output`, `summary`, `polished`, `orig_transcript`. Unknown
|
||||
keys are ignored; the raw payload is logged to stderr so you can see what
|
||||
AudioPen actually sent if shape drifts.
|
||||
|
||||
If AudioPen's webhook format changes in a way the current extractor misses,
|
||||
check `journalctl --user -u audiopen-webhook.service` for the logged shape and
|
||||
update the key list in `extract_content()` at the top of `main.py`.
|
||||
|
||||
## Alternative: email + IMAP fetch
|
||||
|
||||
If running a public-reachable receiver isn't viable, the plan has an email-path
|
||||
fallback documented in `docs/setup/audiopen.md`. It's entirely outbound: AudioPen
|
||||
emails each note to a dedicated address, and a cron on the desktop pulls via
|
||||
IMAP and drops files in `inbox/raw/`. Same ingest wrapper handles the rest.
|
||||
@@ -0,0 +1,34 @@
|
||||
# audiopen-ingest path + service units — example
|
||||
#
|
||||
# These two units let the ingest wrapper fire automatically whenever
|
||||
# inbox/raw/ gains a file. Install:
|
||||
#
|
||||
# cp audiopen-ingest.path.example ~/.config/systemd/user/audiopen-ingest.path
|
||||
# cp audiopen-ingest.service.example ~/.config/systemd/user/audiopen-ingest.service
|
||||
# systemctl --user daemon-reload
|
||||
# systemctl --user enable --now audiopen-ingest.path
|
||||
#
|
||||
# (Adjust %h paths or vault location as needed.)
|
||||
|
||||
# ---- audiopen-ingest.path ----
|
||||
[Unit]
|
||||
Description=Watch council/inbox/raw for new AudioPen drops
|
||||
|
||||
[Path]
|
||||
PathChanged=/var/mnt/data/projects/council/inbox/raw
|
||||
Unit=audiopen-ingest.service
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
|
||||
# ---- audiopen-ingest.service ----
|
||||
# (Put this in a separate file — audiopen-ingest.service — systemd does not
|
||||
# allow two units in one file.)
|
||||
#
|
||||
# [Unit]
|
||||
# Description=Normalise AudioPen drops into council/inbox/
|
||||
#
|
||||
# [Service]
|
||||
# Type=oneshot
|
||||
# ExecStart=/var/mnt/data/projects/council/scripts/audiopen-ingest.sh
|
||||
# Environment=COUNCIL_REPO=/var/mnt/data/projects/council
|
||||
@@ -0,0 +1,36 @@
|
||||
# audiopen-webhook user systemd unit — example
|
||||
#
|
||||
# Install to ~/.config/systemd/user/audiopen-webhook.service, then:
|
||||
# systemctl --user daemon-reload
|
||||
# systemctl --user enable --now audiopen-webhook.service
|
||||
#
|
||||
# Check logs:
|
||||
# journalctl --user -u audiopen-webhook.service -f
|
||||
#
|
||||
# The shared secret lives in ~/.config/audiopen-webhook/env (chmod 600).
|
||||
# Example env file contents:
|
||||
# AUDIOPEN_WEBHOOK_SECRET=<random-32-char-string>
|
||||
# # Optional overrides:
|
||||
# # AUDIOPEN_WEBHOOK_HOST=127.0.0.1
|
||||
# # AUDIOPEN_WEBHOOK_PORT=8765
|
||||
# # COUNCIL_INBOX_RAW=/var/mnt/data/projects/council/inbox/raw
|
||||
|
||||
[Unit]
|
||||
Description=AudioPen webhook receiver (writes to council/inbox/raw)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
EnvironmentFile=%h/.config/audiopen-webhook/env
|
||||
ExecStart=/usr/bin/python3 %h/projects/council/scripts/audiopen-webhook/main.py
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
# Narrow permissions:
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=read-only
|
||||
ReadWritePaths=/var/mnt/data/projects/council/inbox/raw
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
Executable
+139
@@ -0,0 +1,139 @@
|
||||
#!/usr/bin/env python3
|
||||
"""AudioPen webhook receiver.
|
||||
|
||||
Listens for POST requests at /audiopen/<shared-secret> and writes the polished-prose
|
||||
payload to COUNCIL_INBOX_RAW as a bare markdown file. A separate cron/systemd job
|
||||
(or inotify) then runs audiopen-ingest.sh to normalise into the PKB inbox.
|
||||
|
||||
Zero external dependencies (stdlib only). Designed to sit behind Tailscale Funnel
|
||||
or similar outbound-initiated tunnel — binds to 127.0.0.1 by default.
|
||||
|
||||
Environment:
|
||||
AUDIOPEN_WEBHOOK_SECRET required. Shared secret in the URL path.
|
||||
AUDIOPEN_WEBHOOK_HOST default 127.0.0.1
|
||||
AUDIOPEN_WEBHOOK_PORT default 8765
|
||||
COUNCIL_INBOX_RAW default /var/mnt/data/projects/council/inbox/raw
|
||||
|
||||
AudioPen's webhook payload shape varies by version; we accept any of these JSON keys
|
||||
for the body: `body`, `output`, `summary`, `polished`. For the title: `title`,
|
||||
`name`. Unknown fields are ignored. The raw payload is kept in an HTTP access log
|
||||
on stderr so you can inspect it if AudioPen changes shape.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
import datetime as _dt
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
|
||||
|
||||
INBOX_RAW = pathlib.Path(
|
||||
os.environ.get("COUNCIL_INBOX_RAW", "/var/mnt/data/projects/council/inbox/raw")
|
||||
)
|
||||
SECRET = os.environ.get("AUDIOPEN_WEBHOOK_SECRET")
|
||||
HOST = os.environ.get("AUDIOPEN_WEBHOOK_HOST", "127.0.0.1")
|
||||
PORT = int(os.environ.get("AUDIOPEN_WEBHOOK_PORT", "8765"))
|
||||
|
||||
_SLUG_RE = re.compile(r"[^a-z0-9]+")
|
||||
|
||||
|
||||
def slugify(text: str, max_len: int = 60) -> str:
|
||||
s = _SLUG_RE.sub("-", text.strip().lower()).strip("-")
|
||||
return (s or "untitled")[:max_len]
|
||||
|
||||
|
||||
def extract_content(payload: dict) -> tuple[str, str]:
|
||||
title = str(
|
||||
payload.get("title")
|
||||
or payload.get("name")
|
||||
or ""
|
||||
).strip()
|
||||
body = str(
|
||||
payload.get("body")
|
||||
or payload.get("output")
|
||||
or payload.get("summary")
|
||||
or payload.get("polished")
|
||||
or payload.get("orig_transcript")
|
||||
or ""
|
||||
).strip()
|
||||
if not title and body:
|
||||
title = body.splitlines()[0][:60]
|
||||
return title or "Untitled", body
|
||||
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
server_version = "audiopen-webhook/1"
|
||||
|
||||
def _reply(self, status: int, msg: str = "") -> None:
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type", "text/plain; charset=utf-8")
|
||||
self.send_header("Content-Length", str(len(msg)))
|
||||
self.end_headers()
|
||||
if msg:
|
||||
self.wfile.write(msg.encode())
|
||||
|
||||
def do_POST(self) -> None:
|
||||
parts = [p for p in self.path.split("/") if p]
|
||||
if len(parts) < 2 or parts[0] != "audiopen" or parts[1] != SECRET:
|
||||
self._reply(404, "not found")
|
||||
return
|
||||
|
||||
length = int(self.headers.get("Content-Length", "0") or 0)
|
||||
raw = self.rfile.read(length) if length > 0 else b""
|
||||
try:
|
||||
payload = json.loads(raw) if raw else {}
|
||||
except json.JSONDecodeError:
|
||||
self._reply(400, "invalid JSON")
|
||||
return
|
||||
if not isinstance(payload, dict):
|
||||
self._reply(400, "expected JSON object")
|
||||
return
|
||||
|
||||
title, body = extract_content(payload)
|
||||
if not body:
|
||||
self._reply(400, "empty body")
|
||||
return
|
||||
|
||||
slug = slugify(title)
|
||||
ts = _dt.datetime.now().strftime("%Y-%m-%d-%H%M")
|
||||
filename = f"{ts}_{slug}.md"
|
||||
|
||||
INBOX_RAW.mkdir(parents=True, exist_ok=True)
|
||||
target = INBOX_RAW / filename
|
||||
# Avoid overwriting on same-minute collision
|
||||
suffix = 2
|
||||
while target.exists():
|
||||
target = INBOX_RAW / f"{ts}_{slug}_{suffix}.md"
|
||||
suffix += 1
|
||||
|
||||
target.write_text(f"# {title}\n\n{body}\n", encoding="utf-8")
|
||||
sys.stderr.write(f"[audiopen-webhook] wrote {target}\n")
|
||||
self._reply(200, "ok\n")
|
||||
|
||||
def do_GET(self) -> None:
|
||||
# Health check at /health; everything else returns 404.
|
||||
if self.path == "/health":
|
||||
self._reply(200, "ok\n")
|
||||
else:
|
||||
self._reply(404, "not found")
|
||||
|
||||
def log_message(self, fmt: str, *args) -> None:
|
||||
sys.stderr.write(f"[audiopen-webhook] {self.address_string()} {fmt % args}\n")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if not SECRET:
|
||||
sys.stderr.write("AUDIOPEN_WEBHOOK_SECRET env var is required.\n")
|
||||
return 1
|
||||
sys.stderr.write(
|
||||
f"[audiopen-webhook] listening on http://{HOST}:{PORT}, inbox_raw={INBOX_RAW}\n"
|
||||
)
|
||||
HTTPServer((HOST, PORT), Handler).serve_forever()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,42 @@
|
||||
[project]
|
||||
name = "council-scripts"
|
||||
version = "0.1.0"
|
||||
description = "Operational scripts for the Council PKB — AudioPen ingest, webhook receiver, inbox normalisation."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = []
|
||||
|
||||
# Empty today. Stdlib-only code in this directory. Add entries here when a real
|
||||
# pip-installable dependency arrives (e.g. watchdog, imap-tools). Bootstrap:
|
||||
#
|
||||
# cd scripts
|
||||
# python3 -m venv .venv
|
||||
# .venv/bin/pip install -e .
|
||||
#
|
||||
# Or with uv:
|
||||
#
|
||||
# cd scripts
|
||||
# uv venv
|
||||
# uv pip install -e .
|
||||
|
||||
[project.optional-dependencies]
|
||||
# Example shape for when deps arrive — split by tool so you can install only what
|
||||
# you run. Uncomment and populate as needed.
|
||||
#
|
||||
# imap = [
|
||||
# "imap-tools>=1.4.0",
|
||||
# ]
|
||||
# watch = [
|
||||
# "watchdog>=4.0.0",
|
||||
# ]
|
||||
|
||||
[build-system]
|
||||
requires = ["setuptools>=68"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[tool.setuptools]
|
||||
# There's no package to build — these are standalone scripts. Point setuptools
|
||||
# at an empty package list so `pip install -e .` succeeds as a no-op installer
|
||||
# for the dependency manifest, without trying to import a module named
|
||||
# `council-scripts`.
|
||||
packages = []
|
||||
Reference in New Issue
Block a user