jpmschweitzer/council
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add AudioPen ingest pipeline and sync architecture setup docs

Browse Source 
Voice-note capture as the primary mobile input channel. AudioPen
polishes on their cloud, POSTs to a self-hosted receiver on the
desktop (exposed via Tailscale Funnel so no router ports open and
nothing behind Authentik). The receiver writes to inbox/raw/; a bash
normaliser wraps each drop in frontmatter and moves it to inbox/;
/triage-inbox routes them from there.

scripts/audiopen-ingest.sh — 80-line bash normaliser. Idempotent;
safe to re-run. Extracts title, slugifies, computes capture timestamp
from mtime, rewrites with fleeting-note frontmatter.

scripts/audiopen-webhook/main.py — stdlib-only Python HTTP server.
Zero pip deps; binds to 127.0.0.1 by default. Accepts flexible
payload shapes (title/name + body/output/summary/polished/
orig_transcript) so AudioPen version drift is logged rather than
silently dropped.

scripts/audiopen-webhook/*.example — systemd user-unit templates for
the receiver and the path/service pair that fires the ingest wrapper
on inbox/raw/ changes.

scripts/pyproject.toml + README.md — Python venv convention. Zero
deps today; venv location reserved at scripts/.venv/ (gitignored),
manifest at scripts/pyproject.toml, bootstrap documented for both
plain pip and uv. Optional-dependencies groups let individual tools
pull what they need without bloating the whole env.

docs/setup/audiopen.md — full setup guide: generating the shared
secret, installing the systemd units, pairing with Tailscale Funnel,
configuring AudioPen's webhook, and the IMAP-fallback path for
setups that can't run a public-reachable receiver.

docs/setup/sync.md — companion guide: Gitea SSH remote (works even
with Authentik gating HTTPS), obsidian-git plugin configuration,
Syncthing desktop↔phone pairing with the critical .stignore patterns,
and the three phone-role tiers so the user can pick Tier 1 / 2 / 3
at their own pace.

.gitignore gains **/.venv/, **/venv/, **/__pycache__/, and *.pyc so
nobody accidentally commits a materialised environment.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jpmschweitzer
2026-04-19 20:17:48 +02:00
co-authored by Claude Opus 4.7
parent 2d5615b1ba
commit 2c8df9425b
10 changed files with 850 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/setup/audiopen.md
+210
View File
@@ -0,0 +1,210 @@
# AudioPen → PKB setup
Voice-note capture pipeline. AudioPen records on phone, polishes on their cloud, POSTs
the polished prose to your own infra. This vault's `inbox/raw/` receives the drop;
`scripts/audiopen-ingest.sh` normalises frontmatter and moves it to `inbox/`;
`/triage-inbox` routes it into `daily/`, `notes/`, `projects/`, or `reading/`.
Two ingest paths are supported. Pick one. Either works with the same
`/triage-inbox` downstream.
---
## Path A — Webhook via Tailscale Funnel (recommended)
**Requires.** Tailscale installed on the desktop with Funnel enabled on your tailnet.
Tailscale Funnel provides an outbound-initiated HTTPS endpoint (`*.ts.net`) with no
router configuration and no exposed ports. Nothing sits behind Authentik; AudioPen
hits the Funnel URL directly.
### 1. Generate a shared secret
```sh
head -c 24 /dev/urandom | base64 | tr -d '/+=' > /tmp/secret
cat /tmp/secret
```
Store it privately; it becomes part of the webhook URL.
### 2. Install the receiver systemd unit
```sh
mkdir -p ~/.config/audiopen-webhook ~/.config/systemd/user
printf 'AUDIOPEN_WEBHOOK_SECRET=%s\n' "$(cat /tmp/secret)" > ~/.config/audiopen-webhook/env
chmod 600 ~/.config/audiopen-webhook/env
cp /var/mnt/data/projects/council/scripts/audiopen-webhook/audiopen-webhook.service.example \
~/.config/systemd/user/audiopen-webhook.service
systemctl --user daemon-reload
systemctl --user enable --now audiopen-webhook.service
systemctl --user status audiopen-webhook.service # should show active (running)
```
Tail logs:
```sh
journalctl --user -u audiopen-webhook.service -f
```
### 3. Install the ingest path + service units
```sh
# audiopen-ingest.service — normalises raw drops
cat > ~/.config/systemd/user/audiopen-ingest.service <<'EOF'
[Unit]
Description=Normalise AudioPen drops into council/inbox/
[Service]
Type=oneshot
ExecStart=/var/mnt/data/projects/council/scripts/audiopen-ingest.sh
Environment=COUNCIL_REPO=/var/mnt/data/projects/council
EOF
# audiopen-ingest.path — fires the service whenever inbox/raw/ gains a file
cat > ~/.config/systemd/user/audiopen-ingest.path <<'EOF'
[Unit]
Description=Watch council/inbox/raw for new AudioPen drops
[Path]
PathChanged=/var/mnt/data/projects/council/inbox/raw
Unit=audiopen-ingest.service
[Install]
WantedBy=default.target
EOF
systemctl --user daemon-reload
systemctl --user enable --now audiopen-ingest.path
```
### 4. Expose the receiver via Tailscale Funnel
```sh
tailscale funnel --bg 8765
tailscale funnel status # show the public URL
```
Your endpoint is now `https://<your-host>.<tailnet>.ts.net:443/`. AudioPen will
POST to `https://<your-host>.<tailnet>.ts.net/audiopen/<secret>`.
### 5. Configure AudioPen
In AudioPen (web or mobile app): settings → integrations → add a generic webhook.
- **URL:** `https://<your-host>.<tailnet>.ts.net/audiopen/<secret>`
- **Method:** POST
- **Content-Type:** application/json
- **Trigger:** after each recording is polished
AudioPen typically sends JSON with keys like `title`, `body`, `orig_transcript`.
The receiver accepts any of these; if AudioPen changes shape, check
`journalctl --user -u audiopen-webhook.service` for the logged payload and
update `scripts/audiopen-webhook/main.py::extract_content()` if needed.
### 6. Test end-to-end
```sh
SECRET=$(awk -F= '/AUDIOPEN_WEBHOOK_SECRET/ {print $2}' ~/.config/audiopen-webhook/env)
FUNNEL=$(tailscale funnel status | awk '/https:/ {print $1; exit}')
curl -X POST -H 'Content-Type: application/json' \
-d '{"title":"Test","body":"Hello from curl. This is a test drop."}' \
"${FUNNEL%/}/audiopen/${SECRET}"
# → ok
ls /var/mnt/data/projects/council/inbox/raw # should contain the test file briefly
sleep 2
ls /var/mnt/data/projects/council/inbox # should now contain the normalised file
```
Then record an actual voice note in AudioPen, wait for polishing, confirm the same
flow. Run `/triage-inbox` in Claude Code when you're ready to route the contents.
---
## Path B — Email → IMAP fetch (no-open-ports fallback)
Use this if you don't run Tailscale / Cloudflare Tunnel and don't want to stand one
up. Fully outbound: AudioPen emails each note; a cron on the desktop pulls via IMAP
and drops files in `inbox/raw/`.
### 1. Dedicated email address
Create or repurpose a mailbox for AudioPen's output. Options:
- Fastmail alias → Fastmail IMAP on desktop (works).
- Self-hosted mail server → pull via IMAP from localhost (works).
- Gmail "+audiopen" alias → fine but adds Google to the dependency chain.
### 2. IMAP fetch script (stub — you'll customise for your mail server)
```python
#!/usr/bin/env python3
"""Minimal IMAP fetcher: pulls unread messages from a mailbox, writes them as
bare markdown to inbox/raw/. Uses stdlib imaplib. Keep credentials in
~/.config/audiopen-mail/env or similar."""
import imaplib, email, os, re, pathlib, datetime
HOST = os.environ["IMAP_HOST"]
USER = os.environ["IMAP_USER"]
PASS = os.environ["IMAP_PASS"]
MBOX = os.environ.get("IMAP_MAILBOX", "INBOX")
INBOX = pathlib.Path(os.environ.get("COUNCIL_INBOX_RAW",
"/var/mnt/data/projects/council/inbox/raw"))
INBOX.mkdir(parents=True, exist_ok=True)
m = imaplib.IMAP4_SSL(HOST)
m.login(USER, PASS)
m.select(MBOX)
typ, data = m.search(None, 'UNSEEN')
for num in data[0].split():
typ, msg_data = m.fetch(num, '(RFC822)')
msg = email.message_from_bytes(msg_data[0][1])
title = (msg.get("Subject") or "Untitled").strip()
body = ""
if msg.is_multipart():
for p in msg.walk():
if p.get_content_type() == "text/plain":
body = p.get_payload(decode=True).decode('utf-8', errors='replace')
break
else:
body = msg.get_payload(decode=True).decode('utf-8', errors='replace')
slug = re.sub(r"[^a-z0-9]+", "-", title.lower()).strip("-")[:60] or "untitled"
ts = datetime.datetime.now().strftime("%Y-%m-%d-%H%M")
(INBOX / f"{ts}_{slug}.md").write_text(f"# {title}\n\n{body}\n")
m.store(num, '+FLAGS', '\\Seen')
m.logout()
```
Save as `scripts/audiopen-imap-fetch.py`. Run it every N minutes via a systemd
user-timer or cron. The same `audiopen-ingest.path` unit (from Path A, step 3)
picks up the drops from there.
### 3. Configure AudioPen
Settings → integrations → email output → recipient = your dedicated address.
Each recording → one email. Done.
---
## Routine
Once everything is wired:
1. **Capture**: voice note → AudioPen → (webhook or email) → desktop.
2. **Ingest**: automatic — you don't run anything; systemd fires on file change.
3. **Route**: when you want, open a Claude Code session in the vault and run
`/triage-inbox`. Koskela proposes destinations; you approve in batch.
4. **Weekly/monthly**: the promoted notes are already in their homes; `/reflect`,
`/review-week`, and `/review-month` will see them naturally.
## Troubleshooting
| Symptom | Check |
|---|---|
| Files pile up in `inbox/raw/` but never normalise | `systemctl --user status audiopen-ingest.path audiopen-ingest.service` — path unit enabled? service errored? Run the script manually once: `bash scripts/audiopen-ingest.sh`. |
| Webhook POSTs 200 but no file lands | Check the systemd logs for the receiver: `journalctl --user -u audiopen-webhook.service -n 50`. The logged path should be writable and under `inbox/raw/`. |
| Webhook POSTs 404 | Secret mismatch between AudioPen URL and `~/.config/audiopen-webhook/env`. Regenerate and re-set both sides. |
| Funnel URL unreachable | `tailscale status`; `tailscale funnel status`. Re-run `tailscale funnel --bg 8765`. Confirm Funnel is enabled on your tailnet (admin → DNS → HTTPS Certificates; admin → Access controls → Funnel). |
| Normalised file missing frontmatter | Bug in the ingest wrapper; open the raw file, run the script with `bash -x scripts/audiopen-ingest.sh` to trace. |
docs/setup/sync.md
+174
View File
@@ -0,0 +1,174 @@
# Sync architecture — setup guide
Three independent sync channels. Setting each up on a fresh machine.
## 1. Gitea over SSH — version history + off-site backup
**Already configured** on the desktop for this repo:
```sh
$ git remote -v
origin ssh://git@git.schweitz.net:2222/jpmschweitzer/council.git (fetch)
origin ssh://git@git.schweitz.net:2222/jpmschweitzer/council.git (push)
```
On a fresh machine:
```sh
git clone ssh://git@git.schweitz.net:2222/jpmschweitzer/council.git
cd council
```
Requires an SSH key registered in Gitea (Gitea settings → SSH keys → add).
Gitea is behind Authentik for HTTPS, so HTTPS clones won't work without the browser
auth dance. SSH bypasses that entirely — the ssh daemon on Gitea handles key auth
independently.
### obsidian-git plugin (desktop)
The `obsidian-git` plugin is already installed in `.obsidian/plugins/`. Configure it
to auto-commit and push on your chosen cadence:
1. Obsidian settings → Community plugins → Obsidian Git.
2. "Remote URL" — leave blank; it uses the repo's existing `origin`.
3. "Auto commit-and-sync interval" — e.g. 10 minutes. Set to 0 to disable auto-push.
4. "Commit message on auto commit-and-sync" — use the default template.
5. Confirm it can push: Command Palette → "Obsidian Git: Push".
### What NOT to commit
Already set in `.gitignore`:
- `.env`, `.env.local`, `.env.*.local`
- `.obsidian/workspace.json`, `.obsidian/workspace-mobile.json`, `.obsidian/cache`
- `.claude/settings.local.json`
- `**/.venv/`, `**/venv/`, `**/__pycache__/`, `*.pyc`
---
## 2. Syncthing — desktop ↔ phone peer-to-peer sync
Optional. Skip if you're picking **Tier 1** from the phone-role decision (AudioPen
only, no mobile vault).
### Desktop setup
Syncthing is available on Bazzite via rpm-ostree (layered) or Flatpak. Flatpak is
simpler:
```sh
flatpak install flathub me.kozec.syncthingtk
# or
rpm-ostree install syncthing
```
Start it:
```sh
systemctl --user enable --now syncthing.service
# Web UI:
xdg-open http://127.0.0.1:8384
```
In the Syncthing web UI:
1. **Add folder**: path = `/var/mnt/data/projects/council/`, folder ID = `council`.
2. **Ignore patterns** (crucial — click "Edit" → "Ignore Patterns" on the folder):
```
.git
.obsidian/workspace.json
.obsidian/workspace-mobile.json
.obsidian/cache
.trash
**.sync-conflict-*
scripts/.venv
```
This list matters. Syncthing must never propagate `.git/` — partial packs mid-commit
would corrupt the mirror.
### Phone setup (Android)
1. Install **Syncthing-Fork** from F-Droid (more robust than upstream Syncthing on
Android).
2. Pair with desktop: desktop shows its device ID in the web UI → enter on phone
→ desktop accepts the pairing.
3. On the phone, accept the `council` folder share. Set its target directory to
e.g. `/sdcard/Documents/council/`.
4. Install **Obsidian Mobile**. Open as vault → pick the Syncthing-watched path.
### Conflict discipline
Both devices edit the same file while disconnected → Syncthing creates
`<filename>.sync-conflict-YYYYMMDD-HHMMSS-deviceID.md`. Rare if you edit one device
at a time. To resolve: read both, copy the good bits into the canonical file,
delete the conflict.
### How the Syncthing/git channels interact
- **Git lives on desktop only.** The phone never sees `.git/` (it's in `.stignore`).
- **Syncthing replicates content only.** The phone has the markdown; it doesn't have
history.
- **To push from desktop**: just `git push` as usual. obsidian-git automates it.
- **To pull new content on desktop from phone**: Syncthing already did it; there's
nothing to pull. Review the changes in `git status` and commit them like any
other working-tree edit.
---
## 3. AudioPen webhook via Tailscale Funnel — voice-note ingestion
See [`docs/setup/audiopen.md`](./audiopen.md). Independent of the two above.
---
## Phone-role decision
When you set up Obsidian Mobile (if at all), pick a tier:
| Tier | Phone runs | Trade-off |
|---|---|---|
| 1 | AudioPen only | Simplest. No vault on phone. Read/edit on desktop. |
| 2 | AudioPen + browser → Gitea markdown view | Occasional phone read via Authentik-gated Gitea. |
| 3 | AudioPen + Syncthing-Fork + Obsidian Mobile | Full vault on phone. Maximum capability, moderate battery + ~few hundred MB storage for the vault. |
All three are compatible with the rest of the stack. The AudioPen pipeline
(path A or B) is identical across tiers.
---
## Verification
After everything is set up:
```sh
# Git remote reachable
git ls-remote origin | head -3 # should list refs from Gitea
# Syncthing running (if Tier 2/3)
systemctl --user is-active syncthing.service
# → active
# Tailscale Funnel for AudioPen webhook
tailscale funnel status
# should list the https endpoint for :8765
# AudioPen receiver running
systemctl --user is-active audiopen-webhook.service
# → active
# Push a test commit
cd /var/mnt/data/projects/council
echo "# test" > /tmp/sync-test.md
mv /tmp/sync-test.md daily/2099-01-01.md
git add daily/2099-01-01.md
git commit -m "test sync pipeline" -m "Co-Authored-By: me"
git push
# should succeed
# On phone (Tier 3): wait a moment, open Obsidian Mobile, verify daily/2099-01-01.md appears.
# Clean up
git rm daily/2099-01-01.md
git commit -m "drop sync test"
git push
```