docs: record the duplicate wiki ingestion bug

Two uvicorn workers each hold a LISTEN connection on wiki_page_changes, so
Postgres delivers every notification to both and each page edit is ingested
twice. Found while verifying the v1.9.1 reconnect fix — the reconnect logged
twice, which is what gave it away.

Proven with a colon-free pg_notify payload that is rejected before ingestion
runs: one NOTIFY, two "Invalid notification payload" lines. Recorded with the
reproduction rather than the conclusion alone, since whether this corrupts data
or is only wasteful has not been established.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-08-08 21:51:30 +02:00
co-authored by Claude
parent ecbb0861a0
commit 06e4543224
+42
View File
@@ -2,6 +2,48 @@
Outstanding work items for Library Desk.
## Bugs
### Every wiki page change is ingested twice
`Dockerfile` runs `uvicorn ... --workers 2`, and `startup_event` in `src/main.py`
creates a `WikiChangeListener` per worker. Each opens its own PostgreSQL `LISTEN`
connection on `wiki_page_changes`, and Postgres delivers `NOTIFY` to *every*
listening session — so both workers process the same event.
Proven against production v1.9.1 on 2026-08-08:
- `pg_stat_activity` shows two `library_desk_listener` sessions, both
`LISTEN "wiki_page_changes"`
- "Wiki.js change listener started successfully" is logged twice at startup
- a single `pg_notify('wiki_page_changes', 'claude-probe-no-colons')` produced
exactly two "Invalid notification payload" lines, one per worker (that payload
is rejected before any ingestion runs, so it is a side-effect-free probe)
The debounce in `wiki_change_listener.py` (`self._recent_notifications`) is an
in-process dict and cannot dedupe across workers. No Redis lock or other
cross-process guard exists in the `process_wiki_page_change` path in
`src/routers/webhooks.py`.
**Impact:** each edit runs ingestion twice — duplicate embedding generation
against Ollama plus duplicate Qdrant and Neo4j writes. On tower-of-joy, Ollama
shares the GPU with Speaches/Whisper, so the wasted work has a cost beyond CPU.
**Not yet established:** whether this is merely wasteful or actually corrupting.
Check whether repeated ingestion of the same page creates duplicate Neo4j
entities/relationships or duplicate Qdrant points — that changes the severity.
**Options to weigh:**
- run the listener in one place only (single-worker sidecar, or elect an owner)
so the subscription is singular by construction
- keep per-worker listeners and add a short-TTL Redis claim lock keyed on
`page_id`, so only the first worker to claim a notification processes it.
Redis is already a dependency — the job manager uses it.
Whatever approach is taken must preserve the reconnect supervision added in
v1.9.1 and its tests in `tests/test_wiki_change_listener.py`.
## Stub Endpoints to Implement
The following endpoints in `src/main.py` return stub responses and need real implementations: