jpmschweitzerandClaude
06e4543224
Two uvicorn workers each hold a LISTEN connection on wiki_page_changes, so Postgres delivers every notification to both and each page edit is ingested twice. Found while verifying the v1.9.1 reconnect fix — the reconnect logged twice, which is what gave it away. Proven with a colon-free pg_notify payload that is rejected before ingestion runs: one NOTIFY, two "Invalid notification payload" lines. Recorded with the reproduction rather than the conclusion alone, since whether this corrupts data or is only wasteful has not been established. Co-Authored-By: Claude <noreply@anthropic.com>
86 lines
3.2 KiB
Markdown
86 lines
3.2 KiB
Markdown
# TODO
|
|
|
|
Outstanding work items for Library Desk.
|
|
|
|
## Bugs
|
|
|
|
### Every wiki page change is ingested twice
|
|
|
|
`Dockerfile` runs `uvicorn ... --workers 2`, and `startup_event` in `src/main.py`
|
|
creates a `WikiChangeListener` per worker. Each opens its own PostgreSQL `LISTEN`
|
|
connection on `wiki_page_changes`, and Postgres delivers `NOTIFY` to *every*
|
|
listening session — so both workers process the same event.
|
|
|
|
Proven against production v1.9.1 on 2026-08-08:
|
|
|
|
- `pg_stat_activity` shows two `library_desk_listener` sessions, both
|
|
`LISTEN "wiki_page_changes"`
|
|
- "Wiki.js change listener started successfully" is logged twice at startup
|
|
- a single `pg_notify('wiki_page_changes', 'claude-probe-no-colons')` produced
|
|
exactly two "Invalid notification payload" lines, one per worker (that payload
|
|
is rejected before any ingestion runs, so it is a side-effect-free probe)
|
|
|
|
The debounce in `wiki_change_listener.py` (`self._recent_notifications`) is an
|
|
in-process dict and cannot dedupe across workers. No Redis lock or other
|
|
cross-process guard exists in the `process_wiki_page_change` path in
|
|
`src/routers/webhooks.py`.
|
|
|
|
**Impact:** each edit runs ingestion twice — duplicate embedding generation
|
|
against Ollama plus duplicate Qdrant and Neo4j writes. On tower-of-joy, Ollama
|
|
shares the GPU with Speaches/Whisper, so the wasted work has a cost beyond CPU.
|
|
|
|
**Not yet established:** whether this is merely wasteful or actually corrupting.
|
|
Check whether repeated ingestion of the same page creates duplicate Neo4j
|
|
entities/relationships or duplicate Qdrant points — that changes the severity.
|
|
|
|
**Options to weigh:**
|
|
|
|
- run the listener in one place only (single-worker sidecar, or elect an owner)
|
|
so the subscription is singular by construction
|
|
- keep per-worker listeners and add a short-TTL Redis claim lock keyed on
|
|
`page_id`, so only the first worker to claim a notification processes it.
|
|
Redis is already a dependency — the job manager uses it.
|
|
|
|
Whatever approach is taken must preserve the reconnect supervision added in
|
|
v1.9.1 and its tests in `tests/test_wiki_change_listener.py`.
|
|
|
|
## Stub Endpoints to Implement
|
|
|
|
The following endpoints in `src/main.py` return stub responses and need real implementations:
|
|
|
|
### Ingestion Status Endpoints
|
|
|
|
#### `POST /ingest/check-updates`
|
|
Check which documents need updating based on content hashes. Used by Scheduler to determine what changed since last sync.
|
|
|
|
**Implementation needed:**
|
|
1. Query existing documents by path
|
|
2. Compare content hashes
|
|
3. Return list of updates needed
|
|
|
|
#### `GET /ingest/status/{document_id}`
|
|
Get processing status for a document.
|
|
|
|
**Implementation needed:**
|
|
- Status tracking system (Redis or database)
|
|
- Track ingestion progress per document
|
|
|
|
#### `GET /ingest/repo-status/{repository}`
|
|
Get indexing status for an entire repository.
|
|
|
|
**Implementation needed:**
|
|
- Repository-level statistics
|
|
- Track which documents from a repo are indexed
|
|
|
|
### Deduplication
|
|
|
|
#### `POST /deduplicate/check`
|
|
Check for duplicate or highly similar documents using vector similarity and graph analysis.
|
|
|
|
**Implementation needed:**
|
|
1. Get document embedding from Qdrant
|
|
2. Find similar vectors above threshold
|
|
3. Check graph relationships
|
|
4. Return candidates with similarity scores
|
|
|