library-desk/TODO.md

TODO

Outstanding work items for Library Desk.

Bugs

Every wiki page change is ingested twice

Dockerfile runs uvicorn ... --workers 2, and startup_event in src/main.py creates a WikiChangeListener per worker. Each opens its own PostgreSQL LISTEN connection on wiki_page_changes, and Postgres delivers NOTIFY to every listening session — so both workers process the same event.

Proven against production v1.9.1 on 2026-08-08:

• pg_stat_activity shows two library_desk_listener sessions, both LISTEN "wiki_page_changes"
• "Wiki.js change listener started successfully" is logged twice at startup
• a single pg_notify('wiki_page_changes', 'claude-probe-no-colons') produced exactly two "Invalid notification payload" lines, one per worker (that payload is rejected before any ingestion runs, so it is a side-effect-free probe)

The debounce in wiki_change_listener.py (self._recent_notifications) is an in-process dict and cannot dedupe across workers. No Redis lock or other cross-process guard exists in the process_wiki_page_change path in src/routers/webhooks.py.

Impact: each edit runs ingestion twice — duplicate embedding generation against Ollama plus duplicate Qdrant and Neo4j writes. On tower-of-joy, Ollama shares the GPU with Speaches/Whisper, so the wasted work has a cost beyond CPU.

Not yet established: whether this is merely wasteful or actually corrupting. Check whether repeated ingestion of the same page creates duplicate Neo4j entities/relationships or duplicate Qdrant points — that changes the severity.

Options to weigh:

• run the listener in one place only (single-worker sidecar, or elect an owner) so the subscription is singular by construction
• keep per-worker listeners and add a short-TTL Redis claim lock keyed on page_id, so only the first worker to claim a notification processes it. Redis is already a dependency — the job manager uses it.

Whatever approach is taken must preserve the reconnect supervision added in v1.9.1 and its tests in tests/test_wiki_change_listener.py.

Stub Endpoints to Implement

The following endpoints in src/main.py return stub responses and need real implementations:

Ingestion Status Endpoints

POST /ingest/check-updates

Check which documents need updating based on content hashes. Used by Scheduler to determine what changed since last sync.

Implementation needed:

1. Query existing documents by path
2. Compare content hashes
3. Return list of updates needed

GET /ingest/status/{document_id}

Get processing status for a document.

Implementation needed:

• Status tracking system (Redis or database)
• Track ingestion progress per document

GET /ingest/repo-status/{repository}

Get indexing status for an entire repository.

Implementation needed:

• Repository-level statistics
• Track which documents from a repo are indexed

Deduplication

POST /deduplicate/check

Check for duplicate or highly similar documents using vector similarity and graph analysis.

Implementation needed:

1. Get document embedding from Qdrant
2. Find similar vectors above threshold
3. Check graph relationships
4. Return candidates with similarity scores