Commit Graph
3 Commits
Author SHA1 Message Date
jpmschweitzerandClaude Fable 5 9681a63757 fix: upsert document vectors before pruning stale chunks
DocumentSyncService._index_vectors ran delete_by_filter on the
document's existing chunks FIRST and only then embedded; if the
embedding pass failed (Ollama down) the Paperless document was left
with zero vectors until the next successful sync - the same
zero-vector hazard already fixed for wiki pages in
VectorService.update_from_page.

Chunk ids are now deterministic uuid5 (document_{id}_chunk_{i}) so
re-upserting overwrites in place; new points are upserted first, then
stale points (including legacy random-uuid4 ones) are pruned via
scroll + delete_by_ids, and only after a successful upsert.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
2026-07-14 15:19:03 +02:00
jpmschweitzerandClaude Fable 5 9ceec1464a fix: WS5 hazards batch - CORS, scheduler auth, reranker dedupe, wiring, write txns
- CORS: drop allow_credentials (wildcard origin + credentials told
  browsers to attach credentials for any site); origins configurable via
  CORS_ALLOW_ORIGINS (default * is safe without credentials). Verified
  live: preflight no longer advertises access-control-allow-credentials.
- Scheduler tasks: auth moved from a plain Authorization header (which
  the Scheduler's rest_api_executor does NOT env-substitute) to its
  auth {type: bearer, token: ${LIBRARY_API_KEY}} block, substituted from
  the Scheduler's own environment at execution time. The registrar no
  longer resolves the real key client-side, so it can never be persisted
  into the scheduled_tasks.config JSONB column. Also fixed: JSON bodies
  moved from the ignored "body" key to "payload" (the executor only
  reads config["payload"], so the tasks would have POSTed empty bodies
  and failed required-user validation).
- Reranker: parsed ranking indices are deduplicated preserving first
  occurrence (an LLM answer like "3,3,1" duplicated a result).
- HybridRAG wiring consolidated into dependencies.get_hybrid_rag_service
  (now including volatile_service); the inline copies in /query/hybrid
  and /wiki/pages/smart-create are gone - smart-create previously ran
  without the volatile leg, and the singleton was unused.
- Remaining Neo4j writes (GraphService ingestion/deletes/purges/entity
  mentions, webhook rename+delete cleanup, document-sync _index_graph,
  consolidation mark-processed/add-entity) moved from auto-commit
  execute_query to execute_write managed transactions with retry.

Verified end-to-end on the local dev server as llm_tester: /query/hybrid
200 with all five legs ok (volatile now active), background persistence
landed as one transaction.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
2026-07-14 14:50:03 +02:00
jpmschweitzerandClaude Fable 5 c35d3c1fa9 fix: await ensure_collection and survive partial embed failures in document sync
- ensure_collection was called without await, so the coroutine never ran
  and fresh tenants had no collection when the upsert hit Qdrant
- a single None entry from embed_batch poisoned the point batch and
  aborted the whole document upsert; failed chunks are now skipped with
  a warning (all-failed raises and the IndexResult reports failure)
- raw client.delete/client.upsert calls now go through the async wrapper
  (delete_by_filter and the new batch upsert_points method)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
2026-07-14 13:00:59 +02:00