SchedulerClient.register_volatile_fetch (consolidation's prefetch
routing) registered tasks that were dead on arrival:
- JSON body stored under 'body', which rest_api_executor ignores
(it only reads config['payload'])
- no auth block, so the scheduled POST to /volatile/fetch would 401
against library-desk's verify_api_key
- user placed in the body while /volatile/fetch endpoints require it
as a query parameter (RequiredUserQuery) - would 422 regardless
The task config now carries user in the URL query string (encoded),
an empty payload, and auth {type: bearer, token: ${LIBRARY_API_KEY}}
substituted Scheduler-side (never stored raw).
SchedulerClient also sent no Authorization to the Scheduler API itself,
so registration 401'd silently at consolidation time; it now sends
Bearer auth from the new scheduler_api_key setting.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
- CORS: drop allow_credentials (wildcard origin + credentials told
browsers to attach credentials for any site); origins configurable via
CORS_ALLOW_ORIGINS (default * is safe without credentials). Verified
live: preflight no longer advertises access-control-allow-credentials.
- Scheduler tasks: auth moved from a plain Authorization header (which
the Scheduler's rest_api_executor does NOT env-substitute) to its
auth {type: bearer, token: ${LIBRARY_API_KEY}} block, substituted from
the Scheduler's own environment at execution time. The registrar no
longer resolves the real key client-side, so it can never be persisted
into the scheduled_tasks.config JSONB column. Also fixed: JSON bodies
moved from the ignored "body" key to "payload" (the executor only
reads config["payload"], so the tasks would have POSTed empty bodies
and failed required-user validation).
- Reranker: parsed ranking indices are deduplicated preserving first
occurrence (an LLM answer like "3,3,1" duplicated a result).
- HybridRAG wiring consolidated into dependencies.get_hybrid_rag_service
(now including volatile_service); the inline copies in /query/hybrid
and /wiki/pages/smart-create are gone - smart-create previously ran
without the volatile leg, and the singleton was unused.
- Remaining Neo4j writes (GraphService ingestion/deletes/purges/entity
mentions, webhook rename+delete cleanup, document-sync _index_graph,
consolidation mark-processed/add-entity) moved from auto-commit
execute_query to execute_write managed transactions with retry.
Verified end-to-end on the local dev server as llm_tester: /query/hybrid
200 with all five legs ok (volatile now active), background persistence
landed as one transaction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
Every vector call ran on the sync QdrantClient inside async wrapper
methods, blocking the FastAPI event loop per Qdrant round-trip. The
wrapper now holds an AsyncQdrantClient (timeout via QDRANT_TIMEOUT,
default 30s) and awaits all client calls; the wrapper API is unchanged.
Call sites off the wrapper were fixed too: the HybridRAG document leg
now uses the async search_vectors wrapper instead of the deprecated raw
client.search (also fixing its call to the nonexistent ollama.embed_text
which made the leg permanently report 'failed'), the health check awaits
get_collections, and document_sync's raw delete/upsert calls are awaited
(routed through wrappers in the next commit).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
Replace the four stub endpoints with real implementations, all requiring
an explicit tenant user (Phase B rule):
- /ingest/check-updates: GraphService now records a SHA-256 content_hash
on every Document node at ingestion time; the endpoint compares those
stored hashes against current Wiki.js page content in one UNWIND Cypher
query per tenant and returns changed/new/deleted page lists (entity-stub
pages excluded, pre-hash-tracking documents flagged stored_hash_missing).
- /ingest/status/{job_id}: backed by the Redis JobManager; jobs are
tenant-scoped (foreign jobs 404). /ingest/page, /ingest/batch and
/ingest/all now create job records and return job_id.
- /ingest/repo-status/{repository}: wiki page count vs indexed Document
nodes under users/{tenant}/{repository} plus tenant job stats.
- /deduplicate/check: tenant-scoped Qdrant similarity scan; chunk pairs
above ~0.9 cosine from different pages grouped per page pair with best
score and page references (read-only).
Supporting changes: get_job_manager dependency (+ shutdown close),
scroll_all_points can return vectors, VectorService.find_duplicate_pairs,
src/core/hashing.compute_content_hash. 13 new offline unit tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
A live /query/hybrid probe as user=llm_tester returned jpmschweitzer
pages. Audit of all legs (vector, graph, web-persistence, volatile,
documents) plus enrichment/persistence found and fixed these unscoped
paths:
- vector_service.update_from_page and graph_service.update_from_page now
refuse pages outside users/{user}/ - previously any tenant could
ingest any wiki page (incl. another tenant's) into its own collection
and graph labels, which is how foreign content entered the vector leg.
- ingestion_service.ingest_all_pages clamps path_prefix to the caller's
namespace (segment-exact, sanitized comparison) and defaults to
users/{user}; /ingest/all returns 400 on cross-tenant prefixes.
- hybrid_rag_service._persist_search_for_librarian linked SearchQuery
nodes to unscoped (d:Document {page_id}); now matches only
User_{Tenant}_Document nodes.
- graph_service: _get_entity_mention_count, entity-stub mention/related
queries, generate_entity_stubs, find/purge_orphan_entities matched
unscoped Document nodes; cleanup_broken_relationships matched all
tenants' SearchQuery nodes; _entity_has_wiki_page listed all wiki
pages. All are now tenant-label / namespace scoped.
- volatile_service collection names now use the sanitized user id.
- is_path_in_user_namespace enforces a path-segment boundary
(users/llm_tester2 is not llm_tester's namespace) and treats
hyphen/underscore tenant spellings as the same sanitized tenant.
- New offline unit tests per leg (mocked clients) assert the
tenant-scoped collection/label/path is used and cross-tenant access
is refused.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Remove the implicit jpmschweitzer default tenant (DEFAULT_USER) from
src/core/multi_tenancy.py and every endpoint and request model that
inherited it (~40 endpoints across /query, /wiki, /vector, /graph,
/ingest, /volatile, /documents, /stats, /rag).
- Add validate_required_user() + RequiredUser pydantic type in
multi_tenancy and a shared require_user FastAPI dependency
(RequiredUserQuery) that rejects missing, empty, and whitespace-only
users with 422, following the /maintenance/* pattern.
- Wiki page create / smart-create / dossier request models now require
user (no fallback in wiki_service).
- /maintenance/cleanup/test-data derives the tenant from the page path
instead of using the production tenant collection.
- Wiki.js change listener skips changes when no tenant user can be
derived from the notification email instead of defaulting to the
production tenant.
- Consolidation service internal helpers no longer default to the
production tenant.
- Tool catalog marks user as required with honest descriptions.
- OpenAPI descriptions updated honestly; CHANGELOG notes that callers
(tatlock, Scheduler ingest tasks) must now send explicit user.
- Offline tests: 422 coverage for query/body endpoints, required-user
validator tests; updated legacy tests that assumed a default tenant.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add SchedulerClient to communicate with external scheduler service for
registering volatile prefetch tasks discovered during HybridRAG searches.
- Add scheduler_client.py with full REST API for task CRUD operations
- Add scheduler_url config setting (default: http://scheduler:8090)
- Update consolidation service to use scheduler for prefetch registration
- Add scheduler health checks to startup/shutdown lifecycle
When HybridRAG classifies web content as prefetch-worthy, it now creates
scheduled tasks that periodically refresh the volatile cache via the
external scheduler service.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add MemoryRouteClassification and MemoryRoutingResult models
- Implement unified classifier (_classify_web_results_unified) that routes
web results to: wiki, volatile, file (Paperless), prefetch, or skip
- Add routing methods: _route_to_volatile, _route_to_files, _register_prefetch
- Update _process_search to use unified classifier instead of separate analysis
- Add get_volatile_cache_service factory to dependencies
- Wire volatile_service and settings_client into ConsolidationService
- Update ConsolidationResult/Response with new routing counters
Test fixes:
- Fix WikiJSClient fixtures to use api_token instead of username/password
- Fix entity linking test assertions to expect full user-namespaced paths
- Add sample_unified_classification fixture for new classifier format
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add /documents router with webhook, upload, search, health endpoints
- Create DocumentSyncService for indexing documents to vectors/graph
- Add PaperlessClient for REST API integration
- Configure dependency injection for Paperless client
- Add document models for webhook payloads and responses
- Event-driven architecture via Paperless workflow webhooks
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Complete maintenance subsystem for index health and cleanup:
Endpoints:
- GET /maintenance/health - lightweight (or detailed) health check
- POST /maintenance/cleanup/all - full orphan cleanup
- POST /maintenance/cleanup/vectors - purge orphan vector chunks
- POST /maintenance/cleanup/graph - purge orphan graph nodes
- POST /maintenance/reconcile-index - cleanup + reindex missing pages
Bidirectional orphan detection:
- find_documents_without_vectors() in GraphService
- find_chunks_without_graph_nodes() in VectorService
Redis integration:
- Tracks last_cleanup timestamp for scheduler visibility
Config additions:
- Document store, volatile cache, and maintenance settings
- VectorServiceDep and GraphServiceDep type aliases
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add OLLAMA_EMBEDDING_MODEL for embeddings (nomic-embed-text)
- OLLAMA_MODEL now used for all LLM operations (mistral-nemo-large:latest)
- Remove separate reranker_model setting
- Update WikiPageWriter to use settings instead of hardcoded model
- Improves VRAM efficiency by keeping one model hot
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add /rag/search endpoint for web, news, and image search via SearXNG
- Add /content/extract and /content/extract/batch endpoints
- Add ContentExtractor client using Trafilatura for content extraction
- Enhance HybridRAG web search with full content extraction
- Add Redis caching for search results
- Add new configuration options for search and extraction timeouts
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
