The comment still said five, and named only neo4j and qdrant as
unbounded. Both were true one commit ago. Also states the property that
makes adding probes safe: concurrent bounds do not sum, so wall time is
one bound regardless of count.
Co-Authored-By: Claude <noreply@anthropic.com>
paperless, system_settings and scheduler ran serially after the five
probes bounded in 80229f2, with no timeout of their own (scheduler
defaults to 30.0s, system_settings' connect() is unbounded) — the same
defect class, just further down the same function.
All eight probes now join one asyncio.gather(return_exceptions=True),
each bounded at 2s via the existing _bounded_probe. paperless and
system_settings keep their three-state result: an unconfigured service
is excluded from the probe list entirely rather than run through the
bound, so "not configured" (None) cannot collapse into "unhealthy"
(False) the way it would if the bound's bool-only wrapper were applied
uniformly. scheduler has no config gate and joins the gather plainly.
Neither is read by /health today — only src/main.py and
tests/test_integration.py call check_service_health(), and both read
only the original five — so this is unreachable from the live
endpoint. Bounded rather than deleted: bounding cannot break a
consumer that reads these keys later, deleting could.
check_service_health() ran neo4j, qdrant, wikijs, searxng and ollama
serially with await, and only ollama's client carried its own timeout.
Neo4j (connection_timeout=30.0) and Qdrant (timeout=30.0) fell back to
driver defaults far past the container healthcheck's 10s timeout, so a
hung (not failing) dependency blocked the whole chain and flipped the
container unhealthy for a reason unrelated to its own liveness.
Each probe now runs under asyncio.wait_for bounded at 2s — chosen
against the 10s healthcheck timeout so five concurrent bounded probes
cannot approach it even if all five hang — and all five run
concurrently under asyncio.gather(return_exceptions=True), so one
probe timing out or raising cannot block or cancel the others.
Gating (neo4j+qdrant only), the unconditional 200 response, ollama's
existing 5.0s client-level timeout, and the paperless/system_settings/
scheduler probes are unchanged.
105 findings to zero. Most were mechanical — 67 unused imports, and assorted
f-strings without placeholders. Three groups needed a decision.
The 15 F821 "undefined name" were forward references, not runtime errors. Each
annotation is quoted — `-> "WikiService"`, `Optional["IngestionService"]` — with
the real import inside the function body to break an import cycle. A quoted
annotation is never evaluated, so the code ran; the names were simply
unresolvable to any checker. They now have a TYPE_CHECKING block, which costs
nothing at import time and keeps the cycle broken.
The 6 E402 split two ways. `import secrets`, `Security`, `Request` and
`HTTPBearer` in dependencies.py had drifted below several hundred lines of
factory functions for no reason — stdlib and fastapi, no cycle to avoid — and
moved up. The other three are deliberate and now say so: the VectorService and
GraphService aliases import back into dependencies.py, and main.py's routers
expect a configured app, so both must stay put.
Bare `except:` narrowed to `except Exception:` in three places, which stops them
swallowing KeyboardInterrupt and SystemExit.
The 5 unused locals were all genuinely dead. One is worth naming rather than
fixing: qdrant_client.delete()'s return value was bound and never read, so a
failed delete is indistinguishable from a successful one — the assignment is
gone, but nothing checks the status either way and that has not changed here.
`timing = {}` in _retrieve_parallel looked like it might mean the reported
per-leg timings were always zero; traced, and they come from output["timing"],
so the local was only vestigial.
426 passed, 29 skipped, unchanged. The app imports and the service aliases still
resolve, which is the check that mattered after moving imports in
dependencies.py.
The gate still prints "not gated here yet: test (T-56)" — lint is green, tests
remain unwired, and that is left visible rather than silently absent.
Co-Authored-By: Claude <noreply@anthropic.com>
The wikijs-integration.js embedded a full-privilege API key that was
served to every wiki visitor — it unlocked all 66 authenticated
endpoints, including page/vector deletes and index purges. That key
has been rotated out of service.
The two browser endpoints (/ingest/page, /entity-linking/link-page)
now authenticate via the NPM /library-desk/ proxy location instead of a
key: Authentik forward-auth for external users, LAN bypass for internal,
verified by a trusted proxy marker header. This is safe because
library-desk binds loopback-only, so NPM is the sole path that can set
that header. The browser holds no secret; the script calls same-origin
with credentials. Machine callers (the Scheduler) keep the Bearer key
on the container-network endpoints. verify_api_key now compares in
constant time.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SchedulerClient.register_volatile_fetch (consolidation's prefetch
routing) registered tasks that were dead on arrival:
- JSON body stored under 'body', which rest_api_executor ignores
(it only reads config['payload'])
- no auth block, so the scheduled POST to /volatile/fetch would 401
against library-desk's verify_api_key
- user placed in the body while /volatile/fetch endpoints require it
as a query parameter (RequiredUserQuery) - would 422 regardless
The task config now carries user in the URL query string (encoded),
an empty payload, and auth {type: bearer, token: ${LIBRARY_API_KEY}}
substituted Scheduler-side (never stored raw).
SchedulerClient also sent no Authorization to the Scheduler API itself,
so registration 401'd silently at consolidation time; it now sends
Bearer auth from the new scheduler_api_key setting.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
- CORS: drop allow_credentials (wildcard origin + credentials told
browsers to attach credentials for any site); origins configurable via
CORS_ALLOW_ORIGINS (default * is safe without credentials). Verified
live: preflight no longer advertises access-control-allow-credentials.
- Scheduler tasks: auth moved from a plain Authorization header (which
the Scheduler's rest_api_executor does NOT env-substitute) to its
auth {type: bearer, token: ${LIBRARY_API_KEY}} block, substituted from
the Scheduler's own environment at execution time. The registrar no
longer resolves the real key client-side, so it can never be persisted
into the scheduled_tasks.config JSONB column. Also fixed: JSON bodies
moved from the ignored "body" key to "payload" (the executor only
reads config["payload"], so the tasks would have POSTed empty bodies
and failed required-user validation).
- Reranker: parsed ranking indices are deduplicated preserving first
occurrence (an LLM answer like "3,3,1" duplicated a result).
- HybridRAG wiring consolidated into dependencies.get_hybrid_rag_service
(now including volatile_service); the inline copies in /query/hybrid
and /wiki/pages/smart-create are gone - smart-create previously ran
without the volatile leg, and the singleton was unused.
- Remaining Neo4j writes (GraphService ingestion/deletes/purges/entity
mentions, webhook rename+delete cleanup, document-sync _index_graph,
consolidation mark-processed/add-entity) moved from auto-commit
execute_query to execute_write managed transactions with retry.
Verified end-to-end on the local dev server as llm_tester: /query/hybrid
200 with all five legs ok (volatile now active), background persistence
landed as one transaction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
Every vector call ran on the sync QdrantClient inside async wrapper
methods, blocking the FastAPI event loop per Qdrant round-trip. The
wrapper now holds an AsyncQdrantClient (timeout via QDRANT_TIMEOUT,
default 30s) and awaits all client calls; the wrapper API is unchanged.
Call sites off the wrapper were fixed too: the HybridRAG document leg
now uses the async search_vectors wrapper instead of the deprecated raw
client.search (also fixing its call to the nonexistent ollama.embed_text
which made the leg permanently report 'failed'), the health check awaits
get_collections, and document_sync's raw delete/upsert calls are awaited
(routed through wrappers in the next commit).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
Replace the four stub endpoints with real implementations, all requiring
an explicit tenant user (Phase B rule):
- /ingest/check-updates: GraphService now records a SHA-256 content_hash
on every Document node at ingestion time; the endpoint compares those
stored hashes against current Wiki.js page content in one UNWIND Cypher
query per tenant and returns changed/new/deleted page lists (entity-stub
pages excluded, pre-hash-tracking documents flagged stored_hash_missing).
- /ingest/status/{job_id}: backed by the Redis JobManager; jobs are
tenant-scoped (foreign jobs 404). /ingest/page, /ingest/batch and
/ingest/all now create job records and return job_id.
- /ingest/repo-status/{repository}: wiki page count vs indexed Document
nodes under users/{tenant}/{repository} plus tenant job stats.
- /deduplicate/check: tenant-scoped Qdrant similarity scan; chunk pairs
above ~0.9 cosine from different pages grouped per page pair with best
score and page references (read-only).
Supporting changes: get_job_manager dependency (+ shutdown close),
scroll_all_points can return vectors, VectorService.find_duplicate_pairs,
src/core/hashing.compute_content_hash. 13 new offline unit tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
Remove the implicit jpmschweitzer default tenant (DEFAULT_USER) from
src/core/multi_tenancy.py and every endpoint and request model that
inherited it (~40 endpoints across /query, /wiki, /vector, /graph,
/ingest, /volatile, /documents, /stats, /rag).
- Add validate_required_user() + RequiredUser pydantic type in
multi_tenancy and a shared require_user FastAPI dependency
(RequiredUserQuery) that rejects missing, empty, and whitespace-only
users with 422, following the /maintenance/* pattern.
- Wiki page create / smart-create / dossier request models now require
user (no fallback in wiki_service).
- /maintenance/cleanup/test-data derives the tenant from the page path
instead of using the production tenant collection.
- Wiki.js change listener skips changes when no tenant user can be
derived from the notification email instead of defaulting to the
production tenant.
- Consolidation service internal helpers no longer default to the
production tenant.
- Tool catalog marks user as required with honest descriptions.
- OpenAPI descriptions updated honestly; CHANGELOG notes that callers
(tatlock, Scheduler ingest tasks) must now send explicit user.
- Offline tests: 422 coverage for query/body endpoints, required-user
validator tests; updated legacy tests that assumed a default tenant.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add SchedulerClient to communicate with external scheduler service for
registering volatile prefetch tasks discovered during HybridRAG searches.
- Add scheduler_client.py with full REST API for task CRUD operations
- Add scheduler_url config setting (default: http://scheduler:8090)
- Update consolidation service to use scheduler for prefetch registration
- Add scheduler health checks to startup/shutdown lifecycle
When HybridRAG classifies web content as prefetch-worthy, it now creates
scheduled tasks that periodically refresh the volatile cache via the
external scheduler service.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add MemoryRouteClassification and MemoryRoutingResult models
- Implement unified classifier (_classify_web_results_unified) that routes
web results to: wiki, volatile, file (Paperless), prefetch, or skip
- Add routing methods: _route_to_volatile, _route_to_files, _register_prefetch
- Update _process_search to use unified classifier instead of separate analysis
- Add get_volatile_cache_service factory to dependencies
- Wire volatile_service and settings_client into ConsolidationService
- Update ConsolidationResult/Response with new routing counters
Test fixes:
- Fix WikiJSClient fixtures to use api_token instead of username/password
- Fix entity linking test assertions to expect full user-namespaced paths
- Add sample_unified_classification fixture for new classifier format
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add /documents router with webhook, upload, search, health endpoints
- Create DocumentSyncService for indexing documents to vectors/graph
- Add PaperlessClient for REST API integration
- Configure dependency injection for Paperless client
- Add document models for webhook payloads and responses
- Event-driven architecture via Paperless workflow webhooks
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Complete maintenance subsystem for index health and cleanup:
Endpoints:
- GET /maintenance/health - lightweight (or detailed) health check
- POST /maintenance/cleanup/all - full orphan cleanup
- POST /maintenance/cleanup/vectors - purge orphan vector chunks
- POST /maintenance/cleanup/graph - purge orphan graph nodes
- POST /maintenance/reconcile-index - cleanup + reindex missing pages
Bidirectional orphan detection:
- find_documents_without_vectors() in GraphService
- find_chunks_without_graph_nodes() in VectorService
Redis integration:
- Tracks last_cleanup timestamp for scheduler visibility
Config additions:
- Document store, volatile cache, and maintenance settings
- VectorServiceDep and GraphServiceDep type aliases
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add OLLAMA_EMBEDDING_MODEL for embeddings (nomic-embed-text)
- OLLAMA_MODEL now used for all LLM operations (mistral-nemo-large:latest)
- Remove separate reranker_model setting
- Update WikiPageWriter to use settings instead of hardcoded model
- Improves VRAM efficiency by keeping one model hot
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add /rag/search endpoint for web, news, and image search via SearXNG
- Add /content/extract and /content/extract/batch endpoints
- Add ContentExtractor client using Trafilatura for content extraction
- Enhance HybridRAG web search with full content extraction
- Add Redis caching for search results
- Add new configuration options for search and extraction timeouts
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
