library-desk/CHANGELOG.md

Changelog

All notable changes to Library Desk will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Fixed

• The Wiki.js change listener now survives a database restart. It held a single LISTEN connection with no supervision, so when the connection dropped it was gone permanently while running stayed True — the service kept reporting healthy and silently stopped indexing every page edit until someone restarted the container. This happened for real on 2026-08-08 when postgres-shared was redeployed. It now detects the drop via asyncpg's termination callback and reconnects with bounded exponential backoff (1s doubling to a 60s cap), retrying indefinitely because a database in maintenance does come back and giving up would recreate the same silent deafness.
• WikiChangeListener.running is derived from the live connection instead of being assigned once at startup, so it can no longer claim a subscription that does not exist.

Added

• /health reports the change listener under services.wiki_listener (subscribed, reconnects, last_gap_seconds). Nothing previously exposed its state anywhere, which is why a dead listener went unnoticed. It is deliberately excluded from the overall healthy/degraded verdict: it recovers on its own, and flipping the container unhealthy for the duration of a database outage would add a restart loop to an incident rather than information.
• On reconnect the listener logs the outage duration and warns that NOTIFY events emitted during the gap were lost and cannot be replayed, pointing at POST /maintenance/integrity-check to reconcile. A reconciliation pass is not performed automatically — tenant attribution for a changed page is non-trivial here, and guessing it wrong writes content into the wrong user's namespace.

[1.9.0] - 2026-07-20

Security

• The Wiki.js integration buttons no longer embed an API key in the browser. The re-index and entity-link endpoints now authenticate via the NPM /library-desk/ proxy (Authentik session for external users, LAN bypass for internal), verified by a trusted proxy marker header. The previously-embedded key was a full-privilege key served to every wiki visitor; it has been rotated out of service.
• verify_api_key now uses a constant-time comparison.

Changed

• static/wikijs-integration.js calls library-desk same-origin (/library-desk/...) with credentials: same-origin and no Authorization header. Update the Wiki.js code-injection snippet to <script src="/library-desk/static/wikijs-integration.js">.
• Machine callers (the Scheduler) continue to use the Bearer API key against the container-network endpoints; only the two browser endpoints switched to proxy auth.

[1.8.1] - 2026-07-19

Changed

• Container images are now pushed via the git.schweitz.net registry endpoint (the .internal registry domain is being retired); no change to the image name consumed by Watchtower.
• .env.example service URL defaults now use docker-dataplane container names (wiki:3000, neo4j:7687, searxng:8080, paperless:8000, ollama:11434) instead of host IP + published port, ahead of the Phase 4 port lockdown; AGENTS.md deploy health-check URL corrected from port 8000 to 8089.

[1.8.0] - 2026-07-14

Added

• Stub endpoints implemented (/ingest/check-updates, /ingest/status/{job_id}, /ingest/repo-status/{repository}, /deduplicate/check) — all previously returned canned "not yet implemented" responses; all now require an explicit user (Phase B rule):
  • /ingest/check-updates compares the content_hash now recorded on the tenant's Neo4j Document nodes at ingestion time against the SHA-256 of current Wiki.js page content in a single UNWIND Cypher query, returning changed / new / deleted page lists (auto-generated entity stubs excluded; documents whose stored hash predates hash tracking are flagged stored_hash_missing and count as changed).
  • /ingest/status/{job_id} is backed by the Redis JobManager (jobs are tenant-scoped; other tenants' jobs return 404). /ingest/page, /ingest/batch and /ingest/all now record job entries and return a job_id.
  • /ingest/repo-status/{repository} reports wiki page count vs indexed Document-node count under users/{tenant}/{repository} plus the tenant's Redis job statistics.
  • /deduplicate/check runs a tenant-scoped Qdrant similarity scan: wiki chunk pairs above the threshold (default 0.9 cosine) grouped per page pair with best score, matching chunk-pair count, and page references. Read-only.
• Job + Scheduler task plumbing — In-process hourly job_cleanup_loop (started at app startup, cancelled at shutdown) reclaims expired Redis job-set memberships (JobManager.cleanup_expired_jobs). docs/scheduler-tasks.md defines the four production Scheduler task payloads for the deploy checklist (nightly integrity 04:30, weekly quality report Sunday 03:00, daily Paperless orphan-cleanup 05:00 on the existing endpoint, and disabling test_example_task) with exact HTTP bodies (explicit user=jpmschweitzer, ${LIBRARY_API_KEY} auth placeholder). scripts/register_scheduler_tasks.py reads the Scheduler API location from SCHEDULER_URL and registers them — dry-run by default (prints payloads), --execute gated and requiring LIBRARY_API_KEY.
• Weekly quality report — POST /maintenance/quality-report {user} runs the duplicate scan, flags stale pages (not updated in N days AND ≤ M SearchQuery hits from the graph data), lists pages missing tags/description, folds in the latest integrity-check results (Redis-cached or run inline), and writes a dated report page to users/{user}/system/quality-reports/YYYY-MM-DD (same-day reruns update the same page — the page id is remembered in Redis because the Wiki.js listing lags page creation). Response returns the full report content + page path. Verified end-to-end against the local dev server as llm_tester.
• Nightly integrity check — POST /maintenance/integrity-check {user} (read-only: reports, never auto-fixes) reports per tenant: wiki pages with ZERO vectors in Qdrant (silent-skip reindex victims), orphaned vectors whose wiki page no longer exists, unexpected Qdrant collections (test-tenant residue and unknown namespaces flagged; other services' collections counted as foreign), Neo4j Document nodes without wiki counterparts, plus counts and duration. The latest report is cached in Redis (30 days) so the weekly quality report can fold it in.

Fixed (Phase D review batch)

• Runtime prefetch registration is executable - SchedulerClient.register_volatile_fetch (used by consolidation's _register_prefetch) registered tasks that were dead on arrival three ways: the JSON body sat under the ignored body key (rest_api_executor only reads config["payload"]), there was no auth block (the scheduled POST would 401 against library-desk's verify_api_key), and user was in the body while every /volatile/fetch endpoint requires it as a QUERY parameter (would 422). The config now puts user in the URL query string (URL-encoded), an empty payload, and auth: {type: bearer, token: "${LIBRARY_API_KEY}"} (substituted from the Scheduler's environment; never stored raw). SchedulerClient itself also sent no Authorization to the Scheduler API, so registration failed silently at consolidation time — it now sends Authorization: Bearer <SCHEDULER_API_KEY> (new scheduler_api_key setting; a client without a key logs a warning).
• Document sync uses delete-last reindex order - DocumentSyncService._index_vectors deleted the document's existing chunks (delete_by_filter) BEFORE embedding, so a failed embedding pass (e.g. Ollama down) left the Paperless document with zero vectors until the next successful sync — the exact hazard already fixed for wiki pages in VectorService.update_from_page. Chunk point ids are now deterministic (uuid5 of document_{id}_chunk_{i}, replacing random uuid4) so re-upserting overwrites in place; new points are upserted first and stale points (including legacy uuid4 ones) are pruned afterwards, only after a successful upsert.
• 5MB response cap now aborts the download - ContentExtractor._fetch buffered the entire response body in memory before truncating to MAX_RESPONSE_BYTES, so the cap protected Trafilatura but not memory/bandwidth (a multi-hundred-MB URL was still fully downloaded, on up to max_urls_per_batch concurrent fetches). Fetches are now streamed (client.stream + aiter_bytes) and the connection is closed as soon as the cap is reached.
• Registrar authenticates to the Scheduler API - scripts/register_scheduler_tasks.py --execute sent no Authorization header while the Scheduler's task-management endpoints are Bearer-guarded (verify_api_key: 401 on missing key), so the existence probes 401'd (misread as "task absent") and every registration failed; only the public /health gate passed. --execute now requires SCHEDULER_API_KEY in the environment (refuses to run without it, key is never stored) and sends Authorization: Bearer $SCHEDULER_API_KEY on all of its own HTTP calls. Deploy note updated alongside the existing LIBRARY_API_KEY requirement.

Fixed (hazards batch)

• CORS wildcard + credentials removed - allow_origins=["*"] combined with allow_credentials=True told browsers to attach credentials for any site. Credentials are now disabled (all real callers are server-to-server and use the Authorization header, which wildcard-origin CORS without credentials still permits) and the origin list is configurable via CORS_ALLOW_ORIGINS (comma-separated, default *).
• Scheduler task auth no longer stores raw tokens - the Phase C Scheduler task definitions put Authorization: Bearer ${LIBRARY_API_KEY} in plain headers and the registrar substituted the REAL key client-side on --execute, which would persist it in the Scheduler's scheduled_tasks.config JSONB column — and the Scheduler's rest_api_executor does not substitute env vars in plain headers anyway (only in url/payload/auth). The definitions now use the executor's auth: {type: bearer, token: "${LIBRARY_API_KEY}"} block, substituted from the SCHEDULER's environment at execution time; the registrar sends the placeholder verbatim and no longer needs (or accepts) the key. Also fixed: the JSON body moved from the ignored body key to payload (the executor only reads config["payload"], so the tasks would have POSTed empty bodies and failed Phase B user validation).
• Reranker index parser dedupes - an LLM ranking answer like 3,3,1 inserted the same result into the final ranking twice; parsed indices are now deduplicated preserving first occurrence.
• Single HybridRAG wiring point - three separate constructions existed: an unused dependencies.get_hybrid_rag_service singleton lacking volatile_service, an inline per-request copy in the /query/hybrid router, and another inline copy in /wiki/pages/smart-create also lacking volatile_service. All callers now use the dependencies singleton, which includes volatile_service (smart-create research can now hit the volatile cache leg).
• Hot-path Neo4j writes use write transactions - remaining graph writes ran as auto-commit execute_query calls (no retry, no transaction-function semantics): GraphService ingestion (update_from_page document + entity queries), delete_page, create_entity_mentions, document/paperless/collection node deletion, orphan-entity purge, stale-document purge, cleanup_broken_relationships; the webhook rename/delete cleanup writes; document-sync _index_graph; and consolidation's _mark_search_processed/_add_entity_to_graph. All now go through execute_write (managed transaction with driver retry). Read paths are unchanged.

Changed (performance)

• Content extractor hardened - ContentExtractor now downloads pages with httpx.AsyncClient under real connect (3s) and read timeouts on the event loop; only the CPU-bound Trafilatura parse runs in the thread pool. Previously trafilatura.fetch_url ran inside the worker thread with no caller-side timeout control, so an asyncio.wait_for timeout abandoned the thread while it kept downloading for up to ~30s. Trafilatura now parses each document ONCE via bare_extraction (text + metadata together) — the old code ran extract() twice (the XML pass was computed and discarded) plus bare_extraction, three full parses per page. extract_batch caps full-page extractions per call (default 8; overflow URLs return unsuccessful so the web leg falls back to the search snippet), responses are capped at 5MB before parsing, and thread-pool queue depth is logged for backpressure visibility.
• Top-k enrichment with one batched lookup - Phase 3 (_enrich_with_related_dossiers) ran a sequential Neo4j round-trip for EVERY fused result and the final trim then discarded most of the output. It now enriches only the results that can still reach the response (the Phase 4 rerank slice of 20 when reranking is enabled, otherwise final_result_count) and resolves all of them in ONE UNWIND-batched Cypher query (GraphService.get_related_documents_batch, tenant-scoped like the single-page variant, per-page ordering/limit preserved). Unenriched tail results carry an empty related_dossiers list as before.
• Search persistence off the hot path, one atomic transaction - HybridRAG Phase 6 (_persist_search_for_librarian) no longer gates the /query/hybrid response: the search_id is generated up front and returned immediately while the Neo4j write runs as a background task (strong task references held so tasks are not GC'd mid-flight). The write itself collapsed from ~21+ sequential auto-commit queries (SearchQuery node + per-document FOUND links + per-web-result WebResult nodes) into ONE UNWIND-based execute_write transaction, so a mid-way failure can no longer leave a partial SearchQuery graph behind. The persisted shape (SearchQuery properties incl. processed: false, tenant labels, FOUND relationship properties, WebResult properties) is unchanged and pinned by tests/test_search_persistence.py against exactly what the consolidation service queries. timing.persistence_ms now reports 0 (no longer on the request path).
• Batched embeddings + delete-last reindex - OllamaClient.embed_batch now sends ONE batched /api/embed request (verified against the live Ollama; the old "batch" looped one /api/embeddings call per chunk) with a per-text fallback preserving partial-success semantics. VectorService.update_from_page embeds all chunks in that single call and upserts them in one Qdrant batch, and the reindex order is reversed: new points are upserted BEFORE stale points are pruned (deterministic uuid5 chunk ids make the overwrite safe), so a mid-way failure can no longer leave a page with zero vectors — the old order deleted everything first. The summary now reports status (success/partial/failed) and chunks_skipped instead of unconditional success=True; a fully failed embedding pass keeps the old vectors and reports failure. Measured on a real 7-chunk page ingest as llm_tester against the local server: ~375ms → ~181ms median (3 runs each).
• Document sync indexing fixed - DocumentSyncService._index_vectors now awaits ensure_collection (the coroutine was created but never ran, so fresh tenants had no collection at upsert time), filters out None entries from embed_batch so one failed chunk embedding no longer aborts the whole document upsert (all-failed still reports failure), and routes the raw client.delete/client.upsert calls through the async wrapper (delete_by_filter / new batch upsert_points). Offline unit tests added.
• Async Qdrant client - QdrantClientWrapper now uses AsyncQdrantClient with an explicit timeout (QDRANT_TIMEOUT, default 30s). Every vector call previously ran on the synchronous client inside async wrapper methods, blocking the FastAPI event loop for the duration of each Qdrant round-trip. The wrapper API is unchanged (all methods were already async), so call sites only gained real awaits. The HybridRAG document leg was moved off the deprecated raw client.search onto the wrapper's search_vectors (fixing a latent AttributeError: it called the nonexistent ollama.embed_text, so the leg always reported failed), and the health check awaits get_collections.

Changed

• BREAKING: user is now required on every tenant-data endpoint - The implicit jpmschweitzer default tenant (DEFAULT_USER) has been removed everywhere. All endpoints that read or write tenant data (/query/*, /wiki/*, /vector/*, /graph/*, /ingest/*, /volatile/*, /documents/*, /stats, /rag/search) now reject requests without an explicit, non-empty, non-whitespace user (HTTP 422), matching the existing /maintenance/* pattern. A shared validator (require_user dependency / RequiredUser model type) also rejects blank users. The Wiki.js change listener now skips changes whose notification email yields no user instead of attributing them to the production tenant. Caller coordination required: tatlock and the Scheduler ingest/prefetch/consolidation tasks must send an explicit user on every call — see the deploy checklist.

Fixed (security)

• Cross-tenant leaks in HybridRAG legs and ingestion closed - A live probe as user=llm_tester returned jpmschweitzer pages. Root causes fixed:

  • Ingestion namespace enforcement: vector and graph update_from_page now refuse pages whose wiki path is outside users/{user}/ (previously any tenant could ingest any page id — including another tenant's — into its own collection/labels, which is how foreign content ended up in the vector leg). /ingest/all clamps path_prefix to the caller's namespace (400 on cross-tenant prefixes) and defaults to users/{user}.
  • Search persistence: the FOUND link in HybridRAG phase 6 matched (d:Document {page_id}) unscoped, attaching the caller's SearchQuery to other tenants' Document nodes; it now matches only User_{Tenant}_Document nodes.
  • Graph enrichment/consolidation queries scoped: _get_entity_mention_count, entity-stub generation, orphan-entity find/purge, and cleanup_broken_relationships matched unscoped Document/SearchQuery nodes; all now use the tenant's labels. Entity-page existence checks list only the tenant's wiki namespace.
  • Volatile collections sanitized: volatile_{user} collection names now use the sanitized user id (same scheme as document collections).
  • Namespace matching hardened: is_path_in_user_namespace now enforces a path-segment boundary (users/llm_tester2 is no longer inside llm_tester's namespace) and compares sanitized tenant segments.
  • Offline unit tests added per leg (vector, graph, volatile, documents, enrichment, persistence, ingestion) asserting the tenant-scoped collection/label/path is used.

• /query/graph and /graph/query hardened to read-only - The documented "automatic user scoping" was a no-op (a live probe confirmed any user string could read the whole graph) and the client permitted writes. Raw Cypher queries are now (1) rejected with 400 when they contain write clauses (CREATE/MERGE/DELETE/DETACH/SET/REMOVE/DROP/FOREACH/LOAD CSV) or any CALL procedure (conservative denylist on the uppercased query), and (2) executed through a Neo4j session opened with default_access_mode=READ_ACCESS so the database itself refuses writes as a backstop. The endpoints are now honestly documented as admin/debug, unscoped read-only: results are not restricted to the caller's tenant labels — use /graph/nodes for tenant-scoped access.

Added

• Degradation signaling - HybridRAGResponse now includes source_status (per-leg 'ok'/'failed'/'disabled' for vector, graph, web, volatile, documents) and degraded (true when any enabled leg failed). Retrieval legs report errors instead of silently swallowing them; failed legs are logged at WARNING. Both fields are additive and optional, so clients that ignore them are unaffected.
• Hard-isolated test-tenant lifecycle for the test suite - tests/conftest.py rewritten: the production host default (192.168.86.149) is gone (TEST_HOST env, safe localhost default; the API under test is the local wakeup server via LIBRARY_DESK_URL, never the production container on 8089). The suite is pinned to the reserved test tenant llm_tester; a session guard aborts the entire run if the effective tenant is jpmschweitzer or outside the reserved llm_tester* namespace. Integration tests are marked and only run with RUN_INTEGRATION_TESTS=1 (plus a passing guard). A session-scoped teardown deletes ALL llm_tester artifacts created during the run — Qdrant *_llm_tester collections, Neo4j User_Llm_Tester*-labelled nodes, the users/llm_tester wiki subtree, and llm_tester Redis keys on the service DB — with hard tenant assertions before every delete. Legacy integration tests were pinned to the test tenant (no more production-namespace reads).
• Live tenant-isolation test - tests/test_tenant_isolation_live.py (integration-marked, guard-gated): creates and ingests a wiki page as llm_tester against the local wakeup server + shared services, asserts /query/hybrid as llm_tester retrieves its own content with ZERO results from the jpmschweitzer tenant, and asserts a third nonexistent tenant (llm_tester_void, still inside the reserved namespace — nothing is ever written as jpmschweitzer) gets zero results entirely. Teardown removes everything it created.
• Test-residue purge tooling - scripts/purge_test_artifacts.py: dry-run by DEFAULT (--execute required for real deletion), targets only confirmed test residue (Qdrant library_desk_llm_tester / test_user / library_desk_test_user / memories_llm_tester / volatile_llm_tester / core_ai_user_test_* / anything containing llm_tester; Neo4j User_Llm_Tester*-labelled nodes and legacy users/llm* Document nodes; llm_tester Redis keys on the service DB), prints counts per target, hard-aborts if any target rule ever matches a jpmschweitzer-namespaced identifier, and documents the snapshot prerequisite (Qdrant snapshot API + neo4j-admin database dump) in its docstring.
• Offline unit tests - New mock-based tests (no live services) for model-name resolution under the env collision, per-leg failure signaling, the /stats page-count prefix, LLM-call timeouts, and Wiki.js listing pagination.

Fixed

• Ollama generation model env collision - Renamed the generation-model setting ollama_model to ollama_llm_model (env: OLLAMA_LLM_MODEL, default gemma4:e2b). The container env OLLAMA_MODEL=nomic-embed-text (meant for embeddings) was shadowing the generation model, breaking Phase 0 keyword extraction and Phase 4 LLM re-ranking on every request. Startup now logs the resolved generation model.
• LLM call timeouts - Phase 0 keyword extraction and Phase 4 re-ranking are wrapped in a 12s asyncio.wait_for with graceful fallback, so a hung Ollama call can no longer gate retrieval for the full 120s client timeout.
• /stats wiki page count - The endpoint passed the bare user name as path prefix (matching nothing) and always reported 0 pages; it now counts pages under users/{user}.
• Wiki.js page listing - list_pages applied the API-side limit before client-side path/tag filters, dropping matching pages that sort late; the limit now applies after filtering. list_all_pages replaced its fake pagination loop with a real limit-growth loop (Wiki.js 2.x pages.list has no offset argument) that fetches until the API returns fewer pages than requested.
• Consolidation loop silently drained its queue on LLM failure - Root cause of the 30-minute knowledge-consolidation loop processing 0 searches ("No unprocessed searches found" in prod): the OLLAMA_MODEL env collision (see below) made every consolidation /api/generate call fail with HTTP 400 ("nomic-embed-text" does not support generate — confirmed in prod logs and by direct Ollama probe), classification returned empty, and the loop STILL marked every SearchQuery processed: true — permanently consuming the queue with zero pages ever created (live Neo4j: 197/200 SearchQuery nodes processed with no output). LLM-infrastructure failure now raises ConsolidationLLMUnavailableError: the affected searches stay unprocessed (retried next run), the batch aborts after the first failure, and the response reports searches_deferred. Every run now logs searches_processed and duration_ms (also new response fields). The lookback boundary is now timezone-aware UTC. Regression tests added.
• Wiki.js update_page without tags - Wiki.js 2.x requires tags on the update mutation (the server unconditionally maps over it); every update_page(page_id, content=...) call without tags failed with Cannot read properties of undefined (reading 'map') — this silently broke the consolidation service's page-update path too. The client now preserves the page's current tags when the caller does not supply any.
• Wiki.js listing completeness under pre-filter limits - Observed live: pages.list(limit=100) returned 43 pages while 140 existed (limit=500 returned all) — Wiki.js applies the limit BEFORE its own visibility filtering, so "fewer pages than requested" does not mean the listing is complete and the limit-growth loop stopped early, silently truncating listings (page counts, cleanups, integrity scans). The loop now grows the limit until the returned count stops increasing (fixed point), at the cost of one confirming fetch.

[1.7.3] - 2026-01-07

Fixed

• Endpoint TTL defaults - Updated all fetch endpoint defaults to match namespace TTLs (2x refresh interval)

[1.7.2] - 2026-01-07

Fixed

• Volatile TTL doubled - TTL now 2x refresh interval to survive missed/delayed scheduler runs

[1.7.1] - 2026-01-07

Fixed

• CI workflow - Updated Gitea Actions to trigger on tag push (matching core-api)

[1.7.0] - 2026-01-07

Added

• Combined Environment Endpoint - POST /volatile/fetch/environment/{city}
  • Fetches weather and air quality concurrently with asyncio.gather()
  • Single geocode lookup shared between both API calls
  • More efficient than calling weather and air_quality separately
  • Reduces wall-clock time and eliminates redundant geocoding

Fixed

• Scheduler executor name - Fixed rest_api → rest_api_executor in SchedulerTask model and register_volatile_fetch() to prevent "Executor module not found" errors

[1.6.2] - 2025-12-30

Added

• System Statistics Endpoint - GET /stats

  • Neo4j: node counts by type (Document, Entity, Collection, Search)
  • Qdrant: collection counts, total vectors, per-collection breakdown
  • Wiki.js: total page count
  • Paperless: documents, tags, correspondents, document types

• Weather/Forecast Separation - Split weather into two distinct namespaces

  • POST /volatile/fetch/weather/{city} - Current conditions only (1hr TTL)
  • POST /volatile/fetch/forecast/{city} - 7-day outlook (12hr TTL)
  • Different update frequencies for efficient caching
  • FORECAST namespace added to volatile namespaces

Changed

• Weather namespace TTL changed from 30 minutes to 1 hour (current conditions)
• Forecast data now stored separately with 12 hour TTL

[1.6.1] - 2025-12-30

Added

• Weather Forecast Support - Enhanced weather fetch with 7-day daily forecasts
  • Current conditions now include UV index
  • Daily forecasts with high/low temps, conditions, precipitation chance, UV max
  • Natural language text summary with multi-day outlook
• Sun Times Endpoint - POST /volatile/fetch/sun/{city}
  • Sunrise and sunset times (HH:MM and ISO formats)
  • Daylight duration in seconds and hours
  • Separate volatile namespace with 24hr TTL
  • Useful for home automation light triggers
• Air Quality Endpoint - POST /volatile/fetch/air_quality/{city}
  • European and US AQI indices
  • Pollutants: PM2.5, PM10, ozone, nitrogen dioxide, sulphur dioxide, carbon monoxide
  • Pollen data (grass, birch, alder) for European locations (seasonal)
  • Hourly refresh (1hr TTL)
• New Base Models
  • SunTimes dataclass for sunrise/sunset data
  • AirQuality dataclass with AQI and pollutants
  • AirQualityProvider abstract interface
• New Volatile Namespace - SUN for sunrise/sunset times (86400s default TTL)

Changed

• Weather fetch now uses get_forecast() instead of get_current() for richer data
• OpenMeteoProvider now implements both WeatherProvider and AirQualityProvider

[1.6.0] - 2025-12-29

Added

• Memory System Implementation - Complete three-tier memory architecture

  • Volatile Fetch Endpoints - Scheduler-driven prefetch for ephemeral data
    • POST /volatile/fetch/{namespace}/{key} - Fetch and cache external data
    • Weather, news, and financial data providers integrated
    • Auto-caching with namespace-specific TTLs
  • Unified Memory Routing - LLM-based classification of web results
    • Routes content to wiki (stable), volatile (ephemeral), file (documents), or prefetch (scheduled)
    • Integrated into consolidation service post-processor
  • Document Recall in HybridRAG - Paperless documents as fourth retrieval source
    • Documents searched alongside wiki, volatile, and web in parallel
    • New config: enable_documents, document_limit, document_threshold
    • paperless_id field in results for document attribution
    • document_ms timing in performance breakdown

• Scheduler Integration - External scheduler service for prefetch task management

  • SchedulerClient - Full REST API client for task CRUD operations
  • register_volatile_fetch() convenience method for prefetch registration
  • Consolidation service now creates scheduled tasks for prefetch-worthy content
  • Health checks integrated into startup/shutdown lifecycle

Changed

• HybridRAG now searches 4 sources in parallel (wiki, volatile, documents, web)
• Consolidation service uses external scheduler instead of settings storage for prefetch

[1.5.0] - 2025-12-26

Added

• Central Settings Database - Tatlock-wide configuration via PostgreSQL

  • SettingsClient for async access to system_settings database
  • User-scoped settings with global fallback
  • API config storage with enabled toggle and per-source category filters
  • JSON Schema support for future UI rendering

• External API Providers - Modular src/apis/ package with swappable implementations

  • OpenMeteoProvider - Weather with geocoding (free, no API key)
  • NOSProvider - Dutch news RSS (16 categories including sports)
  • BBCProvider - English news RSS (21 categories including sports)
  • AggregatedNewsProvider - Merges sources chronologically with category filtering
  • AlphaVantageProvider - Stock/crypto quotes (API key from settings DB)
  • Abstract base classes for provider interoperability

• Provider Dependency Injection

  • WeatherProviderDep, NewsProviderDep, AlphaVantageProviderDep type aliases
  • Async initialization with settings database integration
  • Lifecycle management in shutdown_clients()

• Development Dependencies - requirements-dev.txt

  • pip-audit for security vulnerability scanning
  • ruff for code quality
  • Testing packages moved from main requirements

Changed

• News sources configurable via news.sources setting
• Per-source category filtering via api.{source}.categories
• Categories default to all if not specified

[1.4.8] - 2025-12-25

Added

• Paperless Orphan Cleanup - POST /maintenance/cleanup/paperless endpoint
  • Detects documents deleted from Paperless but still indexed in Library Desk
  • Removes orphaned vectors and graph nodes
  • Supports dry_run=true for preview mode

[1.4.7] - 2025-12-25

Fixed

• Paperless Custom Field Update - Fixed 400 error when marking documents as indexed
  • Paperless API requires field ID (integer) not field name (string)
  • Now looks up library_indexed field ID before updating
  • Webhook params format: doc_url and title from Jinja templates

Added

• Webhook Debug Endpoint - POST /documents/webhook-capture for development testing

[1.4.6] - 2025-12-25

Fixed

• Paperless Webhook Payload Format - Updated model to match Paperless include_document=true format
  • Paperless sends id instead of document_id
  • Paperless sends full document data including content, title, tags, etc.
  • Webhook now uses content from payload, skipping extra Paperless API call
  • Added extra = "ignore" to handle additional Paperless fields

[1.4.5] - 2025-12-25

Added

• Document Storage Integration - Paperless-ngx integration for PDFs, images, and documents
  • Event-driven architecture via Paperless webhooks
  • POST /documents/webhook - Receive document events from Paperless workflows
  • POST /documents/upload - Upload files directly to Paperless
  • POST /documents/upload-url - Download and upload documents from URL
  • POST /documents/search - Semantic search across indexed documents
  • GET /documents/health - Paperless connectivity health check
• DocumentSyncService - Indexes Paperless documents into vectors and graph
  • Fetches document content via Paperless API
  • Chunks text and generates embeddings for Qdrant
  • Creates Document nodes in Neo4j knowledge graph
  • Supports multi-tenancy via user parameter in webhook URL
• PaperlessClient - REST API client for Paperless-ngx
  • Document retrieval, upload, and update operations
  • Health check support
• Paperless Workflow Configuration
  • Production workflow: Document Added (NOT tagged llm-test) → webhook to Library Desk
  • Test workflow: Document Added (tagged llm-test) → webhook with test user

Changed

• Updated src/config.py with Paperless configuration settings
• Added PaperlessDep dependency injection for document endpoints

[1.4.4] - 2025-12-24

Added

• Test Data Cleanup Endpoint - POST /maintenance/cleanup/test-data
  • Purges LLM test data from wiki, graph, and vectors
  • Security-restricted to test user namespace only (users/llm-tester/*, users/llm_tester/*)
  • Supports dry_run=true (default) to preview before deleting
  • Scheduler task configured for weekly cleanup (Sunday 3:00 AM)

[1.4.3] - 2025-12-24

Changed

• Volatile Cache System Refactored to Vector Storage
  • Backend migrated from Redis to Qdrant for semantic search capability
  • Data converted to natural language for embedding and semantic retrieval
  • Collection naming: volatile_{user} for per-user isolation
  • TTL implemented via ttl_expiry timestamp in vector payload
  • Simplified endpoints:
    • GET /volatile/search?q=... - Semantic search across volatile data
    • POST /volatile/store?namespace=...&key=... - Store with query params
    • GET /volatile/{namespace}/{key} - Get specific record
    • DELETE /volatile/{namespace}/{key} - Delete record
  • Removed namespace-specific URL patterns (simpler API for LLM tool use)

Added

• HybridRAG Volatile Integration - Volatile cache now included in multi-source search
  • Volatile results get priority boost in RRF fusion (current data ranks higher)
  • New config options: enable_volatile, volatile_limit (default 1), volatile_threshold
  • Timing breakdown includes volatile_ms
• Volatile Cleanup Endpoint - POST /maintenance/cleanup/volatile
  • Purges expired records across all volatile_* collections
  • Scheduler task for every 10 minutes recommended
  • Returns per-collection cleanup counts
• Natural Language Conversion - Structured data converted for embedding
  • Template-based conversion for each namespace (weather, news, financial, etc.)
  • Fallback for custom namespaces

[1.4.2] - 2025-12-24

Added

• Volatile Cache System - Ephemeral data storage with TTL
  • GET /volatile/{namespace}/{key} - Retrieve cached record
  • POST /volatile/{namespace}/{key} - Store/update record with TTL
  • DELETE /volatile/{namespace}/{key} - Remove record
  • GET /volatile/{namespace} - List keys in namespace
  • DELETE /volatile/{namespace} - Clear all records in namespace
  • GET /volatile/stats - Cache statistics by namespace
  • GET /volatile/scheduled - Records needing refresh (for scheduler)
  • GET /volatile/namespaces - List available namespaces with default TTLs
• Volatile Namespaces - Predefined categories with appropriate TTLs:
  • weather (30min) - Weather conditions and forecasts
  • news (1hr) - Headlines and breaking news
  • financial (5min) - Stock prices, exchange rates
  • transit (5min) - Train/bus schedules, delays
  • traffic (10min) - Commute times, road conditions
  • air_quality (1hr) - Pollution, pollen counts
  • sports (1min) - Live scores, matches
  • social (10min) - Social notifications
  • system (1min) - Service health status
  • context (1hr) - Session state
  • custom (1hr) - User-defined data
• Refresh Schedule Support - Optional cron expressions for scheduler integration

[1.4.1] - 2025-12-24

Fixed

• Wiki.js API token now optional - GraphQL API works without authentication
• Container startup failure when WIKI_GRAPHQL_API env var not set

[1.4.0] - 2025-12-24

Added

• Maintenance Router - New /maintenance endpoints for system health and cleanup
  • GET /maintenance/health - Lightweight health check (detailed mode available)
  • POST /maintenance/cleanup/all - Full orphan cleanup (vectors + graph)
  • POST /maintenance/cleanup/vectors - Purge orphan vector chunks
  • POST /maintenance/cleanup/graph - Purge orphan graph nodes
  • POST /maintenance/reconcile-index - Combined cleanup + reindex missing pages
• Bidirectional Orphan Detection - Cross-validate vectors and graph nodes
  • find_documents_without_vectors() - Graph nodes missing vector chunks
  • find_chunks_without_graph_nodes() - Vector chunks missing graph nodes
• Qdrant Client Methods - Bulk operations for maintenance
  • scroll_all_points() - Iterate all points with pagination
  • delete_by_ids() - Batch delete by point IDs
• Graph Service Cleanup - Node deletion methods
  • delete_document_node() - Remove document and relationships
  • delete_collection_node() - Remove collection and contained documents
  • get_all_document_references() - Get all document references for validation
• Redis Timestamp Tracking - last_cleanup timestamp for scheduler integration
• Memory System Plan - Documented three-tier architecture (volatile/documents/knowledge)

Changed

• Wiki.js Authentication - Switched from username/password to API token
  • New WIKI_GRAPHQL_API environment variable for JWT token
  • Deprecated WIKIJS_USERNAME and WIKIJS_PASSWORD (kept for backwards compatibility)
• Service Dependencies - Added VectorServiceDep and GraphServiceDep type aliases

Fixed

• Wiki.js client now properly handles API token auth without login flow

[1.3.3] - 2025-12-23

Added

• Temperature parameter to OllamaClient.generate_text() for controlling output determinism
• TODO.md tracking remaining stub endpoints to implement
• Wired /query/semantic endpoint to VectorService
• Wired /query/graph endpoint to GraphService

Changed

• Improved LLM prompts based on llm-findings.md recommendations:
  • Keyword extraction: temperature 0.0, negative constraints
  • LLM re-ranking: temperature 0.0, explicit rules
  • Conflict detection: temperature 0.0, analysis steps (CoT)
  • Wiki page creation: temperature 0.3, anti-hallucination constraints
  • Page reconstruction: temperature 0.2, preservation constraints
  • Web results analysis: temperature 0.0, conservative approach
• Test fixtures now use configurable host (TEST_HOST) instead of Docker hostnames

Removed

• Dead code: unused get_default_user() function
• Unused imports from routers (wiki.py, graph.py, hybrid_rag.py)
• Stub endpoints shadowed by real implementations (/stats, /ingest/document, /ingest/batch)

[1.3.2] - 2025-12-22

Changed

• Consolidated Ollama model configuration - All LLM operations now use single OLLAMA_MODEL environment variable
  • Removed separate reranker_model setting
  • HybridRAG re-ranking, consolidation analysis, and wiki page writing all use the same model
  • Improves VRAM efficiency by keeping one model hot
• Added OLLAMA_EMBEDDING_MODEL environment variable for embedding model (previously overloaded OLLAMA_MODEL)
• Updated WikiPageWriter to accept settings instead of hardcoded model name

[1.3.1] - 2025-12-16

Fixed

• Smart create endpoint missing content_extractor dependency causing 500 errors on POST /wiki/pages/smart-create

[1.3.0] - 2025-12-15

Changed

• Two-Stage RRF Architecture - Major refactor to level the playing field between wiki and web results

  • Stage 1: Vector and graph results merged into single "wiki" ranking using mini-RRF
  • Stage 2: Final RRF between wiki (single source) and web (single source)
  • Wiki pages no longer get 2x advantage from appearing in both vector and graph searches
  • Multi-source confirmation still determines wiki internal ranking

• Skip synonyms in graph search - LLM-generated synonyms (e.g., "author") no longer match unrelated graph entities (e.g., "author2000")

  • Vector search still uses synonyms for semantic similarity
  • Graph search uses only core keywords for exact entity matching

Added

• VECTOR_SIMILARITY_THRESHOLD config setting (default: 0.7) to filter weak vector matches
• Deduplication in graph search to prevent same document appearing multiple times

Fixed

• Graph search duplicate entity bug where same document could appear twice if entity linked multiple times

[1.2.1] - 2025-12-15

Fixed

• HybridRAG router missing content_extractor dependency causing 500 errors on /query/hybrid endpoint

[1.2.0] - 2025-12-15

Added

• RAG Search Endpoint (POST /rag/search)

  • Web, news, and image search via SearXNG
  • Full content extraction using Trafilatura (F1 score 0.958)
  • Redis caching with configurable TTL
  • Markdown sources summary for LLM consumption
  • Returns both extracted content and original snippets

• Content Extraction Endpoints (/content/*)

  • POST /content/extract - Extract content from a single URL
  • POST /content/extract/batch - Batch extraction (up to 20 URLs)
  • Reusable ContentExtractor client for use across the codebase

• HybridRAG Content Extraction Enhancement

  • Web search results now include full extracted content via Trafilatura
  • Falls back to original snippets if extraction fails
  • Improves context quality for LLM re-ranking and consumption

Changed

• Added new configuration options:
  • SEARCH_CACHE_TTL - Search cache TTL in seconds (default: 300)
  • SEARCH_TIMEOUT - SearXNG timeout (default: 10s)
  • CONTENT_EXTRACTION_TIMEOUT - Per-URL extraction timeout (default: 5s)
  • CONTENT_MAX_LENGTH - Max extracted content length (default: 2000)
  • SEARCH_DEFAULT_LIMIT - Default search results (default: 10)

Dependencies

• Added trafilatura~=1.12.0 for content extraction

[1.1.3] - 2025-12-14

Added

• Watchtower update trigger in Gitea workflow after successful build

[1.1.2] - 2025-12-14

Fixed

• Updated registry login URL in Gitea workflow (git.schweitz.net → git.schweitz.internal)

[1.1.1] - 2025-12-14

Fixed

• Updated container registry tag URLs in Gitea workflow (git.schweitz.net → git.schweitz.internal)

Added

• Tests for Smart Page Creation feature (test_smart_create.py)
  • Model validation tests for WikiSmartCreateRequest/Response
  • WikiService.smart_create_page method tests
  • Bidirectional entity linking utility tests
  • Endpoint validation tests

[1.1.0] - 2025-12-11

Added

• Smart Page Creation Endpoint (POST /wiki/pages/smart-create)

  • Combines HybridRAG research with LLM content generation
  • Searches existing wiki, knowledge graph, and web for topic context
  • Uses WikiPageWriter to synthesize findings into structured wiki content
  • Auto-generates page path from topic if not provided
  • Returns research summary with source counts

• Bidirectional Entity Linking

  • New shared utility (entity_linking_utils.py) for reusable entity linking
  • Forward links: Links entities mentioned in new pages to existing entity pages
  • Backward links: Updates existing pages that mention the new entity
  • Runs automatically in background after smart page creation

• Version Management

  • Added pyproject.toml with project metadata and version
  • Version is now read from pyproject.toml (single source of truth)
  • Health check endpoint returns current version
  • FastAPI docs show current version

Changed

• Updated config.py to read version from pyproject.toml
• Updated main.py to use centralized version

[1.0.0] - 2025-12-10

Added

• Initial release extracted from portainer-core
• Wiki page management (/wiki/pages CRUD endpoints)
• HybridRAG search (/query/hybrid) with vector, graph, and web search
• Knowledge graph operations (/graph/*)
• Vector search operations (/vector/*)
• Knowledge consolidation from search results (/consolidate/knowledge)
• Entity linking and extraction
• Wiki.js change listener for auto-processing user edits
• Multi-tenant architecture with user namespace isolation