library-desk/CHANGELOG.md

Changelog

All notable changes to Library Desk will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Changed

• BREAKING: user is now required on every tenant-data endpoint - The implicit jpmschweitzer default tenant (DEFAULT_USER) has been removed everywhere. All endpoints that read or write tenant data (/query/*, /wiki/*, /vector/*, /graph/*, /ingest/*, /volatile/*, /documents/*, /stats, /rag/search) now reject requests without an explicit, non-empty, non-whitespace user (HTTP 422), matching the existing /maintenance/* pattern. A shared validator (require_user dependency / RequiredUser model type) also rejects blank users. The Wiki.js change listener now skips changes whose notification email yields no user instead of attributing them to the production tenant. Caller coordination required: tatlock and the Scheduler ingest/prefetch/consolidation tasks must send an explicit user on every call — see the deploy checklist.

Added

• Degradation signaling - HybridRAGResponse now includes source_status (per-leg 'ok'/'failed'/'disabled' for vector, graph, web, volatile, documents) and degraded (true when any enabled leg failed). Retrieval legs report errors instead of silently swallowing them; failed legs are logged at WARNING. Both fields are additive and optional, so clients that ignore them are unaffected.
• Offline unit tests - New mock-based tests (no live services) for model-name resolution under the env collision, per-leg failure signaling, the /stats page-count prefix, LLM-call timeouts, and Wiki.js listing pagination.

Fixed

• Ollama generation model env collision - Renamed the generation-model setting ollama_model to ollama_llm_model (env: OLLAMA_LLM_MODEL, default gemma4:e2b). The container env OLLAMA_MODEL=nomic-embed-text (meant for embeddings) was shadowing the generation model, breaking Phase 0 keyword extraction and Phase 4 LLM re-ranking on every request. Startup now logs the resolved generation model.
• LLM call timeouts - Phase 0 keyword extraction and Phase 4 re-ranking are wrapped in a 12s asyncio.wait_for with graceful fallback, so a hung Ollama call can no longer gate retrieval for the full 120s client timeout.
• /stats wiki page count - The endpoint passed the bare user name as path prefix (matching nothing) and always reported 0 pages; it now counts pages under users/{user}.
• Wiki.js page listing - list_pages applied the API-side limit before client-side path/tag filters, dropping matching pages that sort late; the limit now applies after filtering. list_all_pages replaced its fake pagination loop with a real limit-growth loop (Wiki.js 2.x pages.list has no offset argument) that fetches until the API returns fewer pages than requested.

[1.7.3] - 2026-01-07

Fixed

• Endpoint TTL defaults - Updated all fetch endpoint defaults to match namespace TTLs (2x refresh interval)

[1.7.2] - 2026-01-07

Fixed

• Volatile TTL doubled - TTL now 2x refresh interval to survive missed/delayed scheduler runs

[1.7.1] - 2026-01-07

Fixed

• CI workflow - Updated Gitea Actions to trigger on tag push (matching core-api)

[1.7.0] - 2026-01-07

Added

• Combined Environment Endpoint - POST /volatile/fetch/environment/{city}
  • Fetches weather and air quality concurrently with asyncio.gather()
  • Single geocode lookup shared between both API calls
  • More efficient than calling weather and air_quality separately
  • Reduces wall-clock time and eliminates redundant geocoding

Fixed

• Scheduler executor name - Fixed rest_api → rest_api_executor in SchedulerTask model and register_volatile_fetch() to prevent "Executor module not found" errors

[1.6.2] - 2025-12-30

Added

• System Statistics Endpoint - GET /stats

  • Neo4j: node counts by type (Document, Entity, Collection, Search)
  • Qdrant: collection counts, total vectors, per-collection breakdown
  • Wiki.js: total page count
  • Paperless: documents, tags, correspondents, document types

• Weather/Forecast Separation - Split weather into two distinct namespaces

  • POST /volatile/fetch/weather/{city} - Current conditions only (1hr TTL)
  • POST /volatile/fetch/forecast/{city} - 7-day outlook (12hr TTL)
  • Different update frequencies for efficient caching
  • FORECAST namespace added to volatile namespaces

Changed

• Weather namespace TTL changed from 30 minutes to 1 hour (current conditions)
• Forecast data now stored separately with 12 hour TTL

[1.6.1] - 2025-12-30

Added

• Weather Forecast Support - Enhanced weather fetch with 7-day daily forecasts
  • Current conditions now include UV index
  • Daily forecasts with high/low temps, conditions, precipitation chance, UV max
  • Natural language text summary with multi-day outlook
• Sun Times Endpoint - POST /volatile/fetch/sun/{city}
  • Sunrise and sunset times (HH:MM and ISO formats)
  • Daylight duration in seconds and hours
  • Separate volatile namespace with 24hr TTL
  • Useful for home automation light triggers
• Air Quality Endpoint - POST /volatile/fetch/air_quality/{city}
  • European and US AQI indices
  • Pollutants: PM2.5, PM10, ozone, nitrogen dioxide, sulphur dioxide, carbon monoxide
  • Pollen data (grass, birch, alder) for European locations (seasonal)
  • Hourly refresh (1hr TTL)
• New Base Models
  • SunTimes dataclass for sunrise/sunset data
  • AirQuality dataclass with AQI and pollutants
  • AirQualityProvider abstract interface
• New Volatile Namespace - SUN for sunrise/sunset times (86400s default TTL)

Changed

• Weather fetch now uses get_forecast() instead of get_current() for richer data
• OpenMeteoProvider now implements both WeatherProvider and AirQualityProvider

[1.6.0] - 2025-12-29

Added

• Memory System Implementation - Complete three-tier memory architecture

  • Volatile Fetch Endpoints - Scheduler-driven prefetch for ephemeral data
    • POST /volatile/fetch/{namespace}/{key} - Fetch and cache external data
    • Weather, news, and financial data providers integrated
    • Auto-caching with namespace-specific TTLs
  • Unified Memory Routing - LLM-based classification of web results
    • Routes content to wiki (stable), volatile (ephemeral), file (documents), or prefetch (scheduled)
    • Integrated into consolidation service post-processor
  • Document Recall in HybridRAG - Paperless documents as fourth retrieval source
    • Documents searched alongside wiki, volatile, and web in parallel
    • New config: enable_documents, document_limit, document_threshold
    • paperless_id field in results for document attribution
    • document_ms timing in performance breakdown

• Scheduler Integration - External scheduler service for prefetch task management

  • SchedulerClient - Full REST API client for task CRUD operations
  • register_volatile_fetch() convenience method for prefetch registration
  • Consolidation service now creates scheduled tasks for prefetch-worthy content
  • Health checks integrated into startup/shutdown lifecycle

Changed

• HybridRAG now searches 4 sources in parallel (wiki, volatile, documents, web)
• Consolidation service uses external scheduler instead of settings storage for prefetch

[1.5.0] - 2025-12-26

Added

• Central Settings Database - Tatlock-wide configuration via PostgreSQL

  • SettingsClient for async access to system_settings database
  • User-scoped settings with global fallback
  • API config storage with enabled toggle and per-source category filters
  • JSON Schema support for future UI rendering

• External API Providers - Modular src/apis/ package with swappable implementations

  • OpenMeteoProvider - Weather with geocoding (free, no API key)
  • NOSProvider - Dutch news RSS (16 categories including sports)
  • BBCProvider - English news RSS (21 categories including sports)
  • AggregatedNewsProvider - Merges sources chronologically with category filtering
  • AlphaVantageProvider - Stock/crypto quotes (API key from settings DB)
  • Abstract base classes for provider interoperability

• Provider Dependency Injection

  • WeatherProviderDep, NewsProviderDep, AlphaVantageProviderDep type aliases
  • Async initialization with settings database integration
  • Lifecycle management in shutdown_clients()

• Development Dependencies - requirements-dev.txt

  • pip-audit for security vulnerability scanning
  • ruff for code quality
  • Testing packages moved from main requirements

Changed

• News sources configurable via news.sources setting
• Per-source category filtering via api.{source}.categories
• Categories default to all if not specified

[1.4.8] - 2025-12-25

Added

• Paperless Orphan Cleanup - POST /maintenance/cleanup/paperless endpoint
  • Detects documents deleted from Paperless but still indexed in Library Desk
  • Removes orphaned vectors and graph nodes
  • Supports dry_run=true for preview mode

[1.4.7] - 2025-12-25

Fixed

• Paperless Custom Field Update - Fixed 400 error when marking documents as indexed
  • Paperless API requires field ID (integer) not field name (string)
  • Now looks up library_indexed field ID before updating
  • Webhook params format: doc_url and title from Jinja templates

Added

• Webhook Debug Endpoint - POST /documents/webhook-capture for development testing

[1.4.6] - 2025-12-25

Fixed

• Paperless Webhook Payload Format - Updated model to match Paperless include_document=true format
  • Paperless sends id instead of document_id
  • Paperless sends full document data including content, title, tags, etc.
  • Webhook now uses content from payload, skipping extra Paperless API call
  • Added extra = "ignore" to handle additional Paperless fields

[1.4.5] - 2025-12-25

Added

• Document Storage Integration - Paperless-ngx integration for PDFs, images, and documents
  • Event-driven architecture via Paperless webhooks
  • POST /documents/webhook - Receive document events from Paperless workflows
  • POST /documents/upload - Upload files directly to Paperless
  • POST /documents/upload-url - Download and upload documents from URL
  • POST /documents/search - Semantic search across indexed documents
  • GET /documents/health - Paperless connectivity health check
• DocumentSyncService - Indexes Paperless documents into vectors and graph
  • Fetches document content via Paperless API
  • Chunks text and generates embeddings for Qdrant
  • Creates Document nodes in Neo4j knowledge graph
  • Supports multi-tenancy via user parameter in webhook URL
• PaperlessClient - REST API client for Paperless-ngx
  • Document retrieval, upload, and update operations
  • Health check support
• Paperless Workflow Configuration
  • Production workflow: Document Added (NOT tagged llm-test) → webhook to Library Desk
  • Test workflow: Document Added (tagged llm-test) → webhook with test user

Changed

• Updated src/config.py with Paperless configuration settings
• Added PaperlessDep dependency injection for document endpoints

[1.4.4] - 2025-12-24

Added

• Test Data Cleanup Endpoint - POST /maintenance/cleanup/test-data
  • Purges LLM test data from wiki, graph, and vectors
  • Security-restricted to test user namespace only (users/llm-tester/*, users/llm_tester/*)
  • Supports dry_run=true (default) to preview before deleting
  • Scheduler task configured for weekly cleanup (Sunday 3:00 AM)

[1.4.3] - 2025-12-24

Changed

• Volatile Cache System Refactored to Vector Storage
  • Backend migrated from Redis to Qdrant for semantic search capability
  • Data converted to natural language for embedding and semantic retrieval
  • Collection naming: volatile_{user} for per-user isolation
  • TTL implemented via ttl_expiry timestamp in vector payload
  • Simplified endpoints:
    • GET /volatile/search?q=... - Semantic search across volatile data
    • POST /volatile/store?namespace=...&key=... - Store with query params
    • GET /volatile/{namespace}/{key} - Get specific record
    • DELETE /volatile/{namespace}/{key} - Delete record
  • Removed namespace-specific URL patterns (simpler API for LLM tool use)

Added

• HybridRAG Volatile Integration - Volatile cache now included in multi-source search
  • Volatile results get priority boost in RRF fusion (current data ranks higher)
  • New config options: enable_volatile, volatile_limit (default 1), volatile_threshold
  • Timing breakdown includes volatile_ms
• Volatile Cleanup Endpoint - POST /maintenance/cleanup/volatile
  • Purges expired records across all volatile_* collections
  • Scheduler task for every 10 minutes recommended
  • Returns per-collection cleanup counts
• Natural Language Conversion - Structured data converted for embedding
  • Template-based conversion for each namespace (weather, news, financial, etc.)
  • Fallback for custom namespaces

[1.4.2] - 2025-12-24

Added

• Volatile Cache System - Ephemeral data storage with TTL
  • GET /volatile/{namespace}/{key} - Retrieve cached record
  • POST /volatile/{namespace}/{key} - Store/update record with TTL
  • DELETE /volatile/{namespace}/{key} - Remove record
  • GET /volatile/{namespace} - List keys in namespace
  • DELETE /volatile/{namespace} - Clear all records in namespace
  • GET /volatile/stats - Cache statistics by namespace
  • GET /volatile/scheduled - Records needing refresh (for scheduler)
  • GET /volatile/namespaces - List available namespaces with default TTLs
• Volatile Namespaces - Predefined categories with appropriate TTLs:
  • weather (30min) - Weather conditions and forecasts
  • news (1hr) - Headlines and breaking news
  • financial (5min) - Stock prices, exchange rates
  • transit (5min) - Train/bus schedules, delays
  • traffic (10min) - Commute times, road conditions
  • air_quality (1hr) - Pollution, pollen counts
  • sports (1min) - Live scores, matches
  • social (10min) - Social notifications
  • system (1min) - Service health status
  • context (1hr) - Session state
  • custom (1hr) - User-defined data
• Refresh Schedule Support - Optional cron expressions for scheduler integration

[1.4.1] - 2025-12-24

Fixed

• Wiki.js API token now optional - GraphQL API works without authentication
• Container startup failure when WIKI_GRAPHQL_API env var not set

[1.4.0] - 2025-12-24

Added

• Maintenance Router - New /maintenance endpoints for system health and cleanup
  • GET /maintenance/health - Lightweight health check (detailed mode available)
  • POST /maintenance/cleanup/all - Full orphan cleanup (vectors + graph)
  • POST /maintenance/cleanup/vectors - Purge orphan vector chunks
  • POST /maintenance/cleanup/graph - Purge orphan graph nodes
  • POST /maintenance/reconcile-index - Combined cleanup + reindex missing pages
• Bidirectional Orphan Detection - Cross-validate vectors and graph nodes
  • find_documents_without_vectors() - Graph nodes missing vector chunks
  • find_chunks_without_graph_nodes() - Vector chunks missing graph nodes
• Qdrant Client Methods - Bulk operations for maintenance
  • scroll_all_points() - Iterate all points with pagination
  • delete_by_ids() - Batch delete by point IDs
• Graph Service Cleanup - Node deletion methods
  • delete_document_node() - Remove document and relationships
  • delete_collection_node() - Remove collection and contained documents
  • get_all_document_references() - Get all document references for validation
• Redis Timestamp Tracking - last_cleanup timestamp for scheduler integration
• Memory System Plan - Documented three-tier architecture (volatile/documents/knowledge)

Changed

• Wiki.js Authentication - Switched from username/password to API token
  • New WIKI_GRAPHQL_API environment variable for JWT token
  • Deprecated WIKIJS_USERNAME and WIKIJS_PASSWORD (kept for backwards compatibility)
• Service Dependencies - Added VectorServiceDep and GraphServiceDep type aliases

Fixed

• Wiki.js client now properly handles API token auth without login flow

[1.3.3] - 2025-12-23

Added

• Temperature parameter to OllamaClient.generate_text() for controlling output determinism
• TODO.md tracking remaining stub endpoints to implement
• Wired /query/semantic endpoint to VectorService
• Wired /query/graph endpoint to GraphService

Changed

• Improved LLM prompts based on llm-findings.md recommendations:
  • Keyword extraction: temperature 0.0, negative constraints
  • LLM re-ranking: temperature 0.0, explicit rules
  • Conflict detection: temperature 0.0, analysis steps (CoT)
  • Wiki page creation: temperature 0.3, anti-hallucination constraints
  • Page reconstruction: temperature 0.2, preservation constraints
  • Web results analysis: temperature 0.0, conservative approach
• Test fixtures now use configurable host (TEST_HOST) instead of Docker hostnames

Removed

• Dead code: unused get_default_user() function
• Unused imports from routers (wiki.py, graph.py, hybrid_rag.py)
• Stub endpoints shadowed by real implementations (/stats, /ingest/document, /ingest/batch)

[1.3.2] - 2025-12-22

Changed

• Consolidated Ollama model configuration - All LLM operations now use single OLLAMA_MODEL environment variable
  • Removed separate reranker_model setting
  • HybridRAG re-ranking, consolidation analysis, and wiki page writing all use the same model
  • Improves VRAM efficiency by keeping one model hot
• Added OLLAMA_EMBEDDING_MODEL environment variable for embedding model (previously overloaded OLLAMA_MODEL)
• Updated WikiPageWriter to accept settings instead of hardcoded model name

[1.3.1] - 2025-12-16

Fixed

• Smart create endpoint missing content_extractor dependency causing 500 errors on POST /wiki/pages/smart-create

[1.3.0] - 2025-12-15

Changed

• Two-Stage RRF Architecture - Major refactor to level the playing field between wiki and web results

  • Stage 1: Vector and graph results merged into single "wiki" ranking using mini-RRF
  • Stage 2: Final RRF between wiki (single source) and web (single source)
  • Wiki pages no longer get 2x advantage from appearing in both vector and graph searches
  • Multi-source confirmation still determines wiki internal ranking

• Skip synonyms in graph search - LLM-generated synonyms (e.g., "author") no longer match unrelated graph entities (e.g., "author2000")

  • Vector search still uses synonyms for semantic similarity
  • Graph search uses only core keywords for exact entity matching

Added

• VECTOR_SIMILARITY_THRESHOLD config setting (default: 0.7) to filter weak vector matches
• Deduplication in graph search to prevent same document appearing multiple times

Fixed

• Graph search duplicate entity bug where same document could appear twice if entity linked multiple times

[1.2.1] - 2025-12-15

Fixed

• HybridRAG router missing content_extractor dependency causing 500 errors on /query/hybrid endpoint

[1.2.0] - 2025-12-15

Added

• RAG Search Endpoint (POST /rag/search)

  • Web, news, and image search via SearXNG
  • Full content extraction using Trafilatura (F1 score 0.958)
  • Redis caching with configurable TTL
  • Markdown sources summary for LLM consumption
  • Returns both extracted content and original snippets

• Content Extraction Endpoints (/content/*)

  • POST /content/extract - Extract content from a single URL
  • POST /content/extract/batch - Batch extraction (up to 20 URLs)
  • Reusable ContentExtractor client for use across the codebase

• HybridRAG Content Extraction Enhancement

  • Web search results now include full extracted content via Trafilatura
  • Falls back to original snippets if extraction fails
  • Improves context quality for LLM re-ranking and consumption

Changed

• Added new configuration options:
  • SEARCH_CACHE_TTL - Search cache TTL in seconds (default: 300)
  • SEARCH_TIMEOUT - SearXNG timeout (default: 10s)
  • CONTENT_EXTRACTION_TIMEOUT - Per-URL extraction timeout (default: 5s)
  • CONTENT_MAX_LENGTH - Max extracted content length (default: 2000)
  • SEARCH_DEFAULT_LIMIT - Default search results (default: 10)

Dependencies

• Added trafilatura~=1.12.0 for content extraction

[1.1.3] - 2025-12-14

Added

• Watchtower update trigger in Gitea workflow after successful build

[1.1.2] - 2025-12-14

Fixed

• Updated registry login URL in Gitea workflow (git.schweitz.net → git.schweitz.internal)

[1.1.1] - 2025-12-14

Fixed

• Updated container registry tag URLs in Gitea workflow (git.schweitz.net → git.schweitz.internal)

Added

• Tests for Smart Page Creation feature (test_smart_create.py)
  • Model validation tests for WikiSmartCreateRequest/Response
  • WikiService.smart_create_page method tests
  • Bidirectional entity linking utility tests
  • Endpoint validation tests

[1.1.0] - 2025-12-11

Added

• Smart Page Creation Endpoint (POST /wiki/pages/smart-create)

  • Combines HybridRAG research with LLM content generation
  • Searches existing wiki, knowledge graph, and web for topic context
  • Uses WikiPageWriter to synthesize findings into structured wiki content
  • Auto-generates page path from topic if not provided
  • Returns research summary with source counts

• Bidirectional Entity Linking

  • New shared utility (entity_linking_utils.py) for reusable entity linking
  • Forward links: Links entities mentioned in new pages to existing entity pages
  • Backward links: Updates existing pages that mention the new entity
  • Runs automatically in background after smart page creation

• Version Management

  • Added pyproject.toml with project metadata and version
  • Version is now read from pyproject.toml (single source of truth)
  • Health check endpoint returns current version
  • FastAPI docs show current version

Changed

• Updated config.py to read version from pyproject.toml
• Updated main.py to use centralized version

[1.0.0] - 2025-12-10

Added

• Initial release extracted from portainer-core
• Wiki page management (/wiki/pages CRUD endpoints)
• HybridRAG search (/query/hybrid) with vector, graph, and web search
• Knowledge graph operations (/graph/*)
• Vector search operations (/vector/*)
• Knowledge consolidation from search results (/consolidate/knowledge)
• Entity linking and extraction
• Wiki.js change listener for auto-processing user edits
• Multi-tenant architecture with user namespace isolation