library-desk/CHANGELOG.md

# Changelog

All notable changes to Library Desk will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [1.9.2] - 2026-08-16

### Fixed

- `/health` no longer reports the whole service unhealthy because one
  dependency is merely slow. Its five probes (neo4j, qdrant, wikijs, searxng,
  ollama) previously ran one after another with no timeout on neo4j or
  qdrant, so a single hung dependency could block the response past the
  container healthcheck's 10s timeout and get the container marked
  unhealthy for a reason unrelated to its own liveness. Each probe is now
  bounded at 2s and all five run concurrently, so one hanging dependency is
  reported unhealthy on its own without holding up the others or the
  response.
- Internal-only: the three probes `check_service_health()` also computes
  (paperless, system_settings, scheduler) received the same bounding and
  concurrency, for consistency and because they carried the identical
  unbounded-hang risk internally. This has no visible effect today — none
  of the three is currently returned by `/health` — but protects a future
  caller that does read them.

## [1.9.1] - 2026-08-08

### Fixed

- The Wiki.js change listener now survives a database restart. It held a single
  `LISTEN` connection with no supervision, so when the connection dropped it was
  gone permanently while `running` stayed `True` — the service kept reporting
  healthy and silently stopped indexing every page edit until someone restarted
  the container. This happened for real on 2026-08-08 when `postgres-shared` was
  redeployed. It now detects the drop via asyncpg's termination callback and
  reconnects with bounded exponential backoff (1s doubling to a 60s cap),
  retrying indefinitely because a database in maintenance does come back and
  giving up would recreate the same silent deafness.
- `WikiChangeListener.running` is derived from the live connection instead of
  being assigned once at startup, so it can no longer claim a subscription that
  does not exist.

### Added

- `/health` reports the change listener under `services.wiki_listener`
  (`subscribed`, `reconnects`, `last_gap_seconds`). Nothing previously exposed
  its state anywhere, which is why a dead listener went unnoticed. It is
  deliberately excluded from the overall healthy/degraded verdict: it recovers on
  its own, and flipping the container unhealthy for the duration of a database
  outage would add a restart loop to an incident rather than information.
- On reconnect the listener logs the outage duration and warns that `NOTIFY`
  events emitted during the gap were lost and cannot be replayed, pointing at
  `POST /maintenance/integrity-check` to reconcile. A reconciliation pass is not
  performed automatically — tenant attribution for a changed page is non-trivial
  here, and guessing it wrong writes content into the wrong user's namespace.

## [1.9.0] - 2026-07-20

### Security

- The Wiki.js integration buttons no longer embed an API key in the browser. The re-index and entity-link endpoints now authenticate via the NPM `/library-desk/` proxy (Authentik session for external users, LAN bypass for internal), verified by a trusted proxy marker header. The previously-embedded key was a full-privilege key served to every wiki visitor; it has been rotated out of service.
- `verify_api_key` now uses a constant-time comparison.

### Changed

- `static/wikijs-integration.js` calls library-desk same-origin (`/library-desk/...`) with `credentials: same-origin` and no `Authorization` header. Update the Wiki.js code-injection snippet to `<script src="/library-desk/static/wikijs-integration.js">`.
- Machine callers (the Scheduler) continue to use the Bearer API key against the container-network endpoints; only the two browser endpoints switched to proxy auth.

## [1.8.1] - 2026-07-19

### Changed

- Container images are now pushed via the `git.schweitz.net` registry endpoint (the `.internal` registry domain is being retired); no change to the image name consumed by Watchtower.
- `.env.example` service URL defaults now use docker-dataplane container names (`wiki:3000`, `neo4j:7687`, `searxng:8080`, `paperless:8000`, `ollama:11434`) instead of host IP + published port, ahead of the Phase 4 port lockdown; AGENTS.md deploy health-check URL corrected from port 8000 to 8089.

## [1.8.0] - 2026-07-14

### Added

- **Stub endpoints implemented** (`/ingest/check-updates`, `/ingest/status/{job_id}`, `/ingest/repo-status/{repository}`, `/deduplicate/check`) — all previously returned canned "not yet implemented" responses; all now require an explicit `user` (Phase B rule):
  - `/ingest/check-updates` compares the `content_hash` now recorded on the tenant's Neo4j Document nodes at ingestion time against the SHA-256 of current Wiki.js page content in a single UNWIND Cypher query, returning `changed` / `new` / `deleted` page lists (auto-generated entity stubs excluded; documents whose stored hash predates hash tracking are flagged `stored_hash_missing` and count as changed).
  - `/ingest/status/{job_id}` is backed by the Redis `JobManager` (jobs are tenant-scoped; other tenants' jobs return 404). `/ingest/page`, `/ingest/batch` and `/ingest/all` now record job entries and return a `job_id`.
  - `/ingest/repo-status/{repository}` reports wiki page count vs indexed Document-node count under `users/{tenant}/{repository}` plus the tenant's Redis job statistics.
  - `/deduplicate/check` runs a tenant-scoped Qdrant similarity scan: wiki chunk pairs above the threshold (default 0.9 cosine) grouped per page pair with best score, matching chunk-pair count, and page references. Read-only.
- **Job + Scheduler task plumbing** — In-process hourly `job_cleanup_loop` (started at app startup, cancelled at shutdown) reclaims expired Redis job-set memberships (`JobManager.cleanup_expired_jobs`). `docs/scheduler-tasks.md` defines the four production Scheduler task payloads for the deploy checklist (nightly integrity 04:30, weekly quality report Sunday 03:00, daily Paperless orphan-cleanup 05:00 on the existing endpoint, and disabling `test_example_task`) with exact HTTP bodies (explicit `user=jpmschweitzer`, `${LIBRARY_API_KEY}` auth placeholder). `scripts/register_scheduler_tasks.py` reads the Scheduler API location from `SCHEDULER_URL` and registers them — dry-run by default (prints payloads), `--execute` gated and requiring `LIBRARY_API_KEY`.
- **Weekly quality report** — `POST /maintenance/quality-report {user}` runs the duplicate scan, flags stale pages (not updated in N days AND ≤ M SearchQuery hits from the graph data), lists pages missing tags/description, folds in the latest integrity-check results (Redis-cached or run inline), and writes a dated report page to `users/{user}/system/quality-reports/YYYY-MM-DD` (same-day reruns update the same page — the page id is remembered in Redis because the Wiki.js listing lags page creation). Response returns the full report content + page path. Verified end-to-end against the local dev server as `llm_tester`.
- **Nightly integrity check** — `POST /maintenance/integrity-check {user}` (read-only: reports, never auto-fixes) reports per tenant: wiki pages with ZERO vectors in Qdrant (silent-skip reindex victims), orphaned vectors whose wiki page no longer exists, unexpected Qdrant collections (test-tenant residue and unknown namespaces flagged; other services' collections counted as foreign), Neo4j Document nodes without wiki counterparts, plus counts and duration. The latest report is cached in Redis (30 days) so the weekly quality report can fold it in.

### Fixed (Phase D review batch)

- **Runtime prefetch registration is executable** - `SchedulerClient.register_volatile_fetch` (used by consolidation's `_register_prefetch`) registered tasks that were dead on arrival three ways: the JSON body sat under the ignored `body` key (`rest_api_executor` only reads `config["payload"]`), there was no `auth` block (the scheduled POST would 401 against library-desk's `verify_api_key`), and `user` was in the body while every `/volatile/fetch` endpoint requires it as a QUERY parameter (would 422). The config now puts `user` in the URL query string (URL-encoded), an empty `payload`, and `auth: {type: bearer, token: "${LIBRARY_API_KEY}"}` (substituted from the Scheduler's environment; never stored raw). `SchedulerClient` itself also sent no Authorization to the Scheduler API, so registration failed silently at consolidation time — it now sends `Authorization: Bearer <SCHEDULER_API_KEY>` (new `scheduler_api_key` setting; a client without a key logs a warning).
- **Document sync uses delete-last reindex order** - `DocumentSyncService._index_vectors` deleted the document's existing chunks (`delete_by_filter`) BEFORE embedding, so a failed embedding pass (e.g. Ollama down) left the Paperless document with zero vectors until the next successful sync — the exact hazard already fixed for wiki pages in `VectorService.update_from_page`. Chunk point ids are now deterministic (uuid5 of `document_{id}_chunk_{i}`, replacing random uuid4) so re-upserting overwrites in place; new points are upserted first and stale points (including legacy uuid4 ones) are pruned afterwards, only after a successful upsert.
- **5MB response cap now aborts the download** - `ContentExtractor._fetch` buffered the entire response body in memory before truncating to `MAX_RESPONSE_BYTES`, so the cap protected Trafilatura but not memory/bandwidth (a multi-hundred-MB URL was still fully downloaded, on up to `max_urls_per_batch` concurrent fetches). Fetches are now streamed (`client.stream` + `aiter_bytes`) and the connection is closed as soon as the cap is reached.
- **Registrar authenticates to the Scheduler API** - `scripts/register_scheduler_tasks.py --execute` sent no `Authorization` header while the Scheduler's task-management endpoints are Bearer-guarded (`verify_api_key`: 401 on missing key), so the existence probes 401'd (misread as "task absent") and every registration failed; only the public `/health` gate passed. `--execute` now requires `SCHEDULER_API_KEY` in the environment (refuses to run without it, key is never stored) and sends `Authorization: Bearer $SCHEDULER_API_KEY` on all of its own HTTP calls. Deploy note updated alongside the existing `LIBRARY_API_KEY` requirement.

### Fixed (hazards batch)

- **CORS wildcard + credentials removed** - `allow_origins=["*"]` combined with `allow_credentials=True` told browsers to attach credentials for any site. Credentials are now disabled (all real callers are server-to-server and use the `Authorization` header, which wildcard-origin CORS without credentials still permits) and the origin list is configurable via `CORS_ALLOW_ORIGINS` (comma-separated, default `*`).
- **Scheduler task auth no longer stores raw tokens** - the Phase C Scheduler task definitions put `Authorization: Bearer ${LIBRARY_API_KEY}` in plain `headers` and the registrar substituted the REAL key client-side on `--execute`, which would persist it in the Scheduler's `scheduled_tasks.config` JSONB column — and the Scheduler's `rest_api_executor` does not substitute env vars in plain headers anyway (only in `url`/`payload`/`auth`). The definitions now use the executor's `auth: {type: bearer, token: "${LIBRARY_API_KEY}"}` block, substituted from the SCHEDULER's environment at execution time; the registrar sends the placeholder verbatim and no longer needs (or accepts) the key. Also fixed: the JSON body moved from the ignored `body` key to `payload` (the executor only reads `config["payload"]`, so the tasks would have POSTed empty bodies and failed Phase B user validation).
- **Reranker index parser dedupes** - an LLM ranking answer like `3,3,1` inserted the same result into the final ranking twice; parsed indices are now deduplicated preserving first occurrence.
- **Single HybridRAG wiring point** - three separate constructions existed: an unused `dependencies.get_hybrid_rag_service` singleton lacking `volatile_service`, an inline per-request copy in the `/query/hybrid` router, and another inline copy in `/wiki/pages/smart-create` also lacking `volatile_service`. All callers now use the dependencies singleton, which includes `volatile_service` (smart-create research can now hit the volatile cache leg).
- **Hot-path Neo4j writes use write transactions** - remaining graph writes ran as auto-commit `execute_query` calls (no retry, no transaction-function semantics): `GraphService` ingestion (`update_from_page` document + entity queries), `delete_page`, `create_entity_mentions`, document/paperless/collection node deletion, orphan-entity purge, stale-document purge, `cleanup_broken_relationships`; the webhook rename/delete cleanup writes; document-sync `_index_graph`; and consolidation's `_mark_search_processed`/`_add_entity_to_graph`. All now go through `execute_write` (managed transaction with driver retry). Read paths are unchanged.

### Changed (performance)

- **Content extractor hardened** - `ContentExtractor` now downloads pages with `httpx.AsyncClient` under real connect (3s) and read timeouts on the event loop; only the CPU-bound Trafilatura parse runs in the thread pool. Previously `trafilatura.fetch_url` ran inside the worker thread with no caller-side timeout control, so an `asyncio.wait_for` timeout abandoned the thread while it kept downloading for up to ~30s. Trafilatura now parses each document ONCE via `bare_extraction` (text + metadata together) — the old code ran `extract()` twice (the XML pass was computed and discarded) plus `bare_extraction`, three full parses per page. `extract_batch` caps full-page extractions per call (default 8; overflow URLs return unsuccessful so the web leg falls back to the search snippet), responses are capped at 5MB before parsing, and thread-pool queue depth is logged for backpressure visibility.
- **Top-k enrichment with one batched lookup** - Phase 3 (`_enrich_with_related_dossiers`) ran a sequential Neo4j round-trip for EVERY fused result and the final trim then discarded most of the output. It now enriches only the results that can still reach the response (the Phase 4 rerank slice of 20 when reranking is enabled, otherwise `final_result_count`) and resolves all of them in ONE UNWIND-batched Cypher query (`GraphService.get_related_documents_batch`, tenant-scoped like the single-page variant, per-page ordering/limit preserved). Unenriched tail results carry an empty `related_dossiers` list as before.
- **Search persistence off the hot path, one atomic transaction** - HybridRAG Phase 6 (`_persist_search_for_librarian`) no longer gates the `/query/hybrid` response: the `search_id` is generated up front and returned immediately while the Neo4j write runs as a background task (strong task references held so tasks are not GC'd mid-flight). The write itself collapsed from ~21+ sequential auto-commit queries (SearchQuery node + per-document FOUND links + per-web-result WebResult nodes) into ONE UNWIND-based `execute_write` transaction, so a mid-way failure can no longer leave a partial SearchQuery graph behind. The persisted shape (SearchQuery properties incl. `processed: false`, tenant labels, `FOUND` relationship properties, WebResult properties) is unchanged and pinned by `tests/test_search_persistence.py` against exactly what the consolidation service queries. `timing.persistence_ms` now reports 0 (no longer on the request path).
- **Batched embeddings + delete-last reindex** - `OllamaClient.embed_batch` now sends ONE batched `/api/embed` request (verified against the live Ollama; the old "batch" looped one `/api/embeddings` call per chunk) with a per-text fallback preserving partial-success semantics. `VectorService.update_from_page` embeds all chunks in that single call and upserts them in one Qdrant batch, and the reindex order is reversed: new points are upserted BEFORE stale points are pruned (deterministic uuid5 chunk ids make the overwrite safe), so a mid-way failure can no longer leave a page with zero vectors — the old order deleted everything first. The summary now reports `status` (`success`/`partial`/`failed`) and `chunks_skipped` instead of unconditional `success=True`; a fully failed embedding pass keeps the old vectors and reports failure. Measured on a real 7-chunk page ingest as `llm_tester` against the local server: ~375ms → ~181ms median (3 runs each).
- **Document sync indexing fixed** - `DocumentSyncService._index_vectors` now awaits `ensure_collection` (the coroutine was created but never ran, so fresh tenants had no collection at upsert time), filters out `None` entries from `embed_batch` so one failed chunk embedding no longer aborts the whole document upsert (all-failed still reports failure), and routes the raw `client.delete`/`client.upsert` calls through the async wrapper (`delete_by_filter` / new batch `upsert_points`). Offline unit tests added.
- **Async Qdrant client** - `QdrantClientWrapper` now uses `AsyncQdrantClient` with an explicit timeout (`QDRANT_TIMEOUT`, default 30s). Every vector call previously ran on the synchronous client inside async wrapper methods, blocking the FastAPI event loop for the duration of each Qdrant round-trip. The wrapper API is unchanged (all methods were already `async`), so call sites only gained real awaits. The HybridRAG document leg was moved off the deprecated raw `client.search` onto the wrapper's `search_vectors` (fixing a latent `AttributeError`: it called the nonexistent `ollama.embed_text`, so the leg always reported `failed`), and the health check awaits `get_collections`.

### Changed

- **BREAKING: `user` is now required on every tenant-data endpoint** - The implicit `jpmschweitzer` default tenant (`DEFAULT_USER`) has been removed everywhere. All endpoints that read or write tenant data (`/query/*`, `/wiki/*`, `/vector/*`, `/graph/*`, `/ingest/*`, `/volatile/*`, `/documents/*`, `/stats`, `/rag/search`) now reject requests without an explicit, non-empty, non-whitespace `user` (HTTP 422), matching the existing `/maintenance/*` pattern. A shared validator (`require_user` dependency / `RequiredUser` model type) also rejects blank users. The Wiki.js change listener now skips changes whose notification email yields no user instead of attributing them to the production tenant. **Caller coordination required:** tatlock and the Scheduler ingest/prefetch/consolidation tasks must send an explicit `user` on every call — see the deploy checklist.

### Fixed (security)

- **Cross-tenant leaks in HybridRAG legs and ingestion closed** - A live probe as `user=llm_tester` returned `jpmschweitzer` pages. Root causes fixed:
  - **Ingestion namespace enforcement**: `vector` and `graph` `update_from_page` now refuse pages whose wiki path is outside `users/{user}/` (previously any tenant could ingest any page id — including another tenant's — into its own collection/labels, which is how foreign content ended up in the vector leg). `/ingest/all` clamps `path_prefix` to the caller's namespace (400 on cross-tenant prefixes) and defaults to `users/{user}`.
  - **Search persistence**: the `FOUND` link in HybridRAG phase 6 matched `(d:Document {page_id})` unscoped, attaching the caller's SearchQuery to other tenants' Document nodes; it now matches only `User_{Tenant}_Document` nodes.
  - **Graph enrichment/consolidation queries scoped**: `_get_entity_mention_count`, entity-stub generation, orphan-entity find/purge, and `cleanup_broken_relationships` matched unscoped `Document`/`SearchQuery` nodes; all now use the tenant's labels. Entity-page existence checks list only the tenant's wiki namespace.
  - **Volatile collections sanitized**: `volatile_{user}` collection names now use the sanitized user id (same scheme as document collections).
  - **Namespace matching hardened**: `is_path_in_user_namespace` now enforces a path-segment boundary (`users/llm_tester2` is no longer inside `llm_tester`'s namespace) and compares sanitized tenant segments.
  - Offline unit tests added per leg (vector, graph, volatile, documents, enrichment, persistence, ingestion) asserting the tenant-scoped collection/label/path is used.

- **`/query/graph` and `/graph/query` hardened to read-only** - The documented "automatic user scoping" was a no-op (a live probe confirmed any user string could read the whole graph) and the client permitted writes. Raw Cypher queries are now (1) rejected with 400 when they contain write clauses (`CREATE`/`MERGE`/`DELETE`/`DETACH`/`SET`/`REMOVE`/`DROP`/`FOREACH`/`LOAD CSV`) or any `CALL` procedure (conservative denylist on the uppercased query), and (2) executed through a Neo4j session opened with `default_access_mode=READ_ACCESS` so the database itself refuses writes as a backstop. The endpoints are now honestly documented as **admin/debug, unscoped read-only**: results are not restricted to the caller's tenant labels — use `/graph/nodes` for tenant-scoped access.

### Added

- **Degradation signaling** - `HybridRAGResponse` now includes `source_status` (per-leg `'ok'`/`'failed'`/`'disabled'` for vector, graph, web, volatile, documents) and `degraded` (true when any enabled leg failed). Retrieval legs report errors instead of silently swallowing them; failed legs are logged at WARNING. Both fields are additive and optional, so clients that ignore them are unaffected.
- **Hard-isolated test-tenant lifecycle for the test suite** - `tests/conftest.py` rewritten: the production host default (`192.168.86.149`) is gone (`TEST_HOST` env, safe `localhost` default; the API under test is the local wakeup server via `LIBRARY_DESK_URL`, never the production container on 8089). The suite is pinned to the reserved test tenant `llm_tester`; a session guard aborts the entire run if the effective tenant is `jpmschweitzer` or outside the reserved `llm_tester*` namespace. Integration tests are marked and only run with `RUN_INTEGRATION_TESTS=1` (plus a passing guard). A session-scoped teardown deletes ALL `llm_tester` artifacts created during the run — Qdrant `*_llm_tester` collections, Neo4j `User_Llm_Tester*`-labelled nodes, the `users/llm_tester` wiki subtree, and `llm_tester` Redis keys on the service DB — with hard tenant assertions before every delete. Legacy integration tests were pinned to the test tenant (no more production-namespace reads).
- **Live tenant-isolation test** - `tests/test_tenant_isolation_live.py` (integration-marked, guard-gated): creates and ingests a wiki page as `llm_tester` against the local wakeup server + shared services, asserts `/query/hybrid` as `llm_tester` retrieves its own content with ZERO results from the `jpmschweitzer` tenant, and asserts a third nonexistent tenant (`llm_tester_void`, still inside the reserved namespace — nothing is ever written as `jpmschweitzer`) gets zero results entirely. Teardown removes everything it created.
- **Test-residue purge tooling** - `scripts/purge_test_artifacts.py`: dry-run by DEFAULT (`--execute` required for real deletion), targets only confirmed test residue (Qdrant `library_desk_llm_tester` / `test_user` / `library_desk_test_user` / `memories_llm_tester` / `volatile_llm_tester` / `core_ai_user_test_*` / anything containing `llm_tester`; Neo4j `User_Llm_Tester*`-labelled nodes and legacy `users/llm*` Document nodes; `llm_tester` Redis keys on the service DB), prints counts per target, hard-aborts if any target rule ever matches a `jpmschweitzer`-namespaced identifier, and documents the snapshot prerequisite (Qdrant snapshot API + `neo4j-admin database dump`) in its docstring.
- **Offline unit tests** - New mock-based tests (no live services) for model-name resolution under the env collision, per-leg failure signaling, the `/stats` page-count prefix, LLM-call timeouts, and Wiki.js listing pagination.

### Fixed

- **Ollama generation model env collision** - Renamed the generation-model setting `ollama_model` to `ollama_llm_model` (env: `OLLAMA_LLM_MODEL`, default `gemma4:e2b`). The container env `OLLAMA_MODEL=nomic-embed-text` (meant for embeddings) was shadowing the generation model, breaking Phase 0 keyword extraction and Phase 4 LLM re-ranking on every request. Startup now logs the resolved generation model.
- **LLM call timeouts** - Phase 0 keyword extraction and Phase 4 re-ranking are wrapped in a 12s `asyncio.wait_for` with graceful fallback, so a hung Ollama call can no longer gate retrieval for the full 120s client timeout.
- **`/stats` wiki page count** - The endpoint passed the bare user name as path prefix (matching nothing) and always reported 0 pages; it now counts pages under `users/{user}`.
- **Wiki.js page listing** - `list_pages` applied the API-side `limit` before client-side path/tag filters, dropping matching pages that sort late; the limit now applies after filtering. `list_all_pages` replaced its fake pagination loop with a real limit-growth loop (Wiki.js 2.x `pages.list` has no offset argument) that fetches until the API returns fewer pages than requested.
- **Consolidation loop silently drained its queue on LLM failure** - Root cause of the 30-minute knowledge-consolidation loop processing 0 searches ("No unprocessed searches found" in prod): the `OLLAMA_MODEL` env collision (see below) made every consolidation `/api/generate` call fail with HTTP 400 (`"nomic-embed-text" does not support generate` — confirmed in prod logs and by direct Ollama probe), classification returned empty, and the loop STILL marked every SearchQuery `processed: true` — permanently consuming the queue with zero pages ever created (live Neo4j: 197/200 SearchQuery nodes processed with no output). LLM-infrastructure failure now raises `ConsolidationLLMUnavailableError`: the affected searches stay unprocessed (retried next run), the batch aborts after the first failure, and the response reports `searches_deferred`. Every run now logs `searches_processed` and `duration_ms` (also new response fields). The lookback boundary is now timezone-aware UTC. Regression tests added.
- **Wiki.js `update_page` without tags** - Wiki.js 2.x requires `tags` on the update mutation (the server unconditionally maps over it); every `update_page(page_id, content=...)` call without tags failed with `Cannot read properties of undefined (reading 'map')` — this silently broke the consolidation service's page-update path too. The client now preserves the page's current tags when the caller does not supply any.
- **Wiki.js listing completeness under pre-filter limits** - Observed live: `pages.list(limit=100)` returned 43 pages while 140 existed (`limit=500` returned all) — Wiki.js applies the limit BEFORE its own visibility filtering, so "fewer pages than requested" does not mean the listing is complete and the limit-growth loop stopped early, silently truncating listings (page counts, cleanups, integrity scans). The loop now grows the limit until the returned count stops increasing (fixed point), at the cost of one confirming fetch.

## [1.7.3] - 2026-01-07

### Fixed

- **Endpoint TTL defaults** - Updated all fetch endpoint defaults to match namespace TTLs (2x refresh interval)

## [1.7.2] - 2026-01-07

### Fixed

- **Volatile TTL doubled** - TTL now 2x refresh interval to survive missed/delayed scheduler runs

## [1.7.1] - 2026-01-07

### Fixed

- **CI workflow** - Updated Gitea Actions to trigger on tag push (matching core-api)

## [1.7.0] - 2026-01-07

### Added

- **Combined Environment Endpoint** - `POST /volatile/fetch/environment/{city}`
  - Fetches weather and air quality concurrently with `asyncio.gather()`
  - Single geocode lookup shared between both API calls
  - More efficient than calling weather and air_quality separately
  - Reduces wall-clock time and eliminates redundant geocoding

### Fixed

- **Scheduler executor name** - Fixed `rest_api` → `rest_api_executor` in SchedulerTask model and register_volatile_fetch() to prevent "Executor module not found" errors

## [1.6.2] - 2025-12-30

### Added

- **System Statistics Endpoint** - `GET /stats`
  - Neo4j: node counts by type (Document, Entity, Collection, Search)
  - Qdrant: collection counts, total vectors, per-collection breakdown
  - Wiki.js: total page count
  - Paperless: documents, tags, correspondents, document types

- **Weather/Forecast Separation** - Split weather into two distinct namespaces
  - `POST /volatile/fetch/weather/{city}` - Current conditions only (1hr TTL)
  - `POST /volatile/fetch/forecast/{city}` - 7-day outlook (12hr TTL)
  - Different update frequencies for efficient caching
  - `FORECAST` namespace added to volatile namespaces

### Changed

- Weather namespace TTL changed from 30 minutes to 1 hour (current conditions)
- Forecast data now stored separately with 12 hour TTL

## [1.6.1] - 2025-12-30

### Added

- **Weather Forecast Support** - Enhanced weather fetch with 7-day daily forecasts
  - Current conditions now include UV index
  - Daily forecasts with high/low temps, conditions, precipitation chance, UV max
  - Natural language text summary with multi-day outlook
- **Sun Times Endpoint** - `POST /volatile/fetch/sun/{city}`
  - Sunrise and sunset times (HH:MM and ISO formats)
  - Daylight duration in seconds and hours
  - Separate volatile namespace with 24hr TTL
  - Useful for home automation light triggers
- **Air Quality Endpoint** - `POST /volatile/fetch/air_quality/{city}`
  - European and US AQI indices
  - Pollutants: PM2.5, PM10, ozone, nitrogen dioxide, sulphur dioxide, carbon monoxide
  - Pollen data (grass, birch, alder) for European locations (seasonal)
  - Hourly refresh (1hr TTL)
- **New Base Models**
  - `SunTimes` dataclass for sunrise/sunset data
  - `AirQuality` dataclass with AQI and pollutants
  - `AirQualityProvider` abstract interface
- **New Volatile Namespace** - `SUN` for sunrise/sunset times (86400s default TTL)

### Changed

- Weather fetch now uses `get_forecast()` instead of `get_current()` for richer data
- `OpenMeteoProvider` now implements both `WeatherProvider` and `AirQualityProvider`

## [1.6.0] - 2025-12-29

### Added

- **Memory System Implementation** - Complete three-tier memory architecture
  - **Volatile Fetch Endpoints** - Scheduler-driven prefetch for ephemeral data
    - `POST /volatile/fetch/{namespace}/{key}` - Fetch and cache external data
    - Weather, news, and financial data providers integrated
    - Auto-caching with namespace-specific TTLs
  - **Unified Memory Routing** - LLM-based classification of web results
    - Routes content to wiki (stable), volatile (ephemeral), file (documents), or prefetch (scheduled)
    - Integrated into consolidation service post-processor
  - **Document Recall in HybridRAG** - Paperless documents as fourth retrieval source
    - Documents searched alongside wiki, volatile, and web in parallel
    - New config: `enable_documents`, `document_limit`, `document_threshold`
    - `paperless_id` field in results for document attribution
    - `document_ms` timing in performance breakdown

- **Scheduler Integration** - External scheduler service for prefetch task management
  - `SchedulerClient` - Full REST API client for task CRUD operations
  - `register_volatile_fetch()` convenience method for prefetch registration
  - Consolidation service now creates scheduled tasks for prefetch-worthy content
  - Health checks integrated into startup/shutdown lifecycle

### Changed

- HybridRAG now searches 4 sources in parallel (wiki, volatile, documents, web)
- Consolidation service uses external scheduler instead of settings storage for prefetch

## [1.5.0] - 2025-12-26

### Added

- **Central Settings Database** - Tatlock-wide configuration via PostgreSQL
  - `SettingsClient` for async access to `system_settings` database
  - User-scoped settings with global fallback
  - API config storage with `enabled` toggle and per-source category filters
  - JSON Schema support for future UI rendering

- **External API Providers** - Modular `src/apis/` package with swappable implementations
  - `OpenMeteoProvider` - Weather with geocoding (free, no API key)
  - `NOSProvider` - Dutch news RSS (16 categories including sports)
  - `BBCProvider` - English news RSS (21 categories including sports)
  - `AggregatedNewsProvider` - Merges sources chronologically with category filtering
  - `AlphaVantageProvider` - Stock/crypto quotes (API key from settings DB)
  - Abstract base classes for provider interoperability

- **Provider Dependency Injection**
  - `WeatherProviderDep`, `NewsProviderDep`, `AlphaVantageProviderDep` type aliases
  - Async initialization with settings database integration
  - Lifecycle management in `shutdown_clients()`

- **Development Dependencies** - `requirements-dev.txt`
  - `pip-audit` for security vulnerability scanning
  - `ruff` for code quality
  - Testing packages moved from main requirements

### Changed

- News sources configurable via `news.sources` setting
- Per-source category filtering via `api.{source}.categories`
- Categories default to all if not specified

## [1.4.8] - 2025-12-25

### Added

- **Paperless Orphan Cleanup** - `POST /maintenance/cleanup/paperless` endpoint
  - Detects documents deleted from Paperless but still indexed in Library Desk
  - Removes orphaned vectors and graph nodes
  - Supports `dry_run=true` for preview mode

## [1.4.7] - 2025-12-25

### Fixed

- **Paperless Custom Field Update** - Fixed 400 error when marking documents as indexed
  - Paperless API requires field ID (integer) not field name (string)
  - Now looks up `library_indexed` field ID before updating
  - Webhook params format: `doc_url` and `title` from Jinja templates

### Added

- **Webhook Debug Endpoint** - `POST /documents/webhook-capture` for development testing

## [1.4.6] - 2025-12-25

### Fixed

- **Paperless Webhook Payload Format** - Updated model to match Paperless `include_document=true` format
  - Paperless sends `id` instead of `document_id`
  - Paperless sends full document data including `content`, `title`, `tags`, etc.
  - Webhook now uses content from payload, skipping extra Paperless API call
  - Added `extra = "ignore"` to handle additional Paperless fields

## [1.4.5] - 2025-12-25

### Added

- **Document Storage Integration** - Paperless-ngx integration for PDFs, images, and documents
  - Event-driven architecture via Paperless webhooks
  - `POST /documents/webhook` - Receive document events from Paperless workflows
  - `POST /documents/upload` - Upload files directly to Paperless
  - `POST /documents/upload-url` - Download and upload documents from URL
  - `POST /documents/search` - Semantic search across indexed documents
  - `GET /documents/health` - Paperless connectivity health check
- **DocumentSyncService** - Indexes Paperless documents into vectors and graph
  - Fetches document content via Paperless API
  - Chunks text and generates embeddings for Qdrant
  - Creates Document nodes in Neo4j knowledge graph
  - Supports multi-tenancy via user parameter in webhook URL
- **PaperlessClient** - REST API client for Paperless-ngx
  - Document retrieval, upload, and update operations
  - Health check support
- **Paperless Workflow Configuration**
  - Production workflow: Document Added (NOT tagged llm-test) → webhook to Library Desk
  - Test workflow: Document Added (tagged llm-test) → webhook with test user

### Changed

- Updated `src/config.py` with Paperless configuration settings
- Added `PaperlessDep` dependency injection for document endpoints

## [1.4.4] - 2025-12-24

### Added

- **Test Data Cleanup Endpoint** - `POST /maintenance/cleanup/test-data`
  - Purges LLM test data from wiki, graph, and vectors
  - Security-restricted to test user namespace only (`users/llm-tester/*`, `users/llm_tester/*`)
  - Supports `dry_run=true` (default) to preview before deleting
  - Scheduler task configured for weekly cleanup (Sunday 3:00 AM)

## [1.4.3] - 2025-12-24

### Changed

- **Volatile Cache System Refactored to Vector Storage**
  - Backend migrated from Redis to Qdrant for semantic search capability
  - Data converted to natural language for embedding and semantic retrieval
  - Collection naming: `volatile_{user}` for per-user isolation
  - TTL implemented via `ttl_expiry` timestamp in vector payload
  - Simplified endpoints:
    - `GET /volatile/search?q=...` - Semantic search across volatile data
    - `POST /volatile/store?namespace=...&key=...` - Store with query params
    - `GET /volatile/{namespace}/{key}` - Get specific record
    - `DELETE /volatile/{namespace}/{key}` - Delete record
  - Removed namespace-specific URL patterns (simpler API for LLM tool use)

### Added

- **HybridRAG Volatile Integration** - Volatile cache now included in multi-source search
  - Volatile results get priority boost in RRF fusion (current data ranks higher)
  - New config options: `enable_volatile`, `volatile_limit` (default 1), `volatile_threshold`
  - Timing breakdown includes `volatile_ms`
- **Volatile Cleanup Endpoint** - `POST /maintenance/cleanup/volatile`
  - Purges expired records across all `volatile_*` collections
  - Scheduler task for every 10 minutes recommended
  - Returns per-collection cleanup counts
- **Natural Language Conversion** - Structured data converted for embedding
  - Template-based conversion for each namespace (weather, news, financial, etc.)
  - Fallback for custom namespaces

## [1.4.2] - 2025-12-24

### Added

- **Volatile Cache System** - Ephemeral data storage with TTL
  - `GET /volatile/{namespace}/{key}` - Retrieve cached record
  - `POST /volatile/{namespace}/{key}` - Store/update record with TTL
  - `DELETE /volatile/{namespace}/{key}` - Remove record
  - `GET /volatile/{namespace}` - List keys in namespace
  - `DELETE /volatile/{namespace}` - Clear all records in namespace
  - `GET /volatile/stats` - Cache statistics by namespace
  - `GET /volatile/scheduled` - Records needing refresh (for scheduler)
  - `GET /volatile/namespaces` - List available namespaces with default TTLs
- **Volatile Namespaces** - Predefined categories with appropriate TTLs:
  - `weather` (30min) - Weather conditions and forecasts
  - `news` (1hr) - Headlines and breaking news
  - `financial` (5min) - Stock prices, exchange rates
  - `transit` (5min) - Train/bus schedules, delays
  - `traffic` (10min) - Commute times, road conditions
  - `air_quality` (1hr) - Pollution, pollen counts
  - `sports` (1min) - Live scores, matches
  - `social` (10min) - Social notifications
  - `system` (1min) - Service health status
  - `context` (1hr) - Session state
  - `custom` (1hr) - User-defined data
- **Refresh Schedule Support** - Optional cron expressions for scheduler integration

## [1.4.1] - 2025-12-24

### Fixed

- Wiki.js API token now optional - GraphQL API works without authentication
- Container startup failure when `WIKI_GRAPHQL_API` env var not set

## [1.4.0] - 2025-12-24

### Added

- **Maintenance Router** - New `/maintenance` endpoints for system health and cleanup
  - `GET /maintenance/health` - Lightweight health check (detailed mode available)
  - `POST /maintenance/cleanup/all` - Full orphan cleanup (vectors + graph)
  - `POST /maintenance/cleanup/vectors` - Purge orphan vector chunks
  - `POST /maintenance/cleanup/graph` - Purge orphan graph nodes
  - `POST /maintenance/reconcile-index` - Combined cleanup + reindex missing pages
- **Bidirectional Orphan Detection** - Cross-validate vectors and graph nodes
  - `find_documents_without_vectors()` - Graph nodes missing vector chunks
  - `find_chunks_without_graph_nodes()` - Vector chunks missing graph nodes
- **Qdrant Client Methods** - Bulk operations for maintenance
  - `scroll_all_points()` - Iterate all points with pagination
  - `delete_by_ids()` - Batch delete by point IDs
- **Graph Service Cleanup** - Node deletion methods
  - `delete_document_node()` - Remove document and relationships
  - `delete_collection_node()` - Remove collection and contained documents
  - `get_all_document_references()` - Get all document references for validation
- **Redis Timestamp Tracking** - `last_cleanup` timestamp for scheduler integration
- **Memory System Plan** - Documented three-tier architecture (volatile/documents/knowledge)

### Changed

- **Wiki.js Authentication** - Switched from username/password to API token
  - New `WIKI_GRAPHQL_API` environment variable for JWT token
  - Deprecated `WIKIJS_USERNAME` and `WIKIJS_PASSWORD` (kept for backwards compatibility)
- **Service Dependencies** - Added `VectorServiceDep` and `GraphServiceDep` type aliases

### Fixed

- Wiki.js client now properly handles API token auth without login flow

## [1.3.3] - 2025-12-23

### Added

- Temperature parameter to `OllamaClient.generate_text()` for controlling output determinism
- `TODO.md` tracking remaining stub endpoints to implement
- Wired `/query/semantic` endpoint to VectorService
- Wired `/query/graph` endpoint to GraphService

### Changed

- **Improved LLM prompts** based on llm-findings.md recommendations:
  - Keyword extraction: temperature 0.0, negative constraints
  - LLM re-ranking: temperature 0.0, explicit rules
  - Conflict detection: temperature 0.0, analysis steps (CoT)
  - Wiki page creation: temperature 0.3, anti-hallucination constraints
  - Page reconstruction: temperature 0.2, preservation constraints
  - Web results analysis: temperature 0.0, conservative approach
- Test fixtures now use configurable host (TEST_HOST) instead of Docker hostnames

### Removed

- Dead code: unused `get_default_user()` function
- Unused imports from routers (wiki.py, graph.py, hybrid_rag.py)
- Stub endpoints shadowed by real implementations (/stats, /ingest/document, /ingest/batch)

## [1.3.2] - 2025-12-22

### Changed

- **Consolidated Ollama model configuration** - All LLM operations now use single `OLLAMA_MODEL` environment variable
  - Removed separate `reranker_model` setting
  - HybridRAG re-ranking, consolidation analysis, and wiki page writing all use the same model
  - Improves VRAM efficiency by keeping one model hot
- Added `OLLAMA_EMBEDDING_MODEL` environment variable for embedding model (previously overloaded `OLLAMA_MODEL`)
- Updated WikiPageWriter to accept settings instead of hardcoded model name

## [1.3.1] - 2025-12-16

### Fixed

- Smart create endpoint missing `content_extractor` dependency causing 500 errors on `POST /wiki/pages/smart-create`

## [1.3.0] - 2025-12-15

### Changed

- **Two-Stage RRF Architecture** - Major refactor to level the playing field between wiki and web results
  - Stage 1: Vector and graph results merged into single "wiki" ranking using mini-RRF
  - Stage 2: Final RRF between wiki (single source) and web (single source)
  - Wiki pages no longer get 2x advantage from appearing in both vector and graph searches
  - Multi-source confirmation still determines wiki internal ranking

- **Skip synonyms in graph search** - LLM-generated synonyms (e.g., "author") no longer match unrelated graph entities (e.g., "author2000")
  - Vector search still uses synonyms for semantic similarity
  - Graph search uses only core keywords for exact entity matching

### Added

- `VECTOR_SIMILARITY_THRESHOLD` config setting (default: 0.7) to filter weak vector matches
- Deduplication in graph search to prevent same document appearing multiple times

### Fixed

- Graph search duplicate entity bug where same document could appear twice if entity linked multiple times

## [1.2.1] - 2025-12-15

### Fixed

- HybridRAG router missing `content_extractor` dependency causing 500 errors on `/query/hybrid` endpoint

## [1.2.0] - 2025-12-15

### Added

- **RAG Search Endpoint** (`POST /rag/search`)
  - Web, news, and image search via SearXNG
  - Full content extraction using Trafilatura (F1 score 0.958)
  - Redis caching with configurable TTL
  - Markdown sources summary for LLM consumption
  - Returns both extracted content and original snippets

- **Content Extraction Endpoints** (`/content/*`)
  - `POST /content/extract` - Extract content from a single URL
  - `POST /content/extract/batch` - Batch extraction (up to 20 URLs)
  - Reusable ContentExtractor client for use across the codebase

- **HybridRAG Content Extraction Enhancement**
  - Web search results now include full extracted content via Trafilatura
  - Falls back to original snippets if extraction fails
  - Improves context quality for LLM re-ranking and consumption

### Changed

- Added new configuration options:
  - `SEARCH_CACHE_TTL` - Search cache TTL in seconds (default: 300)
  - `SEARCH_TIMEOUT` - SearXNG timeout (default: 10s)
  - `CONTENT_EXTRACTION_TIMEOUT` - Per-URL extraction timeout (default: 5s)
  - `CONTENT_MAX_LENGTH` - Max extracted content length (default: 2000)
  - `SEARCH_DEFAULT_LIMIT` - Default search results (default: 10)

### Dependencies

- Added `trafilatura~=1.12.0` for content extraction

## [1.1.3] - 2025-12-14

### Added

- Watchtower update trigger in Gitea workflow after successful build

## [1.1.2] - 2025-12-14

### Fixed

- Updated registry login URL in Gitea workflow (git.schweitz.net → git.schweitz.internal)

## [1.1.1] - 2025-12-14

### Fixed

- Updated container registry tag URLs in Gitea workflow (git.schweitz.net → git.schweitz.internal)

### Added

- Tests for Smart Page Creation feature (`test_smart_create.py`)
  - Model validation tests for WikiSmartCreateRequest/Response
  - WikiService.smart_create_page method tests
  - Bidirectional entity linking utility tests
  - Endpoint validation tests

## [1.1.0] - 2025-12-11

### Added

- **Smart Page Creation Endpoint** (`POST /wiki/pages/smart-create`)
  - Combines HybridRAG research with LLM content generation
  - Searches existing wiki, knowledge graph, and web for topic context
  - Uses WikiPageWriter to synthesize findings into structured wiki content
  - Auto-generates page path from topic if not provided
  - Returns research summary with source counts

- **Bidirectional Entity Linking**
  - New shared utility (`entity_linking_utils.py`) for reusable entity linking
  - Forward links: Links entities mentioned in new pages to existing entity pages
  - Backward links: Updates existing pages that mention the new entity
  - Runs automatically in background after smart page creation

- **Version Management**
  - Added `pyproject.toml` with project metadata and version
  - Version is now read from `pyproject.toml` (single source of truth)
  - Health check endpoint returns current version
  - FastAPI docs show current version

### Changed

- Updated `config.py` to read version from `pyproject.toml`
- Updated `main.py` to use centralized version

## [1.0.0] - 2025-12-10

### Added

- Initial release extracted from portainer-core
- Wiki page management (`/wiki/pages` CRUD endpoints)
- HybridRAG search (`/query/hybrid`) with vector, graph, and web search
- Knowledge graph operations (`/graph/*`)
- Vector search operations (`/vector/*`)
- Knowledge consolidation from search results (`/consolidate/knowledge`)
- Entity linking and extraction
- Wiki.js change listener for auto-processing user edits
- Multi-tenant architecture with user namespace isolation