Commit Graph
85 Commits
Author SHA1 Message Date
jpmschweitzerandClaude Fable 5 8348b4bf92 perf: batch page embeddings via /api/embed and upsert before pruning stale points
- embed_batch now issues one batched /api/embed request (the old loop
  made one /api/embeddings round-trip per chunk) with a per-text
  fallback that preserves None-for-failed semantics
- update_from_page embeds all chunks in that single call and stores
  them in one Qdrant batch upsert (upsert_points)
- reindex order reversed: upsert new points first, then prune stale ids
  (deterministic uuid5 ids make overwrite safe) so a mid-way failure no
  longer leaves the page with zero vectors
- VectorUpdateSummary gains status (success/partial/failed) and
  chunks_skipped; all-embeddings-failed keeps old vectors and reports
  failure instead of success=True

Measured on a 7-chunk page ingest (local server, llm_tester):
~375ms -> ~181ms median over 3 runs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
2026-07-14 13:09:37 +02:00
jpmschweitzerandClaude Fable 5 c35d3c1fa9 fix: await ensure_collection and survive partial embed failures in document sync
- ensure_collection was called without await, so the coroutine never ran
  and fresh tenants had no collection when the upsert hit Qdrant
- a single None entry from embed_batch poisoned the point batch and
  aborted the whole document upsert; failed chunks are now skipped with
  a warning (all-failed raises and the IndexResult reports failure)
- raw client.delete/client.upsert calls now go through the async wrapper
  (delete_by_filter and the new batch upsert_points method)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
2026-07-14 13:00:59 +02:00
jpmschweitzerandClaude Fable 5 406143e7ae perf: switch Qdrant to AsyncQdrantClient with explicit timeout
Every vector call ran on the sync QdrantClient inside async wrapper
methods, blocking the FastAPI event loop per Qdrant round-trip. The
wrapper now holds an AsyncQdrantClient (timeout via QDRANT_TIMEOUT,
default 30s) and awaits all client calls; the wrapper API is unchanged.

Call sites off the wrapper were fixed too: the HybridRAG document leg
now uses the async search_vectors wrapper instead of the deprecated raw
client.search (also fixing its call to the nonexistent ollama.embed_text
which made the leg permanently report 'failed'), the health check awaits
get_collections, and document_sync's raw delete/upsert calls are awaited
(routed through wrappers in the next commit).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
2026-07-14 12:58:01 +02:00
jpmschweitzerandClaude Fable 5 0b346d3a57 feat: add job cleanup loop, Scheduler task definitions, and registrar
- job_cleanup_loop (src/jobs/job_manager.py): hourly in-process pass over
  JobManager.cleanup_expired_jobs, started at app startup and cancelled
  at shutdown; Redis job payloads auto-expire but set memberships do not.
- docs/scheduler-tasks.md: the four production Scheduler task payloads
  for the deploy checklist - nightly integrity check 04:30, weekly
  quality report Sunday 03:00 (day_of_week=6, 0=Monday), daily Paperless
  orphan-cleanup 05:00 hitting the existing
  /maintenance/cleanup/paperless?user=jpmschweitzer&dry_run=false
  endpoint, and disabling test_example_task - with exact HTTP bodies
  (explicit user=jpmschweitzer, Authorization: Bearer ${LIBRARY_API_KEY}
  placeholder).
- scripts/register_scheduler_tasks.py: reads SCHEDULER_URL from env,
  DRY-RUN BY DEFAULT (prints the exact payloads, provably contacts
  nothing), --execute gated and requiring LIBRARY_API_KEY to fill the
  placeholder. NOT executed - definitions delivered for the deploy
  checklist only.

9 new offline tests (loop passes/error-resilience/cancellation, payload
schedules, explicit production user, placeholder, dry-run default).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
2026-07-14 12:30:27 +02:00
jpmschweitzerandClaude Fable 5 51f9ce08ec fix: stop consolidation from consuming searches when the LLM is down
ROOT CAUSE (investigated read-only against prod): the production
container sets OLLAMA_MODEL=nomic-embed-text (the embedding model), which
the pre-rename generation setting also read, so every consolidation
/api/generate call failed with HTTP 400 ('"nomic-embed-text" does not
support generate' - confirmed in prod logs and by a direct Ollama probe).
_classify_web_results_unified swallowed that as an empty classification,
and consolidate_knowledge marked EVERY SearchQuery processed anyway -
permanently draining the queue with zero pages ever created. Live Neo4j
shows 197/200 SearchQuery nodes processed=true with no output; every
subsequent 30-minute run then logged 'No unprocessed searches found'.
The label/tenant scoping was NOT at fault: persistence writes both the
tenant label and the plain :SearchQuery label the loop matches on.

The model resolution itself was already fixed in Phase A (94482bc,
ollama_llm_model / OLLAMA_LLM_MODEL). This commit repairs the pipeline
defect that masked it:

- LLM infrastructure failure (no output from generate) now raises
  ConsolidationLLMUnavailableError instead of returning an empty routing
- consolidate_knowledge leaves those searches UNPROCESSED for the next
  run, aborts the rest of the batch (the LLM is down for all of them),
  and reports searches_deferred
- unparseable-but-present model output is still consumed (avoids
  retrying a bad prompt forever); low-web skips unchanged
- every run logs 'Consolidation run complete: searches_processed=N
  searches_deferred=M duration_ms=X'; both fields added to the response
- lookback boundary is now timezone-aware UTC (Neo4j datetime() reads
  naive strings as UTC, shifting the window on CET hosts)

10 new offline regression tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
2026-07-14 12:26:14 +02:00
jpmschweitzerandClaude Fable 5 191a8be6c5 feat: add weekly quality-report endpoint writing dated wiki report
POST /maintenance/quality-report {user}:
- runs the tenant-scoped duplicate scan (cosine >= threshold, default 0.9)
- flags stale pages: not updated in stale_days AND <= max_search_hits
  SearchQuery FOUND hits from the tenant's graph data
- lists pages missing tags/description (report subtree exempt)
- folds in the latest integrity-check results (Redis cache from
  /maintenance/integrity-check, or run inline when absent)
- writes the dated report to users/{user}/system/quality-reports/YYYY-MM-DD
  via the existing wiki write path; same-day reruns update the same page
  (page id remembered in Redis because the Wiki.js listing lags creation)
- response returns the full markdown report + page path + counts +
  duration_ms

Also fixes WikiJSClient.update_page: Wiki.js 2.x requires tags on the
update mutation (server maps over it unconditionally); calls without tags
failed with "Cannot read properties of undefined (reading 'map')" -
which also silently broke the consolidation page-update path. Current
tags are now preserved when the caller supplies none.

Verified live end-to-end on the local dev server as llm_tester
(tests/test_quality_report_live.py, integration-marked): probe page
flagged for missing metadata, report page written and fetched back,
same-day rerun updates in place, teardown leaves zero llm_tester pages.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
2026-07-14 12:22:19 +02:00
jpmschweitzerandClaude Fable 5 77dc5b00a1 fix: grow Wiki.js listing limit until page count stabilizes
Live evidence during quality-report verification: pages.list(limit=100)
returned 43 pages while 140 existed; limit=500 returned all 140. Wiki.js
applies the limit BEFORE its own visibility filtering, so a response with
fewer pages than requested does NOT prove the listing is complete. The
Phase A limit-growth loop stopped on len < limit and silently truncated
listings (page counts, orphan cleanups, integrity scans, and the quality
report all consume this listing).

The loop now doubles the limit until the returned count stops increasing
(fixed point), at the cost of one confirming fetch. Offline pagination
tests updated, including a regression test simulating the pre-filter
limit behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
2026-07-14 12:19:35 +02:00
jpmschweitzerandClaude Fable 5 8f56b78be7 feat: add read-only nightly integrity-check maintenance endpoint
POST /maintenance/integrity-check {user} reports per tenant, without
ever fixing anything:
- wiki pages with ZERO vectors in Qdrant (silent-skip reindex victims)
- orphaned vectors whose wiki page no longer exists
- unexpected Qdrant collections vs known tenant patterns (test-tenant
  residue and unknown namespaces flagged; foreign services counted)
- Neo4j Document nodes without wiki counterparts
- counts + duration_ms

The latest report is cached in Redis (library:integrity:latest:{user},
30-day TTL) so the weekly quality report can fold it in. Explicit user
required per Phase B. Offline tests assert the report contents, the
collection classification rules, and that no destructive client method
is ever invoked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
2026-07-14 12:08:40 +02:00
jpmschweitzerandClaude Fable 5 86051d8022 feat: implement check-updates, job-backed ingest status, and dedup scan
Replace the four stub endpoints with real implementations, all requiring
an explicit tenant user (Phase B rule):

- /ingest/check-updates: GraphService now records a SHA-256 content_hash
  on every Document node at ingestion time; the endpoint compares those
  stored hashes against current Wiki.js page content in one UNWIND Cypher
  query per tenant and returns changed/new/deleted page lists (entity-stub
  pages excluded, pre-hash-tracking documents flagged stored_hash_missing).
- /ingest/status/{job_id}: backed by the Redis JobManager; jobs are
  tenant-scoped (foreign jobs 404). /ingest/page, /ingest/batch and
  /ingest/all now create job records and return job_id.
- /ingest/repo-status/{repository}: wiki page count vs indexed Document
  nodes under users/{tenant}/{repository} plus tenant job stats.
- /deduplicate/check: tenant-scoped Qdrant similarity scan; chunk pairs
  above ~0.9 cosine from different pages grouped per page pair with best
  score and page references (read-only).

Supporting changes: get_job_manager dependency (+ shutdown close),
scroll_all_points can return vectors, VectorService.find_duplicate_pairs,
src/core/hashing.compute_content_hash. 13 new offline unit tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
2026-07-14 12:06:21 +02:00
jpmschweitzerandClaude Fable 5 eae39aff3e chore: untrack PROJECT_CLAUDIFICATION_HANDOVER.md
The pre-existing untracked handover note was swept into 84e9185 by a
broad git add; restore it to its untracked working-tree state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 11:31:07 +02:00
jpmschweitzerandClaude Fable 5 79dfff6811 feat: add dry-run-first purge tooling for test-tenant residue
scripts/purge_test_artifacts.py removes confirmed test residue from the
shared stores:

- Qdrant: library_desk_llm_tester, test_user, library_desk_test_user,
  memories_llm_tester, volatile_llm_tester, core_ai_user_test_* and any
  collection containing llm_tester / llm-tester
- Neo4j: nodes labelled User_Llm_Tester* (SearchQuery/Document/WebResult
  and sub-tenants) plus legacy llm-tester Document nodes matched by
  users/llm* path
- Redis: *llm_tester* / *llm-tester* keys on the service DB

Safety: --dry-run is the DEFAULT (prints identifiers and counts only);
--execute is required for real deletion; the script exits fatally if a
target rule ever matches a jpmschweitzer-namespaced identifier; the
snapshot prerequisite (Qdrant snapshot API, neo4j-admin database dump)
is documented in the module docstring. Connection settings come from
the repo .env; secrets are never printed.

Verified with a read-only --dry-run against the live stores.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 11:26:59 +02:00
jpmschweitzerandClaude Fable 5 fe8e00e59c test: add live tenant-isolation integration test for /query/hybrid
Guard-gated (RUN_INTEGRATION_TESTS=1 + reserved-tenant guard) test that
runs against the local wakeup server (8778, never the production
container on 8089) with the shared backing services:

- creates a wiki page with a unique marker and ingests it (vectors +
  graph) as llm_tester,
- /query/hybrid as llm_tester must return the tenant's own page and
  ZERO results from the jpmschweitzer tenant (paths, sources, and the
  formatted LLM context are all checked),
- /query/hybrid as a third nonexistent tenant (llm_tester_void, inside
  the reserved namespace so even its persisted SearchQuery stays in
  test space - nothing is ever written as jpmschweitzer) must return
  zero results entirely, on both the marker query and a broad query,
- module teardown deletes the created page; the conftest session
  teardown purges all remaining llm_tester artifacts.

Verified live: 3 passed in 23.55s; post-run checks show 0 *_llm_tester
Qdrant collections, 0 User_Llm_Tester* Neo4j nodes, and 0 wiki pages
under users/llm_tester.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 11:25:16 +02:00
jpmschweitzerandClaude Fable 5 b2399de5f9 test: pin suite to reserved llm_tester tenant with guard and teardown
Rewrite tests/conftest.py for the shared-services testing model where
tenancy is the only isolation wall:

- Remove the hardcoded production host default (192.168.86.149):
  TEST_HOST env with a safe localhost default; LIBRARY_DESK_URL selects
  the local wakeup server (8778), never the production container (8089).
- Pin the suite to the reserved test tenant llm_tester (TEST_TENANT may
  only choose a tenant inside the reserved llm_tester* namespace).
- Session guard (autouse) hard-aborts the whole run if the effective
  tenant is jpmschweitzer or outside the reserved namespace.
- Integration-marked tests only run with RUN_INTEGRATION_TESTS=1 and a
  passing guard; they are skipped otherwise.
- Session-scoped teardown deletes ALL llm_tester artifacts created
  during the run: Qdrant *_llm_tester collections, Neo4j
  User_Llm_Tester* nodes, wiki subtree users/llm_tester (and hyphen
  variant), llm_tester Redis keys on the service DB - with hard
  assert_safe_test_tenant() checks before every delete. Uses a sync
  fixture + asyncio.run to avoid the session loop-scope mismatch.
- Legacy tests/test_integration.py marked integration and pinned to the
  test tenant (taxonomy/list reads no longer touch the production
  namespace; Qdrant tests use the tenant-scoped collection name).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 11:21:23 +02:00
jpmschweitzerandClaude Fable 5 33e62db6a2 fix: ensure the tenant-scoped collection in upsert_document_chunks
ensure_collection() was called with the raw user string while the
upsert targeted get_collection_name(user), creating stray bare-name
collections (e.g. 'test_user') and failing the actual upsert when the
scoped collection did not exist yet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 11:17:36 +02:00
jpmschweitzerandClaude Fable 5 2d8cccaeaa test: align volatile TTL expectations with doubled namespace TTLs
The TTLs were doubled in 4cfad2e (v1.7.2) to survive missed scheduler
runs, but the unit tests kept the old expectations and have been failing
since. Update weather (7200), financial (600), and sports (120)
assertions to the current defaults.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 11:16:27 +02:00
jpmschweitzerandClaude Fable 5 8a1c9ba5f3 fix(security): scope every HybridRAG leg and ingestion path to the caller's tenant
A live /query/hybrid probe as user=llm_tester returned jpmschweitzer
pages. Audit of all legs (vector, graph, web-persistence, volatile,
documents) plus enrichment/persistence found and fixed these unscoped
paths:

- vector_service.update_from_page and graph_service.update_from_page now
  refuse pages outside users/{user}/ - previously any tenant could
  ingest any wiki page (incl. another tenant's) into its own collection
  and graph labels, which is how foreign content entered the vector leg.
- ingestion_service.ingest_all_pages clamps path_prefix to the caller's
  namespace (segment-exact, sanitized comparison) and defaults to
  users/{user}; /ingest/all returns 400 on cross-tenant prefixes.
- hybrid_rag_service._persist_search_for_librarian linked SearchQuery
  nodes to unscoped (d:Document {page_id}); now matches only
  User_{Tenant}_Document nodes.
- graph_service: _get_entity_mention_count, entity-stub mention/related
  queries, generate_entity_stubs, find/purge_orphan_entities matched
  unscoped Document nodes; cleanup_broken_relationships matched all
  tenants' SearchQuery nodes; _entity_has_wiki_page listed all wiki
  pages. All are now tenant-label / namespace scoped.
- volatile_service collection names now use the sanitized user id.
- is_path_in_user_namespace enforces a path-segment boundary
  (users/llm_tester2 is not llm_tester's namespace) and treats
  hyphen/underscore tenant spellings as the same sanitized tenant.
- New offline unit tests per leg (mocked clients) assert the
  tenant-scoped collection/label/path is used and cross-tenant access
  is refused.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 11:15:43 +02:00
jpmschweitzerandClaude Fable 5 a4c299bf9b fix(security)!: make raw Cypher endpoints read-only with write-clause denylist
/query/graph scoping was a documented no-op (graph_service returned the
query unscoped) and neo4j_client permitted writes; a live probe showed a
nonexistent user could read the whole graph.

- Add Neo4jClient.execute_read() that opens the session with
  default_access_mode=READ_ACCESS so the database refuses writes even if
  validation is bypassed.
- GraphService.execute_query() now rejects queries containing
  CREATE/MERGE/DELETE/DETACH/SET/REMOVE/DROP/FOREACH/LOAD or any CALL
  (conservative word-boundary denylist on the uppercased query) and
  executes through the read-only session; the no-op _scope_query_to_user
  is removed.
- Remove the false user-scoping claims from /query/graph (main.py) and
  /graph/query docs and the CypherQueryRequest model: the endpoints are
  documented as admin/debug, unscoped read-only (per-tenant label
  injection for arbitrary Cypher would need a real parser; /graph/nodes
  remains the tenant-scoped path).
- Offline unit tests: denylist coverage (incl. lowercase/multiline/CALL),
  word-boundary false-positive check, and READ_ACCESS session assertion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 11:08:40 +02:00
jpmschweitzerandClaude Fable 5 84e9185371 feat!: require explicit user on every tenant-data endpoint
Remove the implicit jpmschweitzer default tenant (DEFAULT_USER) from
src/core/multi_tenancy.py and every endpoint and request model that
inherited it (~40 endpoints across /query, /wiki, /vector, /graph,
/ingest, /volatile, /documents, /stats, /rag).

- Add validate_required_user() + RequiredUser pydantic type in
  multi_tenancy and a shared require_user FastAPI dependency
  (RequiredUserQuery) that rejects missing, empty, and whitespace-only
  users with 422, following the /maintenance/* pattern.
- Wiki page create / smart-create / dossier request models now require
  user (no fallback in wiki_service).
- /maintenance/cleanup/test-data derives the tenant from the page path
  instead of using the production tenant collection.
- Wiki.js change listener skips changes when no tenant user can be
  derived from the notification email instead of defaulting to the
  production tenant.
- Consolidation service internal helpers no longer default to the
  production tenant.
- Tool catalog marks user as required with honest descriptions.
- OpenAPI descriptions updated honestly; CHANGELOG notes that callers
  (tatlock, Scheduler ingest tasks) must now send explicit user.
- Offline tests: 422 coverage for query/body endpoints, required-user
  validator tests; updated legacy tests that assumed a default tenant.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 11:06:11 +02:00
jpmschweitzerandClaude Fable 5 a66d801abd test: add offline unit tests for Phase A reliability fixes
Mock-based tests (no live services) covering:
- ollama_llm_model resolution under the OLLAMA_MODEL env collision
- per-leg retrieval failure -> source_status/degraded signaling
- Phase 0/Phase 4 LLM timeout fallbacks
- /stats wiki page count using the users/{user} path prefix
- Wiki.js list_all_pages limit-growth pagination and
  list_pages limit-after-filter behavior

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 10:05:51 +02:00
jpmschweitzerandClaude Fable 5 54204599a9 fix: correct /stats wiki page count and Wiki.js listing pagination
- /stats passed the bare user name as path prefix (matched nothing, always
  reported 0 of 138 pages); it now counts pages under users/{user}
- list_pages applied the API-side limit before client-side path/tag filters,
  dropping matching pages that sort late; limit now applies after filtering
- list_all_pages replaced the fake while/break pagination with a real
  limit-growth loop: Wiki.js 2.x pages.list supports only a limit argument
  (no offset - verified via GraphQL introspection), so the client doubles
  the limit until the API returns fewer pages than requested

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 10:03:13 +02:00
jpmschweitzerandClaude Fable 5 5c2284922d feat: add source_status and degraded fields to HybridRAG response
Each retrieval leg (vector, graph, web, volatile, documents) now returns
(results, timing, error) instead of swallowing exceptions to an empty list.
The response reports per-leg status ('ok'/'failed'/'disabled') in
source_status and sets degraded=true when any enabled leg failed. Failed
legs still contribute no results (behavior unchanged) and are logged at
WARNING. Both fields are additive with defaults, so deploy order relative
to consumers does not matter.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 10:01:04 +02:00
jpmschweitzerandClaude Fable 5 94482bcb59 fix: rename generation model setting to ollama_llm_model to avoid OLLAMA_MODEL env collision
The deployed container sets OLLAMA_MODEL=nomic-embed-text for embeddings,
which shadowed the generation-model setting and broke Phase 0 keyword
extraction and Phase 4 LLM re-ranking on every request. The setting is now
ollama_llm_model (env: OLLAMA_LLM_MODEL, default gemma4:e2b), startup logs
the resolved generation model, and Phase 0/Phase 4 LLM calls are wrapped in
a 12s asyncio.wait_for with graceful fallback so a hung call cannot gate
retrieval for the full 120s client timeout.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 09:57:57 +02:00
jpmschweitzerandClaude Opus 4.5 d0e712760d chore: improve check_environment.py script
- Filter out expired records using ttl_expiry
- Add namespace: prefix to headers for clarity

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 18:53:03 +01:00
jpmschweitzerandClaude Opus 4.5 31486018a5 fix: update endpoint TTL defaults to match namespace TTLs
Build and Push / release (push) Successful in 3s
Build and Push / build (push) Successful in 1m16s
All fetch endpoints now use 2x refresh interval as default TTL.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
v1.7.3
2026-01-08 17:35:28 +01:00
jpmschweitzerandClaude Opus 4.5 4cfad2e8bc fix: double volatile TTLs to survive missed scheduler runs
Build and Push / release (push) Successful in 3s
Build and Push / build (push) Successful in 2m13s
TTL = 2x refresh interval ensures data remains valid even if a
scheduled refresh is delayed or fails.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
v1.7.2
2026-01-08 16:54:18 +01:00
jpmschweitzerandClaude Opus 4.5 1e31e74ad6 fix: update CI workflow to trigger on tag push
Build and Push / release (push) Successful in 3s
Build and Push / build (push) Successful in 1m14s
Matches core-api workflow pattern for auto release/build on version tags.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
v1.7.1
2026-01-07 12:05:31 +01:00
jpmschweitzerandClaude Opus 4.5 72f515bf61 feat: add combined environment endpoint for concurrent weather + air quality fetch
Build and Push / release (release) Failing after 3s
Build and Push / build (release) Successful in 1m19s
- POST /volatile/fetch/environment/{city} fetches both in parallel
- Single geocode lookup shared between API calls
- Uses asyncio.gather() for concurrent external requests
- Fix scheduler executor name (rest_api → rest_api_executor)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
v1.7.0
2026-01-07 11:53:46 +01:00
jpmschweitzer 0c085d603e auto release/build on version tag 2026-01-03 20:39:00 +01:00
jpmschweitzerandClaude Opus 4.5 152b2f28c4 release: v1.6.2 - Stats endpoint, weather/forecast separation
Build and Push / build (release) Successful in 30s
- GET /stats endpoint with Neo4j, Qdrant, Wiki.js, Paperless stats
- Split weather into current (1hr TTL) and forecast (12hr TTL)
- New FORECAST namespace for multi-day outlook

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
v1.6.2
2025-12-30 12:50:44 +01:00
jpmschweitzerandClaude Opus 4.5 46b9bcd7a0 feat: separate current weather from forecast into distinct namespaces
- Add FORECAST namespace for multi-day outlook (12hr TTL)
- WEATHER namespace now stores only current conditions (1hr TTL)
- Split fetch_weather into fetch_current_weather + fetch_forecast
- Add POST /volatile/fetch/forecast/{city} endpoint
- Different update frequencies for efficient caching

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 12:44:16 +01:00
jpmschweitzerandClaude Opus 4.5 68eb1add3d feat: add GET /stats endpoint with system statistics
Returns counts for:
- Neo4j: nodes by type (Document, Entity, Collection, Search)
- Qdrant: vectors per collection
- Wiki.js: total page count
- Paperless: documents, tags, correspondents, document types

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 12:07:02 +01:00
jpmschweitzerandClaude Opus 4.5 d6b30570a0 release: v1.6.1 - Weather forecasts, sun times, air quality
Build and Push / build (release) Successful in 52s
- Weather fetch now returns 7-day forecasts with UV index
- New /volatile/fetch/sun/{city} endpoint for sunrise/sunset
- New /volatile/fetch/air_quality/{city} endpoint for AQI and pollutants
- OpenMeteoProvider now implements AirQualityProvider interface

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
v1.6.1
2025-12-30 10:13:06 +01:00
jpmschweitzerandClaude Opus 4.5 6b0530ed79 fix: remove dead automated user filtering code
The _is_automated_user method was never called - loop prevention is
handled by debouncing instead. User email filtering was intentionally
removed because the notification email is the page CREATOR, not editor.

- Remove unused _is_automated_user method
- Update test to verify notifications are processed regardless of user
- Remove obsolete test_automated_user_filtering test

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-29 21:47:55 +01:00
jpmschweitzerandClaude Opus 4.5 0a8c2639a0 docs: update MEMORY_REMEMBER_PLAN with implementation status
Mark all phases as complete (v1.5.0-v1.6.0):
- Settings DB, Phase A, B, C all implemented
- Updated files summary with actual implementations
- Added remaining work section for file upload placeholder

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-29 21:33:10 +01:00
jpmschweitzerandClaude Opus 4.5 943fcd9bf9 release: v1.6.0 - Memory system with scheduler integration
Build and Push / build (release) Successful in 1m20s
Complete three-tier memory architecture:
- Volatile fetch endpoints for scheduler-driven prefetch
- Unified memory routing in consolidation service
- Paperless document recall in HybridRAG
- External scheduler integration for prefetch tasks

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
v1.6.0
2025-12-29 21:00:35 +01:00
jpmschweitzerandClaude Opus 4.5 910b289c9e feat: integrate external scheduler for prefetch task registration
Add SchedulerClient to communicate with external scheduler service for
registering volatile prefetch tasks discovered during HybridRAG searches.

- Add scheduler_client.py with full REST API for task CRUD operations
- Add scheduler_url config setting (default: http://scheduler:8090)
- Update consolidation service to use scheduler for prefetch registration
- Add scheduler health checks to startup/shutdown lifecycle

When HybridRAG classifies web content as prefetch-worthy, it now creates
scheduled tasks that periodically refresh the volatile cache via the
external scheduler service.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-29 20:42:37 +01:00
jpmschweitzerandClaude Opus 4.5 c01033505b feat: add Paperless document recall to HybridRAG
Phase C of memory system: Documents are now a retrieval source alongside
wiki, volatile, and web search.

Changes:
- Add enable_documents, document_limit, document_threshold to HybridRAGConfig
- Add paperless_id field to HybridRAGResult
- Add document_ms timing to TimingBreakdown
- Add document search to parallel retrieval (filters doc_type=document)
- Update RRF fusion to include documents as fourth source
- Add document metadata (correspondent, document_type, tags) to results

HybridRAG now searches 4 sources in parallel:
- Wiki (vector + graph merged)
- Volatile cache (priority boost)
- Paperless documents (new)
- Web search

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-29 15:11:11 +01:00
jpmschweitzerandClaude Opus 4.5 1f47b052d8 feat: add unified memory routing to consolidation service
- Add MemoryRouteClassification and MemoryRoutingResult models
- Implement unified classifier (_classify_web_results_unified) that routes
  web results to: wiki, volatile, file (Paperless), prefetch, or skip
- Add routing methods: _route_to_volatile, _route_to_files, _register_prefetch
- Update _process_search to use unified classifier instead of separate analysis
- Add get_volatile_cache_service factory to dependencies
- Wire volatile_service and settings_client into ConsolidationService
- Update ConsolidationResult/Response with new routing counters

Test fixes:
- Fix WikiJSClient fixtures to use api_token instead of username/password
- Fix entity linking test assertions to expect full user-namespaced paths
- Add sample_unified_classification fixture for new classifier format

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 17:19:20 +01:00
jpmschweitzerandClaude Opus 4.5 ab892745fa feat: add volatile fetch endpoints for scheduler-driven prefetch
- Add VolatileFetchService to orchestrate API fetch and cache storage
- Add POST /volatile/fetch/weather/{city} endpoint
- Add POST /volatile/fetch/news/{category} endpoint
- Add POST /volatile/fetch/stock/{symbol} endpoint
- Add POST /volatile/fetch/crypto/{symbol} endpoint

Endpoints integrate with external API providers (OpenMeteo, NOS/BBC,
AlphaVantage) and store results in volatile cache with configurable TTL.
Designed for scheduler cron jobs to prefetch user-relevant data.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-26 13:40:40 +01:00
jpmschweitzerandClaude Opus 4.5 2b8c229f53 release: v1.5.0 - External API providers and central settings
Build and Push / build (release) Successful in 1m10s
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
v1.5.0
2025-12-26 12:24:19 +01:00
jpmschweitzerandClaude Opus 4.5 5d4a8dba95 feat: add external API providers and central settings database
- Add central settings database client (system_settings on postgres-shared)
  - User-scoped settings with global fallback
  - API config storage with enabled/disabled toggle
  - Per-source category filtering for news

- Add modular external API providers in src/apis/:
  - OpenMeteoProvider: weather with geocoding (free, no key)
  - NOSProvider: Dutch news RSS feeds
  - BBCProvider: English news RSS feeds
  - AggregatedNewsProvider: merges sources with category filtering
  - AlphaVantageProvider: financial quotes (API key from settings DB)

- Add provider dependencies and lifecycle management
- Add requirements-dev.txt with pip-audit for security auditing
- Add MEMORY_REMEMBER_PLAN.md documenting volatile/document memory architecture

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-26 12:07:58 +01:00
jpmschweitzerandClaude Opus 4.5 99eefa291c release: v1.4.9 - fix paperless_id in chunk references
Build and Push / build (release) Successful in 28s
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
v1.4.9
2025-12-25 17:12:08 +01:00
jpmschweitzerandClaude Opus 4.5 1fb1f2a636 fix: include paperless_id in chunk references
get_all_chunk_references was missing paperless_id field needed for
Paperless orphan detection.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-25 17:11:01 +01:00
jpmschweitzerandClaude Opus 4.5 9e7d8394f3 release: v1.4.8 - Paperless orphan cleanup
Build and Push / build (release) Successful in 29s
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
v1.4.8
2025-12-25 17:04:24 +01:00
jpmschweitzerandClaude Opus 4.5 983a934b85 feat: add Paperless orphan cleanup endpoint
- POST /maintenance/cleanup/paperless - detect and clean orphaned Paperless documents
- Checks indexed documents against Paperless API
- Removes vectors and graph nodes for deleted documents
- Supports dry_run mode for preview

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-25 17:00:56 +01:00
jpmschweitzerandClaude Opus 4.5 6d5760c297 release: v1.4.7 - Paperless custom field fix
Build and Push / build (release) Successful in 29s
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
v1.4.7
2025-12-25 16:40:28 +01:00
jpmschweitzerandClaude Opus 4.5 867de65354 fix: use field ID for Paperless custom field updates
Paperless API requires field ID (integer) not field name (string)
when updating custom fields. Now looks up field ID by name before
updating library_indexed custom field.

Also includes webhook debugging endpoint for development.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-25 16:37:26 +01:00
jpmschweitzerandClaude Opus 4.5 f2b8c7d111 fix: update Paperless webhook payload to match include_document format
Build and Push / build (release) Successful in 30s
- Change model field from document_id to id (Paperless sends id)
- Add content, created, modified, added, original_file_name, owner fields
- Add extra="ignore" config to handle additional Paperless fields
- Update sync service to use content from webhook payload
- Skip Paperless API call when content already provided

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
v1.4.6
2025-12-25 14:38:40 +01:00
jpmschweitzer f4352841a2 Merge feature/document-storage: Paperless-ngx integration
Build and Push / build (release) Successful in 30s
v1.4.5
2025-12-25 14:17:11 +01:00
jpmschweitzerandClaude Opus 4.5 4ff3fc4c7a feat: add Paperless-ngx document storage integration
- Add /documents router with webhook, upload, search, health endpoints
- Create DocumentSyncService for indexing documents to vectors/graph
- Add PaperlessClient for REST API integration
- Configure dependency injection for Paperless client
- Add document models for webhook payloads and responses
- Event-driven architecture via Paperless workflow webhooks

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-25 14:16:40 +01:00