jpmschweitzer/library-desk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 5 Wiki Activity
84 Commits 1 Branch 34 Tags
c35d3c1fa99bf2e4ea59360e25b50df95bcbef2b
Commit Graph
6 Commits
Author SHA1 Message Date
jpmschweitzerandClaude Fable 5 86051d8022 feat: implement check-updates, job-backed ingest status, and dedup scan 
Replace the four stub endpoints with real implementations, all requiring
an explicit tenant user (Phase B rule):

- /ingest/check-updates: GraphService now records a SHA-256 content_hash
  on every Document node at ingestion time; the endpoint compares those
  stored hashes against current Wiki.js page content in one UNWIND Cypher
  query per tenant and returns changed/new/deleted page lists (entity-stub
  pages excluded, pre-hash-tracking documents flagged stored_hash_missing).
- /ingest/status/{job_id}: backed by the Redis JobManager; jobs are
  tenant-scoped (foreign jobs 404). /ingest/page, /ingest/batch and
  /ingest/all now create job records and return job_id.
- /ingest/repo-status/{repository}: wiki page count vs indexed Document
  nodes under users/{tenant}/{repository} plus tenant job stats.
- /deduplicate/check: tenant-scoped Qdrant similarity scan; chunk pairs
  above ~0.9 cosine from different pages grouped per page pair with best
  score and page references (read-only).

Supporting changes: get_job_manager dependency (+ shutdown close),
scroll_all_points can return vectors, VectorService.find_duplicate_pairs,
src/core/hashing.compute_content_hash. 13 new offline unit tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
 2026-07-14 12:06:21 +02:00
jpmschweitzerandClaude Fable 5 8a1c9ba5f3 fix(security): scope every HybridRAG leg and ingestion path to the caller's tenant 
A live /query/hybrid probe as user=llm_tester returned jpmschweitzer
pages. Audit of all legs (vector, graph, web-persistence, volatile,
documents) plus enrichment/persistence found and fixed these unscoped
paths:

- vector_service.update_from_page and graph_service.update_from_page now
  refuse pages outside users/{user}/ - previously any tenant could
  ingest any wiki page (incl. another tenant's) into its own collection
  and graph labels, which is how foreign content entered the vector leg.
- ingestion_service.ingest_all_pages clamps path_prefix to the caller's
  namespace (segment-exact, sanitized comparison) and defaults to
  users/{user}; /ingest/all returns 400 on cross-tenant prefixes.
- hybrid_rag_service._persist_search_for_librarian linked SearchQuery
  nodes to unscoped (d:Document {page_id}); now matches only
  User_{Tenant}_Document nodes.
- graph_service: _get_entity_mention_count, entity-stub mention/related
  queries, generate_entity_stubs, find/purge_orphan_entities matched
  unscoped Document nodes; cleanup_broken_relationships matched all
  tenants' SearchQuery nodes; _entity_has_wiki_page listed all wiki
  pages. All are now tenant-label / namespace scoped.
- volatile_service collection names now use the sanitized user id.
- is_path_in_user_namespace enforces a path-segment boundary
  (users/llm_tester2 is not llm_tester's namespace) and treats
  hyphen/underscore tenant spellings as the same sanitized tenant.
- New offline unit tests per leg (mocked clients) assert the
  tenant-scoped collection/label/path is used and cross-tenant access
  is refused.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-14 11:15:43 +02:00
jpmschweitzerandClaude Opus 4.5 1fb1f2a636 fix: include paperless_id in chunk references 
get_all_chunk_references was missing paperless_id field needed for
Paperless orphan detection.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-25 17:11:01 +01:00
jpmschweitzerandClaude Opus 4.5 983a934b85 feat: add Paperless orphan cleanup endpoint 
- POST /maintenance/cleanup/paperless - detect and clean orphaned Paperless documents
- Checks indexed documents against Paperless API
- Removes vectors and graph nodes for deleted documents
- Supports dry_run mode for preview

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-25 17:00:56 +01:00
jpmschweitzerandClaude Opus 4.5 f756ca9490 feat: add maintenance system with index reconciliation 
Complete maintenance subsystem for index health and cleanup:

Endpoints:
- GET /maintenance/health - lightweight (or detailed) health check
- POST /maintenance/cleanup/all - full orphan cleanup
- POST /maintenance/cleanup/vectors - purge orphan vector chunks
- POST /maintenance/cleanup/graph - purge orphan graph nodes
- POST /maintenance/reconcile-index - cleanup + reindex missing pages

Bidirectional orphan detection:
- find_documents_without_vectors() in GraphService
- find_chunks_without_graph_nodes() in VectorService

Redis integration:
- Tracks last_cleanup timestamp for scheduler visibility

Config additions:
- Document store, volatile cache, and maintenance settings
- VectorServiceDep and GraphServiceDep type aliases

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-24 16:22:39 +01:00
jpmschweitzer 95852190ba Initial commit: library-desk service extraction from portainer-core
Build and Push / build (release) Successful in 36s
Details
2025-12-11 17:28:23 +01:00