feat: implement check-updates, job-backed ingest status, and dedup scan
Replace the four stub endpoints with real implementations, all requiring
an explicit tenant user (Phase B rule):
- /ingest/check-updates: GraphService now records a SHA-256 content_hash
on every Document node at ingestion time; the endpoint compares those
stored hashes against current Wiki.js page content in one UNWIND Cypher
query per tenant and returns changed/new/deleted page lists (entity-stub
pages excluded, pre-hash-tracking documents flagged stored_hash_missing).
- /ingest/status/{job_id}: backed by the Redis JobManager; jobs are
tenant-scoped (foreign jobs 404). /ingest/page, /ingest/batch and
/ingest/all now create job records and return job_id.
- /ingest/repo-status/{repository}: wiki page count vs indexed Document
nodes under users/{tenant}/{repository} plus tenant job stats.
- /deduplicate/check: tenant-scoped Qdrant similarity scan; chunk pairs
above ~0.9 cosine from different pages grouped per page pair with best
score and page references (read-only).
Supporting changes: get_job_manager dependency (+ shutdown close),
scroll_all_points can return vectors, VectorService.find_duplicate_pairs,
src/core/hashing.compute_content_hash. 13 new offline unit tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
This commit is contained in:
@@ -452,14 +452,20 @@ class GraphService:
|
||||
user_base_label = get_neo4j_user_base_label(user) # For entities
|
||||
user_doc_label = get_neo4j_user_label(user) # For documents
|
||||
|
||||
# Create/update Document node
|
||||
# Create/update Document node.
|
||||
# content_hash records the fingerprint of the ingested content so
|
||||
# /ingest/check-updates can detect changed pages without re-reading
|
||||
# the graph's source content.
|
||||
from src.core.hashing import compute_content_hash
|
||||
|
||||
doc_query = f"""
|
||||
MERGE (d:{user_doc_label}:Document {{page_id: $page_id}})
|
||||
SET d.title = $title,
|
||||
d.path = $path,
|
||||
d.tags = $tags,
|
||||
d.updated_at = datetime(),
|
||||
d.content_length = $content_length
|
||||
d.content_length = $content_length,
|
||||
d.content_hash = $content_hash
|
||||
RETURN d
|
||||
"""
|
||||
|
||||
@@ -468,7 +474,8 @@ class GraphService:
|
||||
"title": page.get("title"),
|
||||
"path": page.get("path"),
|
||||
"tags": tags,
|
||||
"content_length": len(content)
|
||||
"content_length": len(content),
|
||||
"content_hash": compute_content_hash(content)
|
||||
})
|
||||
|
||||
nodes_created = 1 # Document node
|
||||
|
||||
Reference in New Issue
Block a user