feat: add weekly quality-report endpoint writing dated wiki report
POST /maintenance/quality-report {user}:
- runs the tenant-scoped duplicate scan (cosine >= threshold, default 0.9)
- flags stale pages: not updated in stale_days AND <= max_search_hits
SearchQuery FOUND hits from the tenant's graph data
- lists pages missing tags/description (report subtree exempt)
- folds in the latest integrity-check results (Redis cache from
/maintenance/integrity-check, or run inline when absent)
- writes the dated report to users/{user}/system/quality-reports/YYYY-MM-DD
via the existing wiki write path; same-day reruns update the same page
(page id remembered in Redis because the Wiki.js listing lags creation)
- response returns the full markdown report + page path + counts +
duration_ms
Also fixes WikiJSClient.update_page: Wiki.js 2.x requires tags on the
update mutation (server maps over it unconditionally); calls without tags
failed with "Cannot read properties of undefined (reading 'map')" -
which also silently broke the consolidation page-update path. Current
tags are now preserved when the caller supplies none.
Verified live end-to-end on the local dev server as llm_tester
(tests/test_quality_report_live.py, integration-marked): probe page
flagged for missing metadata, report page written and fetched back,
same-day rerun updates in place, teardown leaves zero llm_tester pages.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
This commit is contained in:
Claude Fable 5
@@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- `/ingest/status/{job_id}` is backed by the Redis `JobManager` (jobs are tenant-scoped; other tenants' jobs return 404). `/ingest/page`, `/ingest/batch` and `/ingest/all` now record job entries and return a `job_id`.
|
||||
- `/ingest/repo-status/{repository}` reports wiki page count vs indexed Document-node count under `users/{tenant}/{repository}` plus the tenant's Redis job statistics.
|
||||
- `/deduplicate/check` runs a tenant-scoped Qdrant similarity scan: wiki chunk pairs above the threshold (default 0.9 cosine) grouped per page pair with best score, matching chunk-pair count, and page references. Read-only.
|
||||
- **Weekly quality report** — `POST /maintenance/quality-report {user}` runs the duplicate scan, flags stale pages (not updated in N days AND ≤ M SearchQuery hits from the graph data), lists pages missing tags/description, folds in the latest integrity-check results (Redis-cached or run inline), and writes a dated report page to `users/{user}/system/quality-reports/YYYY-MM-DD` (same-day reruns update the same page — the page id is remembered in Redis because the Wiki.js listing lags page creation). Response returns the full report content + page path. Verified end-to-end against the local dev server as `llm_tester`.
|
||||
- **Nightly integrity check** — `POST /maintenance/integrity-check {user}` (read-only: reports, never auto-fixes) reports per tenant: wiki pages with ZERO vectors in Qdrant (silent-skip reindex victims), orphaned vectors whose wiki page no longer exists, unexpected Qdrant collections (test-tenant residue and unknown namespaces flagged; other services' collections counted as foreign), Neo4j Document nodes without wiki counterparts, plus counts and duration. The latest report is cached in Redis (30 days) so the weekly quality report can fold it in.
|
||||
|
||||
### Changed
|
||||
@@ -46,6 +47,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- **LLM call timeouts** - Phase 0 keyword extraction and Phase 4 re-ranking are wrapped in a 12s `asyncio.wait_for` with graceful fallback, so a hung Ollama call can no longer gate retrieval for the full 120s client timeout.
|
||||
- **`/stats` wiki page count** - The endpoint passed the bare user name as path prefix (matching nothing) and always reported 0 pages; it now counts pages under `users/{user}`.
|
||||
- **Wiki.js page listing** - `list_pages` applied the API-side `limit` before client-side path/tag filters, dropping matching pages that sort late; the limit now applies after filtering. `list_all_pages` replaced its fake pagination loop with a real limit-growth loop (Wiki.js 2.x `pages.list` has no offset argument) that fetches until the API returns fewer pages than requested.
|
||||
- **Wiki.js `update_page` without tags** - Wiki.js 2.x requires `tags` on the update mutation (the server unconditionally maps over it); every `update_page(page_id, content=...)` call without tags failed with `Cannot read properties of undefined (reading 'map')` — this silently broke the consolidation service's page-update path too. The client now preserves the page's current tags when the caller does not supply any.
|
||||
- **Wiki.js listing completeness under pre-filter limits** - Observed live: `pages.list(limit=100)` returned 43 pages while 140 existed (`limit=500` returned all) — Wiki.js applies the limit BEFORE its own visibility filtering, so "fewer pages than requested" does not mean the listing is complete and the limit-growth loop stopped early, silently truncating listings (page counts, cleanups, integrity scans). The loop now grows the limit until the returned count stops increasing (fixed point), at the cost of one confirming fetch.
|
||||
|
||||
## [1.7.3] - 2026-01-07
|
||||
|
||||
@@ -500,15 +500,21 @@ class WikiJSClient:
|
||||
}
|
||||
"""
|
||||
|
||||
variables = {"id": page_id}
|
||||
# Wiki.js 2.x requires `tags` on the update mutation (the server
|
||||
# unconditionally maps over it; omitting it fails with "Cannot read
|
||||
# properties of undefined (reading 'map')"). Preserve the page's
|
||||
# current tags when the caller does not supply any.
|
||||
if tags is None:
|
||||
current = await self.get_page(page_id)
|
||||
tags = (current or {}).get("tags") or []
|
||||
|
||||
variables = {"id": page_id, "tags": tags}
|
||||
if content is not None:
|
||||
variables["content"] = content
|
||||
if title is not None:
|
||||
variables["title"] = title
|
||||
if description is not None:
|
||||
variables["description"] = description
|
||||
if tags is not None:
|
||||
variables["tags"] = tags
|
||||
if is_published is not None:
|
||||
variables["isPublished"] = is_published
|
||||
|
||||
|
||||
+360
-1
@@ -23,7 +23,7 @@ from src.core.dependencies import (
|
||||
)
|
||||
from src.core.multi_tenancy import RequiredUser, sanitize_user_id
|
||||
from src.config import get_settings
|
||||
from datetime import datetime, timezone
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -1342,3 +1342,362 @@ async def integrity_check(
|
||||
except Exception as e:
|
||||
logger.error(f"Integrity check failed for {request.user}: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail="Integrity check failed")
|
||||
|
||||
|
||||
# ========== Weekly quality report ==========
|
||||
|
||||
QUALITY_REPORT_PATH_TEMPLATE = "users/{tenant}/system/quality-reports/{date}"
|
||||
|
||||
|
||||
class QualityReportRequest(BaseModel):
|
||||
"""Request body for /maintenance/quality-report."""
|
||||
user: RequiredUser = Field(
|
||||
...,
|
||||
description="User identifier (tenant). Required — the report is scoped to this tenant."
|
||||
)
|
||||
stale_days: int = Field(
|
||||
default=30, ge=1, le=365,
|
||||
description="Pages not updated in this many days are stale candidates"
|
||||
)
|
||||
max_search_hits: int = Field(
|
||||
default=1, ge=0, le=100,
|
||||
description="A stale candidate is only flagged when its SearchQuery hit count is <= this"
|
||||
)
|
||||
dedup_threshold: float = Field(
|
||||
default=0.9, ge=0.5, le=1.0,
|
||||
description="Cosine similarity threshold for the duplicate scan"
|
||||
)
|
||||
write_page: bool = Field(
|
||||
default=True,
|
||||
description="Write the dated report page to the tenant's wiki (users/{user}/system/quality-reports/YYYY-MM-DD)"
|
||||
)
|
||||
|
||||
|
||||
class QualityReportResponse(BaseModel):
|
||||
"""Weekly quality report for one tenant."""
|
||||
success: bool
|
||||
user: str
|
||||
generated_at: str
|
||||
page_path: Optional[str] = Field(
|
||||
default=None, description="Wiki path of the written report page (None when write_page=false)"
|
||||
)
|
||||
page_id: Optional[int] = None
|
||||
report: str = Field(description="Full markdown report content")
|
||||
duplicate_groups: List[Dict[str, Any]] = Field(default_factory=list)
|
||||
stale_pages: List[Dict[str, Any]] = Field(default_factory=list)
|
||||
pages_missing_metadata: List[Dict[str, Any]] = Field(default_factory=list)
|
||||
integrity: Optional[Dict[str, Any]] = Field(
|
||||
default=None, description="Latest integrity-check result (cached or run inline)"
|
||||
)
|
||||
counts: Dict[str, int] = Field(default_factory=dict)
|
||||
duration_ms: float = 0.0
|
||||
|
||||
|
||||
def _parse_wiki_timestamp(value: Any) -> Optional[datetime]:
|
||||
"""Parse a Wiki.js ISO timestamp ('...Z' or offset) to aware UTC."""
|
||||
if not value:
|
||||
return None
|
||||
try:
|
||||
parsed = datetime.fromisoformat(str(value).replace("Z", "+00:00"))
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.replace(tzinfo=timezone.utc)
|
||||
return parsed.astimezone(timezone.utc)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
async def _get_search_hit_counts(graph_service: GraphService, user: str) -> Dict[int, int]:
|
||||
"""
|
||||
Per-page SearchQuery FOUND-hit counts from the tenant's graph data.
|
||||
|
||||
Returns {page_id: hits} for every tenant Document node.
|
||||
"""
|
||||
from src.core.multi_tenancy import get_neo4j_user_base_label, get_neo4j_user_label
|
||||
|
||||
base_label = get_neo4j_user_base_label(user)
|
||||
doc_label = get_neo4j_user_label(user)
|
||||
|
||||
query = f"""
|
||||
MATCH (d:{doc_label}:Document)
|
||||
WHERE d.page_id IS NOT NULL
|
||||
OPTIONAL MATCH (sq:{base_label}_SearchQuery:SearchQuery)-[f:FOUND]->(d)
|
||||
RETURN d.page_id AS page_id, count(f) AS hits
|
||||
"""
|
||||
try:
|
||||
rows = await graph_service.neo4j.execute_query(query, {})
|
||||
return {row["page_id"]: row["hits"] for row in rows}
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to get search hit counts for {user}: {e}")
|
||||
return {}
|
||||
|
||||
|
||||
def _render_quality_report(
|
||||
user: str,
|
||||
generated_at: str,
|
||||
duplicate_scan: Dict[str, Any],
|
||||
stale_pages: List[Dict[str, Any]],
|
||||
missing_metadata: List[Dict[str, Any]],
|
||||
integrity: Optional[Dict[str, Any]],
|
||||
stale_days: int,
|
||||
max_search_hits: int,
|
||||
dedup_threshold: float,
|
||||
) -> str:
|
||||
"""Render the markdown report page content."""
|
||||
lines = [
|
||||
f"Automated quality report for tenant `{user}`, generated {generated_at}.",
|
||||
"",
|
||||
"## Summary",
|
||||
"",
|
||||
"| Metric | Count |",
|
||||
"|---|---|",
|
||||
f"| Potential duplicate page pairs (cosine ≥ {dedup_threshold}) | {len(duplicate_scan.get('duplicate_groups', []))} |",
|
||||
f"| Stale pages (> {stale_days}d old, ≤ {max_search_hits} search hits) | {len(stale_pages)} |",
|
||||
f"| Pages missing tags/description | {len(missing_metadata)} |",
|
||||
]
|
||||
if integrity:
|
||||
counts = integrity.get("counts", {})
|
||||
lines += [
|
||||
f"| Pages without vectors (integrity) | {counts.get('pages_without_vectors', 0)} |",
|
||||
f"| Orphaned vector chunks (integrity) | {counts.get('orphaned_vector_chunks', 0)} |",
|
||||
f"| Documents without wiki page (integrity) | {counts.get('documents_without_wiki', 0)} |",
|
||||
f"| Unexpected Qdrant collections (integrity) | {counts.get('unexpected_collections', 0)} |",
|
||||
]
|
||||
|
||||
lines += ["", "## Potential duplicates", ""]
|
||||
groups = duplicate_scan.get("duplicate_groups", [])
|
||||
if groups:
|
||||
for g in groups:
|
||||
pages = g.get("pages", [])
|
||||
refs = " ↔ ".join(f"`{p.get('path')}` ({p.get('title')})" for p in pages)
|
||||
lines.append(
|
||||
f"- {refs} — similarity {g.get('max_similarity', 0):.3f}, "
|
||||
f"{g.get('matching_chunk_pairs', 0)} matching chunk pair(s)"
|
||||
)
|
||||
else:
|
||||
lines.append("_None found._")
|
||||
|
||||
lines += ["", f"## Stale pages (not updated in {stale_days} days, ≤ {max_search_hits} search hits)", ""]
|
||||
if stale_pages:
|
||||
for p in stale_pages:
|
||||
lines.append(
|
||||
f"- `{p['path']}` ({p['title']}) — last updated {p['updated_at']}, "
|
||||
f"{p['search_hits']} search hit(s)"
|
||||
)
|
||||
else:
|
||||
lines.append("_None found._")
|
||||
|
||||
lines += ["", "## Pages missing metadata", ""]
|
||||
if missing_metadata:
|
||||
for p in missing_metadata:
|
||||
lines.append(f"- `{p['path']}` ({p['title']}) — missing: {', '.join(p['missing'])}")
|
||||
else:
|
||||
lines.append("_None found._")
|
||||
|
||||
lines += ["", "## Integrity check", ""]
|
||||
if integrity:
|
||||
lines.append(f"Source: {integrity.get('source', 'unknown')} (generated {integrity.get('generated_at', '?')})")
|
||||
lines.append("")
|
||||
for key, value in integrity.get("counts", {}).items():
|
||||
lines.append(f"- {key}: {value}")
|
||||
unexpected = integrity.get("unexpected_collections", [])
|
||||
if unexpected:
|
||||
lines.append("")
|
||||
lines.append("Unexpected Qdrant collections:")
|
||||
for c in unexpected:
|
||||
lines.append(f"- `{c.get('name')}` ({c.get('category')})")
|
||||
else:
|
||||
lines.append("_No integrity data available._")
|
||||
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
@router.post("/quality-report", response_model=QualityReportResponse)
|
||||
async def quality_report(
|
||||
request: QualityReportRequest,
|
||||
vector_service: VectorServiceDep = None,
|
||||
graph_service: GraphServiceDep = None,
|
||||
wiki_client: WikiJSDep = None,
|
||||
qdrant: QdrantDep = None,
|
||||
redis: RedisDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Weekly quality report for one tenant.
|
||||
|
||||
Runs the duplicate scan, flags stale pages (not updated in N days AND a
|
||||
low SearchQuery hit count from the graph data), lists pages missing
|
||||
tags/description, folds in the latest integrity-check results (cached in
|
||||
Redis by /maintenance/integrity-check, or run inline when absent), and
|
||||
writes a dated report page to the tenant's wiki under
|
||||
`users/{user}/system/quality-reports/YYYY-MM-DD`.
|
||||
|
||||
**Scheduler Task** — weekly Sunday 03:00, see docs/scheduler-tasks.md.
|
||||
"""
|
||||
import json as _json
|
||||
|
||||
start_time = time.time()
|
||||
user = request.user
|
||||
tenant = sanitize_user_id(user)
|
||||
tenant_prefix = f"users/{tenant}"
|
||||
system_prefix = f"{tenant_prefix}/system/"
|
||||
now = datetime.now(timezone.utc)
|
||||
generated_at = now.isoformat()
|
||||
|
||||
try:
|
||||
# 1. Duplicate scan (tenant-scoped, read-only)
|
||||
duplicate_scan = await vector_service.find_duplicate_pairs(
|
||||
user=user,
|
||||
similarity_threshold=request.dedup_threshold
|
||||
)
|
||||
|
||||
# 2. Page inventory + search-hit counts
|
||||
pages = await wiki_client.list_all_pages(path_prefix=tenant_prefix)
|
||||
# The report subtree itself is exempt from quality checks
|
||||
pages = [
|
||||
p for p in pages
|
||||
if not str(p.get("path", "")).lstrip("/").startswith(system_prefix)
|
||||
]
|
||||
hit_counts = await _get_search_hit_counts(graph_service, user)
|
||||
|
||||
stale_cutoff = now - timedelta(days=request.stale_days)
|
||||
stale_pages = []
|
||||
missing_metadata = []
|
||||
|
||||
for p in pages:
|
||||
path = p.get("path", "")
|
||||
title = p.get("title", "")
|
||||
updated_at = _parse_wiki_timestamp(p.get("updatedAt"))
|
||||
hits = hit_counts.get(p.get("id"), 0)
|
||||
|
||||
if updated_at and updated_at < stale_cutoff and hits <= request.max_search_hits:
|
||||
stale_pages.append({
|
||||
"page_id": p.get("id"),
|
||||
"path": path,
|
||||
"title": title,
|
||||
"updated_at": updated_at.date().isoformat(),
|
||||
"search_hits": hits,
|
||||
})
|
||||
|
||||
missing = []
|
||||
if not p.get("tags"):
|
||||
missing.append("tags")
|
||||
if not (p.get("description") or "").strip():
|
||||
missing.append("description")
|
||||
if missing:
|
||||
missing_metadata.append({
|
||||
"page_id": p.get("id"),
|
||||
"path": path,
|
||||
"title": title,
|
||||
"missing": missing,
|
||||
})
|
||||
|
||||
# 3. Integrity results: latest cached report, or run inline
|
||||
integrity: Optional[Dict[str, Any]] = None
|
||||
if redis:
|
||||
try:
|
||||
cached = await redis.get(INTEGRITY_LATEST_KEY.format(user=user))
|
||||
if cached:
|
||||
integrity = _json.loads(cached)
|
||||
integrity["source"] = "cached"
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to read cached integrity report: {e}")
|
||||
if integrity is None:
|
||||
inline = await run_integrity_check(
|
||||
user=user,
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
wiki_client=wiki_client,
|
||||
qdrant=qdrant
|
||||
)
|
||||
integrity = inline.model_dump(mode="json")
|
||||
integrity["source"] = "inline"
|
||||
|
||||
# 4. Render + write the dated report page
|
||||
report_content = _render_quality_report(
|
||||
user=user,
|
||||
generated_at=generated_at,
|
||||
duplicate_scan=duplicate_scan,
|
||||
stale_pages=stale_pages,
|
||||
missing_metadata=missing_metadata,
|
||||
integrity=integrity,
|
||||
stale_days=request.stale_days,
|
||||
max_search_hits=request.max_search_hits,
|
||||
dedup_threshold=request.dedup_threshold,
|
||||
)
|
||||
|
||||
page_path = None
|
||||
page_id = None
|
||||
if request.write_page:
|
||||
date_str = now.date().isoformat()
|
||||
page_path = QUALITY_REPORT_PATH_TEMPLATE.format(tenant=tenant, date=date_str)
|
||||
title = f"Quality Report {date_str}"
|
||||
|
||||
# Same-day reruns must UPDATE the existing page. The Wiki.js page
|
||||
# listing updates asynchronously after creation, so the page id
|
||||
# written today is remembered in Redis and used directly.
|
||||
page_id_key = f"library:quality_report:page:{tenant}:{date_str}"
|
||||
if redis:
|
||||
try:
|
||||
cached_id = await redis.get(page_id_key)
|
||||
if cached_id:
|
||||
page_id = int(cached_id)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to read quality-report page id: {e}")
|
||||
|
||||
if page_id is None:
|
||||
existing = await wiki_client.list_all_pages(path_prefix=page_path)
|
||||
exact = [
|
||||
p for p in existing
|
||||
if str(p.get("path", "")).lstrip("/") == page_path
|
||||
]
|
||||
if exact:
|
||||
page_id = exact[0]["id"]
|
||||
|
||||
if page_id is not None:
|
||||
await wiki_client.update_page(page_id=page_id, content=report_content)
|
||||
logger.info(f"Updated quality report page {page_path} (id={page_id})")
|
||||
else:
|
||||
created = await wiki_client.create_page(
|
||||
path=page_path,
|
||||
title=title,
|
||||
content=report_content,
|
||||
description=f"Automated weekly quality report for {user}",
|
||||
tags=["quality-report", "auto-generated"],
|
||||
is_published=True,
|
||||
)
|
||||
page_id = created.get("id") if created else None
|
||||
logger.info(f"Created quality report page {page_path} (id={page_id})")
|
||||
|
||||
if redis and page_id:
|
||||
try:
|
||||
await redis.setex(page_id_key, 86400 * 2, str(page_id))
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to cache quality-report page id: {e}")
|
||||
|
||||
duration_ms = (time.time() - start_time) * 1000
|
||||
counts = {
|
||||
"duplicate_groups": len(duplicate_scan.get("duplicate_groups", [])),
|
||||
"stale_pages": len(stale_pages),
|
||||
"pages_missing_metadata": len(missing_metadata),
|
||||
"pages_checked": len(pages),
|
||||
}
|
||||
logger.info(f"Quality report for {user}: {counts} in {duration_ms:.0f}ms")
|
||||
|
||||
return QualityReportResponse(
|
||||
success=True,
|
||||
user=user,
|
||||
generated_at=generated_at,
|
||||
page_path=page_path,
|
||||
page_id=page_id,
|
||||
report=report_content,
|
||||
duplicate_groups=duplicate_scan.get("duplicate_groups", []),
|
||||
stale_pages=stale_pages,
|
||||
pages_missing_metadata=missing_metadata,
|
||||
integrity=integrity,
|
||||
counts=counts,
|
||||
duration_ms=duration_ms
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Quality report failed for {user}: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail="Quality report failed")
|
||||
|
||||
@@ -0,0 +1,236 @@
|
||||
"""
|
||||
Offline unit tests for the weekly quality-report endpoint (Phase C item 3).
|
||||
|
||||
All external clients are mocked - no shared services are contacted.
|
||||
Live verification against the local dev server runs in
|
||||
tests/test_quality_report_live.py (integration-marked).
|
||||
"""
|
||||
|
||||
import json
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
|
||||
from src.routers.maintenance import (
|
||||
QualityReportRequest,
|
||||
_parse_wiki_timestamp,
|
||||
quality_report,
|
||||
)
|
||||
|
||||
TEST_USER = "llm_tester"
|
||||
TENANT_PREFIX = f"users/{TEST_USER}"
|
||||
|
||||
|
||||
def _iso(days_ago: int) -> str:
|
||||
return (datetime.now(timezone.utc) - timedelta(days=days_ago)).isoformat()
|
||||
|
||||
|
||||
def _mock_stack(
|
||||
pages,
|
||||
hit_rows,
|
||||
duplicate_scan=None,
|
||||
cached_integrity=None,
|
||||
):
|
||||
vector_service = AsyncMock()
|
||||
vector_service.find_duplicate_pairs = AsyncMock(
|
||||
return_value=duplicate_scan or {"chunks_scanned": 0, "duplicate_groups": []}
|
||||
)
|
||||
# Used only when integrity runs inline
|
||||
vector_service.get_all_chunk_references = AsyncMock(return_value=[])
|
||||
|
||||
graph_service = AsyncMock()
|
||||
graph_service.neo4j = AsyncMock()
|
||||
graph_service.neo4j.execute_query = AsyncMock(return_value=hit_rows)
|
||||
graph_service.get_all_document_references = AsyncMock(return_value=[])
|
||||
|
||||
wiki_client = AsyncMock()
|
||||
# First call: tenant listing; later calls: report-path existence check
|
||||
wiki_client.list_all_pages = AsyncMock(side_effect=[pages, []])
|
||||
wiki_client.create_page = AsyncMock(return_value={"id": 777})
|
||||
|
||||
qdrant = AsyncMock()
|
||||
qdrant.list_collections = AsyncMock(return_value=[])
|
||||
|
||||
redis = AsyncMock()
|
||||
|
||||
async def _redis_get(key):
|
||||
if key.startswith("library:integrity:latest:") and cached_integrity:
|
||||
return json.dumps(cached_integrity)
|
||||
return None
|
||||
|
||||
redis.get = AsyncMock(side_effect=_redis_get)
|
||||
|
||||
return vector_service, graph_service, wiki_client, qdrant, redis
|
||||
|
||||
|
||||
class TestQualityReport:
|
||||
@pytest.mark.asyncio
|
||||
async def test_full_report_flags_and_writes_page(self):
|
||||
pages = [
|
||||
# stale: 60 days old, 0 hits, missing tags+description
|
||||
{"id": 1, "path": f"{TENANT_PREFIX}/old-page", "title": "Old",
|
||||
"tags": [], "description": "", "updatedAt": _iso(60)},
|
||||
# old but frequently found -> NOT stale; has metadata
|
||||
{"id": 2, "path": f"{TENANT_PREFIX}/popular", "title": "Popular",
|
||||
"tags": ["x"], "description": "d", "updatedAt": _iso(60)},
|
||||
# fresh page missing description only
|
||||
{"id": 3, "path": f"{TENANT_PREFIX}/fresh", "title": "Fresh",
|
||||
"tags": ["y"], "description": "", "updatedAt": _iso(1)},
|
||||
# system report page is exempt from all checks
|
||||
{"id": 4, "path": f"{TENANT_PREFIX}/system/quality-reports/2026-07-07",
|
||||
"title": "Old report", "tags": [], "description": "", "updatedAt": _iso(7)},
|
||||
]
|
||||
hit_rows = [
|
||||
{"page_id": 1, "hits": 0},
|
||||
{"page_id": 2, "hits": 9},
|
||||
{"page_id": 3, "hits": 0},
|
||||
]
|
||||
duplicate_scan = {
|
||||
"chunks_scanned": 10,
|
||||
"duplicate_groups": [{
|
||||
"pages": [
|
||||
{"page_id": 1, "path": f"{TENANT_PREFIX}/old-page", "title": "Old"},
|
||||
{"page_id": 2, "path": f"{TENANT_PREFIX}/popular", "title": "Popular"},
|
||||
],
|
||||
"max_similarity": 0.93,
|
||||
"matching_chunk_pairs": 2,
|
||||
}],
|
||||
}
|
||||
cached_integrity = {
|
||||
"generated_at": _iso(0),
|
||||
"counts": {"pages_without_vectors": 1, "orphaned_vector_chunks": 0,
|
||||
"documents_without_wiki": 0, "unexpected_collections": 2},
|
||||
"unexpected_collections": [
|
||||
{"name": "library_desk_ghost", "category": "unknown_tenant"},
|
||||
],
|
||||
}
|
||||
|
||||
vector_service, graph_service, wiki_client, qdrant, redis = _mock_stack(
|
||||
pages, hit_rows, duplicate_scan, cached_integrity
|
||||
)
|
||||
|
||||
result = await quality_report(
|
||||
request=QualityReportRequest(user=TEST_USER, stale_days=30),
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
wiki_client=wiki_client,
|
||||
qdrant=qdrant,
|
||||
redis=redis,
|
||||
api_key="",
|
||||
)
|
||||
|
||||
assert result.success is True
|
||||
|
||||
# Stale: only page 1 (page 2 old but popular, page 3 fresh, page 4 exempt)
|
||||
assert [p["page_id"] for p in result.stale_pages] == [1]
|
||||
|
||||
# Missing metadata: page 1 (tags+description), page 3 (description)
|
||||
missing = {p["page_id"]: p["missing"] for p in result.pages_missing_metadata}
|
||||
assert missing == {1: ["tags", "description"], 3: ["description"]}
|
||||
|
||||
# Dedup folded in
|
||||
assert result.counts["duplicate_groups"] == 1
|
||||
|
||||
# Cached integrity used (no inline re-run needed)
|
||||
assert result.integrity["source"] == "cached"
|
||||
vector_service.get_all_chunk_references.assert_not_awaited()
|
||||
|
||||
# Page written under the dated report path
|
||||
today = datetime.now(timezone.utc).date().isoformat()
|
||||
assert result.page_path == f"{TENANT_PREFIX}/system/quality-reports/{today}"
|
||||
assert result.page_id == 777
|
||||
wiki_client.create_page.assert_awaited_once()
|
||||
create_kwargs = wiki_client.create_page.await_args.kwargs
|
||||
assert create_kwargs["path"] == result.page_path
|
||||
assert "auto-generated" in create_kwargs["tags"]
|
||||
|
||||
# Report content mentions the key findings
|
||||
assert "old-page" in result.report
|
||||
assert "0.930" in result.report
|
||||
assert "library_desk_ghost" in result.report
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_runs_integrity_inline_when_no_cache(self):
|
||||
vector_service, graph_service, wiki_client, qdrant, redis = _mock_stack(
|
||||
pages=[], hit_rows=[], cached_integrity=None
|
||||
)
|
||||
# Inline integrity re-lists all pages: give the side_effect one more value
|
||||
wiki_client.list_all_pages = AsyncMock(side_effect=[[], [], []])
|
||||
|
||||
result = await quality_report(
|
||||
request=QualityReportRequest(user=TEST_USER),
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
wiki_client=wiki_client,
|
||||
qdrant=qdrant,
|
||||
redis=redis,
|
||||
api_key="",
|
||||
)
|
||||
|
||||
assert result.integrity["source"] == "inline"
|
||||
vector_service.get_all_chunk_references.assert_awaited_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_write_page_false_skips_wiki_write(self):
|
||||
vector_service, graph_service, wiki_client, qdrant, redis = _mock_stack(
|
||||
pages=[], hit_rows=[],
|
||||
cached_integrity={"generated_at": _iso(0), "counts": {},
|
||||
"unexpected_collections": []},
|
||||
)
|
||||
|
||||
result = await quality_report(
|
||||
request=QualityReportRequest(user=TEST_USER, write_page=False),
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
wiki_client=wiki_client,
|
||||
qdrant=qdrant,
|
||||
redis=redis,
|
||||
api_key="",
|
||||
)
|
||||
|
||||
assert result.page_path is None
|
||||
assert result.page_id is None
|
||||
wiki_client.create_page.assert_not_awaited()
|
||||
wiki_client.update_page.assert_not_awaited()
|
||||
assert result.report # content still returned
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_same_day_rerun_updates_existing_page(self):
|
||||
today = datetime.now(timezone.utc).date().isoformat()
|
||||
report_path = f"{TENANT_PREFIX}/system/quality-reports/{today}"
|
||||
|
||||
vector_service, graph_service, wiki_client, qdrant, redis = _mock_stack(
|
||||
pages=[], hit_rows=[],
|
||||
cached_integrity={"generated_at": _iso(0), "counts": {},
|
||||
"unexpected_collections": []},
|
||||
)
|
||||
wiki_client.list_all_pages = AsyncMock(side_effect=[
|
||||
[], # tenant listing
|
||||
[{"id": 555, "path": report_path, "title": f"Quality Report {today}"}],
|
||||
])
|
||||
|
||||
result = await quality_report(
|
||||
request=QualityReportRequest(user=TEST_USER),
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
wiki_client=wiki_client,
|
||||
qdrant=qdrant,
|
||||
redis=redis,
|
||||
api_key="",
|
||||
)
|
||||
|
||||
assert result.page_id == 555
|
||||
wiki_client.update_page.assert_awaited_once()
|
||||
wiki_client.create_page.assert_not_awaited()
|
||||
|
||||
def test_requires_user(self):
|
||||
with pytest.raises(Exception):
|
||||
QualityReportRequest(user="")
|
||||
|
||||
|
||||
def test_parse_wiki_timestamp():
|
||||
parsed = _parse_wiki_timestamp("2026-07-14T09:39:45.244Z")
|
||||
assert parsed is not None and parsed.tzinfo is not None
|
||||
assert _parse_wiki_timestamp(None) is None
|
||||
assert _parse_wiki_timestamp("not-a-date") is None
|
||||
@@ -0,0 +1,161 @@
|
||||
"""
|
||||
Live end-to-end test for the weekly quality report (integration, guard-gated).
|
||||
|
||||
Runs against the LOCAL wakeup server (./wakeup.sh, port 8778 — never the
|
||||
production container on 8089) with the shared backing services, entirely
|
||||
under the reserved llm_tester tenant namespace:
|
||||
|
||||
1. Create + ingest a wiki page as ``llm_tester`` (deliberately without tags
|
||||
so the report has something to flag).
|
||||
2. POST /maintenance/quality-report {user: llm_tester}: asserts the report
|
||||
is generated, mentions the created page in the missing-metadata section,
|
||||
folds in integrity results, and writes the dated report page under
|
||||
users/llm_tester/system/quality-reports/YYYY-MM-DD.
|
||||
3. Fetches the written report page back from the wiki.
|
||||
4. Teardown deletes the created pages; the session-scoped teardown in
|
||||
conftest purges every remaining llm_tester artifact.
|
||||
|
||||
Run with:
|
||||
RUN_INTEGRATION_TESTS=1 TEST_HOST=<shared-host> \\
|
||||
LIBRARY_DESK_URL=http://localhost:8778 \\
|
||||
.venv/bin/python -m pytest tests/test_quality_report_live.py -v
|
||||
"""
|
||||
|
||||
import time
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from tests.conftest import (
|
||||
LIBRARY_DESK_URL,
|
||||
PRODUCTION_TENANT,
|
||||
TEST_TENANT,
|
||||
assert_safe_test_tenant,
|
||||
)
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def api():
|
||||
"""HTTP client for the local dev server, with bearer auth."""
|
||||
from src.config import get_settings
|
||||
|
||||
assert ":8089" not in LIBRARY_DESK_URL, (
|
||||
"Refusing to run the live quality-report test against the "
|
||||
"production container (port 8089); point LIBRARY_DESK_URL at ./wakeup.sh"
|
||||
)
|
||||
settings = get_settings()
|
||||
client = httpx.Client(
|
||||
base_url=LIBRARY_DESK_URL,
|
||||
headers={"Authorization": f"Bearer {settings.library_api_key}"},
|
||||
timeout=httpx.Timeout(300.0, connect=10.0),
|
||||
)
|
||||
yield client
|
||||
client.close()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def ingested_page(api):
|
||||
"""Create + ingest a metadata-poor wiki page as the test tenant."""
|
||||
assert_safe_test_tenant(TEST_TENANT)
|
||||
|
||||
slug = f"quality-probe-{uuid.uuid4().hex[:8]}"
|
||||
create_resp = api.post(
|
||||
"/wiki/pages",
|
||||
json={
|
||||
"title": f"Quality Probe {slug}",
|
||||
"path": f"/quality-tests/{slug}",
|
||||
"content": (
|
||||
"# Quality Probe\n\n"
|
||||
"Ephemeral page used to verify the weekly quality report. "
|
||||
"It intentionally has no tags so the report flags it."
|
||||
),
|
||||
"description": "",
|
||||
"tags": [],
|
||||
"user": TEST_TENANT,
|
||||
},
|
||||
)
|
||||
assert create_resp.status_code == 201, create_resp.text
|
||||
page = create_resp.json()
|
||||
page_id = page["id"]
|
||||
assert page["path"].lstrip("/").startswith(f"users/{TEST_TENANT}")
|
||||
|
||||
ingest_resp = api.post(
|
||||
"/ingest/page",
|
||||
json={"page_id": page_id, "user": TEST_TENANT, "force_refresh": True},
|
||||
)
|
||||
assert ingest_resp.status_code == 200, ingest_resp.text
|
||||
|
||||
# Wiki.js updates its page-listing index asynchronously after creation;
|
||||
# the quality report relies on that listing, so wait until the new page
|
||||
# is visible (up to ~30s) before running the report.
|
||||
deadline = time.time() + 30
|
||||
while time.time() < deadline:
|
||||
listing = api.get("/wiki/pages", params={"user": TEST_TENANT})
|
||||
assert listing.status_code == 200, listing.text
|
||||
if any(p["id"] == page_id for p in listing.json().get("pages", [])):
|
||||
break
|
||||
time.sleep(2)
|
||||
else:
|
||||
pytest.fail(f"Page {page_id} never appeared in the Wiki.js listing")
|
||||
|
||||
yield {"page_id": page_id, "path": page["path"]}
|
||||
|
||||
delete_resp = api.delete(f"/wiki/pages/{page_id}", params={"user": TEST_TENANT})
|
||||
assert delete_resp.status_code == 200, delete_resp.text
|
||||
|
||||
|
||||
class TestLiveQualityReport:
|
||||
def test_quality_report_end_to_end(self, api, ingested_page):
|
||||
assert_safe_test_tenant(TEST_TENANT)
|
||||
|
||||
resp = api.post(
|
||||
"/maintenance/quality-report",
|
||||
json={"user": TEST_TENANT, "stale_days": 30},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
data = resp.json()
|
||||
|
||||
assert data["success"] is True
|
||||
assert data["user"] == TEST_TENANT
|
||||
assert data["duration_ms"] > 0
|
||||
|
||||
# The dated report page was written inside the tenant's namespace
|
||||
today = datetime.now(timezone.utc).date().isoformat()
|
||||
expected_path = f"users/{TEST_TENANT}/system/quality-reports/{today}"
|
||||
assert data["page_path"] == expected_path
|
||||
assert data["page_id"]
|
||||
assert PRODUCTION_TENANT not in data["page_path"]
|
||||
|
||||
# The metadata-poor probe page is flagged
|
||||
flagged_ids = {p["page_id"] for p in data["pages_missing_metadata"]}
|
||||
assert ingested_page["page_id"] in flagged_ids
|
||||
|
||||
# Integrity results are folded in (cached or inline)
|
||||
assert data["integrity"] is not None
|
||||
assert data["integrity"]["source"] in ("cached", "inline")
|
||||
assert "counts" in data["integrity"]
|
||||
|
||||
# Report content includes the summary + the probe page path
|
||||
assert "## Summary" in data["report"]
|
||||
assert ingested_page["path"].lstrip("/") in data["report"]
|
||||
|
||||
# The written page is retrievable from the wiki via the API
|
||||
page_resp = api.get(
|
||||
f"/wiki/pages/{data['page_id']}", params={"user": TEST_TENANT}
|
||||
)
|
||||
assert page_resp.status_code == 200, page_resp.text
|
||||
page = page_resp.json()
|
||||
assert page["path"].lstrip("/") == expected_path
|
||||
assert "## Summary" in page["content"]
|
||||
|
||||
# Same-day rerun must update the same page, not create a duplicate
|
||||
rerun = api.post(
|
||||
"/maintenance/quality-report",
|
||||
json={"user": TEST_TENANT, "stale_days": 30},
|
||||
)
|
||||
assert rerun.status_code == 200, rerun.text
|
||||
assert rerun.json()["page_id"] == data["page_id"]
|
||||
Reference in New Issue
Block a user