docs: add scheduler integration and memory system plan

Documentation updates: - AGENTS.md: Added wakeup.sh usage and local testing instructions - LIBRARIAN_INTEGRATION.md: Complete maintenance scheduler docs - Scheduled task configuration for reconcile-index - Endpoint specifications and response formats - docs/MEMORY_SYSTEM_PLAN.md: Three-tier memory architecture - Volatile (Redis TTL) for ephemeral context - Documents (TBD) for git mirrors, PDFs, images - Knowledge (Wiki + Neo4j) for permanent research 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-24 16:33:06 +01:00
parent f756ca9490
commit f139f518ff
3 changed files with 446 additions and 0 deletions
@@ -455,6 +455,154 @@ LIBRARY_BATCH_SIZE=50
 LIBRARY_SYNC_ENABLED=true
 ```

+## Maintenance Tasks
+
+### Index Reconciliation (Daily)
+
+The `reconcile-index` endpoint performs full index maintenance:
+
+1. **Cleanup Phase**: Remove orphaned data
+   - Vector chunks without wiki source
+   - Graph nodes without vectors (bidirectional)
+   - Vectors without graph nodes (bidirectional)
+   - Orphan entities (no MENTIONS relationships)
+   - Broken relationships
+
+2. **Reindex Phase**: Index missing pages
+   - Wiki pages without vector embeddings
+   - Wiki pages without graph Document nodes
+
+**Scheduler Task: `library_reconcile_index`**
+
+```yaml
+Task Name: library_reconcile_index
+Description: Daily index reconciliation - cleanup orphans + reindex missing pages
+Schedule: Daily at 04:00 (after library_sync at 03:30)
+Priority: 10 (system maintenance)
+Service: library
+Executor: POST /maintenance/reconcile-index
+Configuration:
+  - LIBRARY_DESK_URL: http://library-desk:8089
+  - LIBRARY_API_KEY: ${LIBRARY_API_KEY}
+Parameters:
+  - user: jpmschweitzer
+  - dry_run: false
+Outputs:
+  - Vector orphans purged
+  - Entity orphans purged
+  - Missing pages reindexed
+```
+
+### Maintenance Endpoints
+
+| Endpoint | Method | Purpose |
+|----------|--------|---------|
+| `/maintenance/reconcile-index` | POST | **Recommended**: Full cleanup + reindex missing |
+| `/maintenance/cleanup/all` | POST | Cleanup only (orphan removal) |
+| `/maintenance/cleanup/vectors` | POST | Clean orphan vector chunks only |
+| `/maintenance/cleanup/graph` | POST | Clean orphan entities & stale docs only |
+| `/maintenance/health` | GET | Lightweight health check (for uptime monitoring) |
+| `/maintenance/health?detailed=true` | GET | Full analysis with orphan counts |
+| `/maintenance/reindex/{page_id}` | POST | Force re-index a specific page |
+
+### Health Check Modes
+
+**Lightweight (default)** - Use for frequent uptime checks (every 30s):
+```bash
+curl "http://library-desk:8089/maintenance/health?user=jpmschweitzer" \
+  -H "Authorization: Bearer ${LIBRARY_API_KEY}"
+```
+
+Returns only last cleanup timestamp and basic status (no database queries).
+
+**Detailed** - Use for dashboards or before reconciliation:
+```bash
+curl "http://library-desk:8089/maintenance/health?user=jpmschweitzer&detailed=true" \
+  -H "Authorization: Bearer ${LIBRARY_API_KEY}"
+```
+
+Returns full orphan analysis (runs database queries).
+
+### Example Reconcile Request
+
+```bash
+curl -X POST "http://library-desk:8089/maintenance/reconcile-index?user=jpmschweitzer" \
+  -H "Authorization: Bearer ${LIBRARY_API_KEY}"
+```
+
+### Example Response
+
+```json
+{
+  "success": true,
+  "cleanup": {
+    "success": true,
+    "vector_cleanup": {
+      "wiki_chunks": {"orphans_found": 5, "orphans_purged": 5},
+      "document_chunks": {"orphans_found": 0, "orphans_purged": 0},
+      "chunks_without_graph": {"orphans_found": 2, "orphans_purged": 2},
+      "total_chunks_scanned": 1250,
+      "total_orphans_purged": 7
+    },
+    "graph_cleanup": {
+      "orphan_entities": {"orphans_found": 3, "orphans_purged": 3},
+      "stale_wiki_documents": {"orphans_found": 1, "orphans_purged": 1},
+      "stale_store_documents": {"orphans_found": 0, "orphans_purged": 0},
+      "docs_without_vectors": {"orphans_found": 0, "orphans_purged": 0},
+      "broken_relationships_cleaned": 0
+    },
+    "total_duration_ms": 1523.5
+  },
+  "reindex_missing": {
+    "pages_without_vectors": 2,
+    "pages_without_graph": 1,
+    "pages_reindexed": 2,
+    "pages_failed": 0,
+    "failed_page_ids": [],
+    "duration_ms": 3421.2
+  },
+  "total_duration_ms": 4944.7
+}
+```
+
+### Scheduler Integration Code
+
+```python
+# scheduler/src/tasks/library_maintenance.py
+
+async def library_reconcile_index_task(user: str = "jpmschweitzer"):
+    """Run daily Library Desk index reconciliation."""
+
+    async with httpx.AsyncClient() as client:
+        # Run reconcile-index (cleanup + reindex missing)
+        result = await client.post(
+            f"{LIBRARY_DESK_URL}/maintenance/reconcile-index",
+            params={"user": user, "dry_run": False},
+            headers={"Authorization": f"Bearer {LIBRARY_API_KEY}"},
+            timeout=600.0  # 10 minutes for large indexes
+        )
+
+        data = result.json()
+
+        # Log summary
+        cleanup = data["cleanup"]
+        reindex = data["reindex_missing"]
+
+        logger.info(
+            f"Reconcile complete: "
+            f"{cleanup['vector_cleanup']['total_orphans_purged']} vector orphans, "
+            f"{cleanup['graph_cleanup']['orphan_entities']['orphans_purged']} entity orphans, "
+            f"{reindex['pages_reindexed']} pages reindexed"
+        )
+
+        if reindex["pages_failed"] > 0:
+            logger.warning(f"Failed to reindex pages: {reindex['failed_page_ids']}")
+
+        return data
+```
+
+---
+
 ## Next Steps

 1. Implement ingestion endpoints in Library Desk