docs: add scheduler integration and memory system plan

Documentation updates: - AGENTS.md: Added wakeup.sh usage and local testing instructions - LIBRARIAN_INTEGRATION.md: Complete maintenance scheduler docs - Scheduled task configuration for reconcile-index - Endpoint specifications and response formats - docs/MEMORY_SYSTEM_PLAN.md: Three-tier memory architecture - Volatile (Redis TTL) for ephemeral context - Documents (TBD) for git mirrors, PDFs, images - Knowledge (Wiki + Neo4j) for permanent research 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-24 16:33:06 +01:00
parent f756ca9490
commit f139f518ff
3 changed files with 446 additions and 0 deletions
@@ -51,6 +51,21 @@ When changes are ready for deployment:

 ---

+### 🧪 Local Development Setup
+* **Always test locally first** before committing and deploying. The build-deploy loop is slow.
+* **Start the local server** with `./wakeup.sh` - logs are written to `logs/server.log` for easy tailing
+* **Auto-reload**: The wakeup script runs uvicorn in reload mode - code changes are picked up automatically without restart (except for requirements.txt changes)
+* **Test REST endpoints** against `http://localhost:8778` using curl or similar tools
+* **Only deploy** when a phase or feature is complete and tested locally
+* **Environment**: Copy `.env.example` to `.env` and configure for your local setup (Ollama, Redis, Neo4j, Qdrant, Wiki.js hosts)
+* **Running tests**: Always use the venv explicitly to avoid environment mismatches:
+  ```bash
+  .venv/bin/python -m pytest tests/           # All tests
+  .venv/bin/python -m pytest tests/ -v        # Verbose output
+  ```
+
+---
+
 ## 2. FastAPI Architecture & Best Practices
 *Reference: [FastAPI Best Practices](https://github.com/zhanymkanov/fastapi-best-practices)*

@@ -455,6 +455,154 @@ LIBRARY_BATCH_SIZE=50
 LIBRARY_SYNC_ENABLED=true
 ```

+## Maintenance Tasks
+
+### Index Reconciliation (Daily)
+
+The `reconcile-index` endpoint performs full index maintenance:
+
+1. **Cleanup Phase**: Remove orphaned data
+   - Vector chunks without wiki source
+   - Graph nodes without vectors (bidirectional)
+   - Vectors without graph nodes (bidirectional)
+   - Orphan entities (no MENTIONS relationships)
+   - Broken relationships
+
+2. **Reindex Phase**: Index missing pages
+   - Wiki pages without vector embeddings
+   - Wiki pages without graph Document nodes
+
+**Scheduler Task: `library_reconcile_index`**
+
+```yaml
+Task Name: library_reconcile_index
+Description: Daily index reconciliation - cleanup orphans + reindex missing pages
+Schedule: Daily at 04:00 (after library_sync at 03:30)
+Priority: 10 (system maintenance)
+Service: library
+Executor: POST /maintenance/reconcile-index
+Configuration:
+  - LIBRARY_DESK_URL: http://library-desk:8089
+  - LIBRARY_API_KEY: ${LIBRARY_API_KEY}
+Parameters:
+  - user: jpmschweitzer
+  - dry_run: false
+Outputs:
+  - Vector orphans purged
+  - Entity orphans purged
+  - Missing pages reindexed
+```
+
+### Maintenance Endpoints
+
+| Endpoint | Method | Purpose |
+|----------|--------|---------|
+| `/maintenance/reconcile-index` | POST | **Recommended**: Full cleanup + reindex missing |
+| `/maintenance/cleanup/all` | POST | Cleanup only (orphan removal) |
+| `/maintenance/cleanup/vectors` | POST | Clean orphan vector chunks only |
+| `/maintenance/cleanup/graph` | POST | Clean orphan entities & stale docs only |
+| `/maintenance/health` | GET | Lightweight health check (for uptime monitoring) |
+| `/maintenance/health?detailed=true` | GET | Full analysis with orphan counts |
+| `/maintenance/reindex/{page_id}` | POST | Force re-index a specific page |
+
+### Health Check Modes
+
+**Lightweight (default)** - Use for frequent uptime checks (every 30s):
+```bash
+curl "http://library-desk:8089/maintenance/health?user=jpmschweitzer" \
+  -H "Authorization: Bearer ${LIBRARY_API_KEY}"
+```
+
+Returns only last cleanup timestamp and basic status (no database queries).
+
+**Detailed** - Use for dashboards or before reconciliation:
+```bash
+curl "http://library-desk:8089/maintenance/health?user=jpmschweitzer&detailed=true" \
+  -H "Authorization: Bearer ${LIBRARY_API_KEY}"
+```
+
+Returns full orphan analysis (runs database queries).
+
+### Example Reconcile Request
+
+```bash
+curl -X POST "http://library-desk:8089/maintenance/reconcile-index?user=jpmschweitzer" \
+  -H "Authorization: Bearer ${LIBRARY_API_KEY}"
+```
+
+### Example Response
+
+```json
+{
+  "success": true,
+  "cleanup": {
+    "success": true,
+    "vector_cleanup": {
+      "wiki_chunks": {"orphans_found": 5, "orphans_purged": 5},
+      "document_chunks": {"orphans_found": 0, "orphans_purged": 0},
+      "chunks_without_graph": {"orphans_found": 2, "orphans_purged": 2},
+      "total_chunks_scanned": 1250,
+      "total_orphans_purged": 7
+    },
+    "graph_cleanup": {
+      "orphan_entities": {"orphans_found": 3, "orphans_purged": 3},
+      "stale_wiki_documents": {"orphans_found": 1, "orphans_purged": 1},
+      "stale_store_documents": {"orphans_found": 0, "orphans_purged": 0},
+      "docs_without_vectors": {"orphans_found": 0, "orphans_purged": 0},
+      "broken_relationships_cleaned": 0
+    },
+    "total_duration_ms": 1523.5
+  },
+  "reindex_missing": {
+    "pages_without_vectors": 2,
+    "pages_without_graph": 1,
+    "pages_reindexed": 2,
+    "pages_failed": 0,
+    "failed_page_ids": [],
+    "duration_ms": 3421.2
+  },
+  "total_duration_ms": 4944.7
+}
+```
+
+### Scheduler Integration Code
+
+```python
+# scheduler/src/tasks/library_maintenance.py
+
+async def library_reconcile_index_task(user: str = "jpmschweitzer"):
+    """Run daily Library Desk index reconciliation."""
+
+    async with httpx.AsyncClient() as client:
+        # Run reconcile-index (cleanup + reindex missing)
+        result = await client.post(
+            f"{LIBRARY_DESK_URL}/maintenance/reconcile-index",
+            params={"user": user, "dry_run": False},
+            headers={"Authorization": f"Bearer {LIBRARY_API_KEY}"},
+            timeout=600.0  # 10 minutes for large indexes
+        )
+
+        data = result.json()
+
+        # Log summary
+        cleanup = data["cleanup"]
+        reindex = data["reindex_missing"]
+
+        logger.info(
+            f"Reconcile complete: "
+            f"{cleanup['vector_cleanup']['total_orphans_purged']} vector orphans, "
+            f"{cleanup['graph_cleanup']['orphan_entities']['orphans_purged']} entity orphans, "
+            f"{reindex['pages_reindexed']} pages reindexed"
+        )
+
+        if reindex["pages_failed"] > 0:
+            logger.warning(f"Failed to reindex pages: {reindex['failed_page_ids']}")
+
+        return data
+```
+
+---
+
 ## Next Steps

 1. Implement ingestion endpoints in Library Desk
@@ -0,0 +1,283 @@
+# Memory Management System - Implementation Plan
+
+## Overview
+
+A three-tier memory architecture for Library Desk with intelligent orchestration:
+
+| Tier | Storage | Purpose | TTL |
+|------|---------|---------|-----|
+| **Volatile** | Redis | Weather, news, financial, ephemeral context | 5min - 2hr |
+| **Documents** | TBD (research) | Git mirrors, PDFs, video, images | Permanent |
+| **Knowledge** | Wiki + Neo4j | Personal dossiers, research, summaries | Permanent |
+
+**Implementation Priority**: Cleanup → Volatile → Documents
+
+---
+
+## Phase 1: Cleanup System Completion
+
+### Current State
+- **COMPLETE** - All Phase 1 tasks implemented
+- Redis timestamp tracking for last cleanup
+- Bidirectional orphan detection between vectors and graph
+- Scheduler integration endpoints ready
+
+### Tasks
+
+#### 1.1 Add Scheduler Integration Points ✅
+**Files**: `src/routers/maintenance.py`
+
+- [x] Add `last_cleanup` timestamp tracking in Redis
+- [x] Return cleanup stats in format scheduler can log
+- [x] Added `RedisDep` to cleanup endpoints
+
+#### 1.2 Bidirectional Orphan Detection ✅
+**Files**: `src/services/graph_service.py`, `src/services/vector_service.py`
+
+- [x] `find_documents_without_vectors()` - graph nodes with no vectors
+- [x] `find_chunks_without_graph_nodes()` - vectors with no graph node
+- [x] Updated maintenance endpoints to use bidirectional checks
+- [x] Added `chunks_without_graph` and `docs_without_vectors` to response models
+
+#### 1.3 Scheduler Configuration ✅
+**Scheduler-side task definition:**
+```json
+{
+  "task_name": "library_reconcile_index",
+  "schedule": "0 4 * * *",
+  "endpoint": "POST /maintenance/reconcile-index?user=jpmschweitzer",
+  "description": "Daily index reconciliation - cleanup + reindex missing"
+}
+```
+
+- [x] Documented in `LIBRARIAN_INTEGRATION.md`
+- [x] Added `reconcile-index` endpoint (cleanup + reindex missing)
+- [x] Lightweight health check mode for uptime monitoring
+- [x] Detailed health check mode for dashboards
+
+---
+
+## Phase 2: Volatile Memory System
+
+### Architecture
+
+```
+┌─────────────────┐    ┌──────────────┐    ┌─────────────────┐
+│  Library-Desk   │◄───│   Scheduler  │───►│ External APIs   │
+│                 │    │              │    │ (weather, news) │
+│  VolatileCache  │    │  Refresh     │    └─────────────────┘
+│  Service        │    │  Jobs        │
+└────────┬────────┘    └──────────────┘
+         │
+         ▼
+┌─────────────────┐
+│     Redis       │
+│  (DB 4, TTL)    │
+└─────────────────┘
+```
+
+### Data Model
+
+```python
+class VolatileRecord(BaseModel):
+    key: str                           # e.g., "weather:rotterdam"
+    namespace: str                     # e.g., "weather", "news", "financial"
+    data: dict                         # Actual content
+    source: Optional[str]              # Origin API/service
+    created_at: datetime
+    updated_at: datetime
+    ttl: int                           # Seconds until expiration
+    refresh_schedule: Optional[str]    # Cron expression, if repeating
+    user: str                          # Multi-tenant isolation
+```
+
+**Key pattern**: `{user}:volatile:{namespace}:{key_hash}`
+
+### Implementation Order: Integration-First
+
+1. **Start with Consolidation Hook** - Understand data flow through existing system
+2. **Build Service Layer** - VolatileCacheService with Redis operations
+3. **Add API Endpoints** - REST interface for volatile data
+4. **Biographer Integration** - Query user preferences for relevance
+
+### Tasks
+
+#### 2.1 Integrate with Consolidation (FIRST)
+**New file**: `src/services/volatile_service.py`
+
+```python
+class VolatileCacheService:
+    async def get(user, namespace, key) -> Optional[VolatileRecord]
+    async def set(user, namespace, key, data, ttl, refresh_schedule=None)
+    async def delete(user, namespace, key)
+    async def list_namespace(user, namespace) -> List[str]
+    async def get_scheduled(user) -> List[VolatileRecord]  # For scheduler
+```
+
+#### 2.2 Create Volatile API Router
+**New file**: `src/routers/volatile.py`
+
+| Endpoint | Method | Purpose |
+|----------|--------|---------|
+| `/volatile/{namespace}/{key}` | GET | Retrieve record |
+| `/volatile/{namespace}/{key}` | POST | Store/update record |
+| `/volatile/{namespace}/{key}` | DELETE | Remove record |
+| `/volatile/{namespace}` | GET | List keys in namespace |
+| `/volatile/scheduled` | GET | List records needing refresh |
+| `/volatile/stats` | GET | Cache statistics |
+
+#### 2.3 Integrate with Consolidation
+**File**: `src/services/consolidation_service.py`
+
+Add relevance trigger detection:
+1. During consolidation, analyze search results for location/interest patterns
+2. Query tatlock's Biographer collection for user preferences
+3. If match found, create/update volatile refresh schedule
+
+#### 2.4 Biographer Integration
+**File**: `src/core/dependencies.py`
+
+```python
+def get_biographer_qdrant() -> QdrantClientWrapper:
+    """Direct access to tatlock's Biographer collection."""
+    # Configure to connect to tatlock's Qdrant
+```
+
+#### 2.5 Scheduler-Side Configuration
+Document required scheduler tasks:
+```json
+{
+  "task_name": "volatile_refresh",
+  "schedule": "*/15 * * * *",
+  "endpoint": "GET /volatile/scheduled",
+  "follow_up": "For each record, call refresh endpoint with record.refresh_schedule"
+}
+```
+
+---
+
+## Phase 3: Document Storage (Research + Implementation)
+
+### Research Scope
+
+Evaluate FOSS self-hosted options for:
+- Git repository mirroring
+- PDF/document storage with metadata
+- Image/video blob storage
+- Full-text search capability
+
+**Constraints**:
+- Must be self-hosted, Docker-deployable
+- Performance is priority (can wrap complexity in API)
+- No cloud dependencies
+
+**Candidates to evaluate**:
+1. MinIO (S3-compatible object storage) + metadata in Neo4j
+2. Paperless-ngx (document management with OCR)
+3. SeaweedFS (distributed file system)
+4. Custom: filesystem + Neo4j metadata
+
+### Category Descriptors
+
+**Wiki page structure for document collections**:
+```markdown
+# FastAPI Documentation
+
+## Overview
+[LLM-generated summary from web search about FastAPI]
+
+## Collection Statistics
+- **Documents**: 342 files
+- **Last Sync**: 2025-12-24 03:30 UTC
+- **Source**: github.com/tiangolo/fastapi
+- **Coverage**: API reference, tutorials, deployment guides
+
+## What's Included
+[LLM summary of collection contents based on document analysis]
+
+## Related Topics
+- [[Python Web Frameworks]]
+- [[REST API Design]]
+```
+
+### Tasks
+
+#### 3.1 Storage Research
+**Deliverable**: Evaluation document comparing options
+
+#### 3.2 Storage Service Implementation
+**New file**: `src/services/document_store_service.py`
+(Details pending research results)
+
+#### 3.3 Category Descriptor Generation
+**File**: `src/services/consolidation_service.py`
+
+Add LLM-powered category descriptor generation:
+1. Web search for topic overview
+2. Analyze collection contents
+3. Generate/update wiki page with template
+
+---
+
+## Files to Modify/Create
+
+### Phase 1 (Cleanup)
+- `src/routers/maintenance.py` - Add timestamp tracking
+- `src/services/graph_service.py` - Bidirectional validation
+- `src/services/vector_service.py` - Cross-reference checks
+- `LIBRARIAN_INTEGRATION.md` - Scheduler config docs
+
+### Phase 2 (Volatile)
+- `src/services/volatile_service.py` - **NEW**
+- `src/routers/volatile.py` - **NEW**
+- `src/models/volatile.py` - **NEW**
+- `src/core/dependencies.py` - Add Biographer client
+- `src/services/consolidation_service.py` - Relevance triggers
+- `tests/test_volatile.py` - **NEW**
+
+### Phase 3 (Documents)
+- `docs/DOCUMENT_STORAGE_RESEARCH.md` - **NEW**
+- `src/services/document_store_service.py` - **NEW** (post-research)
+- `src/routers/documents.py` - **NEW** (post-research)
+
+---
+
+## Resolved Design Decisions
+
+1. **Biographer Qdrant**: Same Qdrant instance, different collection. Library-Desk queries directly.
+2. **Scheduler API**: Has REST API for task registration. Library-Desk can programmatically create refresh schedules.
+3. **External API calls**: Library-Desk routes through SearXNG for web search. Consider dedicated API integrations for high-value volatiles (weather, financial) for consistent quality.
+
+---
+
+## Future Consideration: Dedicated API Integrations
+
+For volatile data where quality/consistency matters (weather, financial), consider:
+- OpenWeatherMap API for weather (daily refresh cycle)
+- Financial data API (Alpha Vantage, Yahoo Finance)
+- News APIs (NewsAPI, GDELT)
+- **NOS.nl** - Explicit source for Dutch news
+
+This would live in a new `src/clients/` module with:
+- `weather_client.py` - Daily refresh cycle
+- `financial_client.py`
+- `news_client.py` - Include NOS.nl scraper/API for Dutch coverage
+
+These provide structured, reliable data vs. SearXNG web scraping. Implementation deferred to later phase.
+
+---
+
+## Refresh Schedules
+
+**Note:** TTL should be longer than refresh interval to prevent data gaps.
+
+| Volatile Type | TTL | Refresh Cycle | Refresh Interval | Sources |
+|---------------|-----|---------------|------------------|---------|
+| Weather | 86400s (24hr) | Daily | Every 24hr | OpenWeatherMap |
+| Dutch News | 28800s (8hr) | 4x daily | Every 6hr | NOS.nl |
+| Global News | 28800s (8hr) | 4x daily | Every 6hr | NewsAPI, GDELT |
+| Financial | 600s (10min) | On-demand | N/A | Alpha Vantage |
+
+**TTL Logic:**
+- TTL = Refresh Interval × 1.5 (buffer for failed refreshes)
+- On-demand data gets shorter TTL since it's fetched when needed