From f139f518ffc43fa4598e4e6ce32cf9bca9a6116b Mon Sep 17 00:00:00 2001 From: Jeroen Schweitzer Date: Wed, 24 Dec 2025 16:33:06 +0100 Subject: [PATCH] docs: add scheduler integration and memory system plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documentation updates: - AGENTS.md: Added wakeup.sh usage and local testing instructions - LIBRARIAN_INTEGRATION.md: Complete maintenance scheduler docs - Scheduled task configuration for reconcile-index - Endpoint specifications and response formats - docs/MEMORY_SYSTEM_PLAN.md: Three-tier memory architecture - Volatile (Redis TTL) for ephemeral context - Documents (TBD) for git mirrors, PDFs, images - Knowledge (Wiki + Neo4j) for permanent research ๐Ÿค– Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 --- AGENTS.md | 15 ++ LIBRARIAN_INTEGRATION.md | 148 +++++++++++++++++++ docs/MEMORY_SYSTEM_PLAN.md | 283 +++++++++++++++++++++++++++++++++++++ 3 files changed, 446 insertions(+) create mode 100644 docs/MEMORY_SYSTEM_PLAN.md diff --git a/AGENTS.md b/AGENTS.md index f22140e..173623d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -51,6 +51,21 @@ When changes are ready for deployment: --- +### ๐Ÿงช Local Development Setup +* **Always test locally first** before committing and deploying. The build-deploy loop is slow. +* **Start the local server** with `./wakeup.sh` - logs are written to `logs/server.log` for easy tailing +* **Auto-reload**: The wakeup script runs uvicorn in reload mode - code changes are picked up automatically without restart (except for requirements.txt changes) +* **Test REST endpoints** against `http://localhost:8778` using curl or similar tools +* **Only deploy** when a phase or feature is complete and tested locally +* **Environment**: Copy `.env.example` to `.env` and configure for your local setup (Ollama, Redis, Neo4j, Qdrant, Wiki.js hosts) +* **Running tests**: Always use the venv explicitly to avoid environment mismatches: + ```bash + .venv/bin/python -m pytest tests/ # All tests + .venv/bin/python -m pytest tests/ -v # Verbose output + ``` + +--- + ## 2. FastAPI Architecture & Best Practices *Reference: [FastAPI Best Practices](https://github.com/zhanymkanov/fastapi-best-practices)* diff --git a/LIBRARIAN_INTEGRATION.md b/LIBRARIAN_INTEGRATION.md index afc04ae..8131dbc 100644 --- a/LIBRARIAN_INTEGRATION.md +++ b/LIBRARIAN_INTEGRATION.md @@ -455,6 +455,154 @@ LIBRARY_BATCH_SIZE=50 LIBRARY_SYNC_ENABLED=true ``` +## Maintenance Tasks + +### Index Reconciliation (Daily) + +The `reconcile-index` endpoint performs full index maintenance: + +1. **Cleanup Phase**: Remove orphaned data + - Vector chunks without wiki source + - Graph nodes without vectors (bidirectional) + - Vectors without graph nodes (bidirectional) + - Orphan entities (no MENTIONS relationships) + - Broken relationships + +2. **Reindex Phase**: Index missing pages + - Wiki pages without vector embeddings + - Wiki pages without graph Document nodes + +**Scheduler Task: `library_reconcile_index`** + +```yaml +Task Name: library_reconcile_index +Description: Daily index reconciliation - cleanup orphans + reindex missing pages +Schedule: Daily at 04:00 (after library_sync at 03:30) +Priority: 10 (system maintenance) +Service: library +Executor: POST /maintenance/reconcile-index +Configuration: + - LIBRARY_DESK_URL: http://library-desk:8089 + - LIBRARY_API_KEY: ${LIBRARY_API_KEY} +Parameters: + - user: jpmschweitzer + - dry_run: false +Outputs: + - Vector orphans purged + - Entity orphans purged + - Missing pages reindexed +``` + +### Maintenance Endpoints + +| Endpoint | Method | Purpose | +|----------|--------|---------| +| `/maintenance/reconcile-index` | POST | **Recommended**: Full cleanup + reindex missing | +| `/maintenance/cleanup/all` | POST | Cleanup only (orphan removal) | +| `/maintenance/cleanup/vectors` | POST | Clean orphan vector chunks only | +| `/maintenance/cleanup/graph` | POST | Clean orphan entities & stale docs only | +| `/maintenance/health` | GET | Lightweight health check (for uptime monitoring) | +| `/maintenance/health?detailed=true` | GET | Full analysis with orphan counts | +| `/maintenance/reindex/{page_id}` | POST | Force re-index a specific page | + +### Health Check Modes + +**Lightweight (default)** - Use for frequent uptime checks (every 30s): +```bash +curl "http://library-desk:8089/maintenance/health?user=jpmschweitzer" \ + -H "Authorization: Bearer ${LIBRARY_API_KEY}" +``` + +Returns only last cleanup timestamp and basic status (no database queries). + +**Detailed** - Use for dashboards or before reconciliation: +```bash +curl "http://library-desk:8089/maintenance/health?user=jpmschweitzer&detailed=true" \ + -H "Authorization: Bearer ${LIBRARY_API_KEY}" +``` + +Returns full orphan analysis (runs database queries). + +### Example Reconcile Request + +```bash +curl -X POST "http://library-desk:8089/maintenance/reconcile-index?user=jpmschweitzer" \ + -H "Authorization: Bearer ${LIBRARY_API_KEY}" +``` + +### Example Response + +```json +{ + "success": true, + "cleanup": { + "success": true, + "vector_cleanup": { + "wiki_chunks": {"orphans_found": 5, "orphans_purged": 5}, + "document_chunks": {"orphans_found": 0, "orphans_purged": 0}, + "chunks_without_graph": {"orphans_found": 2, "orphans_purged": 2}, + "total_chunks_scanned": 1250, + "total_orphans_purged": 7 + }, + "graph_cleanup": { + "orphan_entities": {"orphans_found": 3, "orphans_purged": 3}, + "stale_wiki_documents": {"orphans_found": 1, "orphans_purged": 1}, + "stale_store_documents": {"orphans_found": 0, "orphans_purged": 0}, + "docs_without_vectors": {"orphans_found": 0, "orphans_purged": 0}, + "broken_relationships_cleaned": 0 + }, + "total_duration_ms": 1523.5 + }, + "reindex_missing": { + "pages_without_vectors": 2, + "pages_without_graph": 1, + "pages_reindexed": 2, + "pages_failed": 0, + "failed_page_ids": [], + "duration_ms": 3421.2 + }, + "total_duration_ms": 4944.7 +} +``` + +### Scheduler Integration Code + +```python +# scheduler/src/tasks/library_maintenance.py + +async def library_reconcile_index_task(user: str = "jpmschweitzer"): + """Run daily Library Desk index reconciliation.""" + + async with httpx.AsyncClient() as client: + # Run reconcile-index (cleanup + reindex missing) + result = await client.post( + f"{LIBRARY_DESK_URL}/maintenance/reconcile-index", + params={"user": user, "dry_run": False}, + headers={"Authorization": f"Bearer {LIBRARY_API_KEY}"}, + timeout=600.0 # 10 minutes for large indexes + ) + + data = result.json() + + # Log summary + cleanup = data["cleanup"] + reindex = data["reindex_missing"] + + logger.info( + f"Reconcile complete: " + f"{cleanup['vector_cleanup']['total_orphans_purged']} vector orphans, " + f"{cleanup['graph_cleanup']['orphan_entities']['orphans_purged']} entity orphans, " + f"{reindex['pages_reindexed']} pages reindexed" + ) + + if reindex["pages_failed"] > 0: + logger.warning(f"Failed to reindex pages: {reindex['failed_page_ids']}") + + return data +``` + +--- + ## Next Steps 1. Implement ingestion endpoints in Library Desk diff --git a/docs/MEMORY_SYSTEM_PLAN.md b/docs/MEMORY_SYSTEM_PLAN.md new file mode 100644 index 0000000..dfb7865 --- /dev/null +++ b/docs/MEMORY_SYSTEM_PLAN.md @@ -0,0 +1,283 @@ +# Memory Management System - Implementation Plan + +## Overview + +A three-tier memory architecture for Library Desk with intelligent orchestration: + +| Tier | Storage | Purpose | TTL | +|------|---------|---------|-----| +| **Volatile** | Redis | Weather, news, financial, ephemeral context | 5min - 2hr | +| **Documents** | TBD (research) | Git mirrors, PDFs, video, images | Permanent | +| **Knowledge** | Wiki + Neo4j | Personal dossiers, research, summaries | Permanent | + +**Implementation Priority**: Cleanup โ†’ Volatile โ†’ Documents + +--- + +## Phase 1: Cleanup System Completion + +### Current State +- **COMPLETE** - All Phase 1 tasks implemented +- Redis timestamp tracking for last cleanup +- Bidirectional orphan detection between vectors and graph +- Scheduler integration endpoints ready + +### Tasks + +#### 1.1 Add Scheduler Integration Points โœ… +**Files**: `src/routers/maintenance.py` + +- [x] Add `last_cleanup` timestamp tracking in Redis +- [x] Return cleanup stats in format scheduler can log +- [x] Added `RedisDep` to cleanup endpoints + +#### 1.2 Bidirectional Orphan Detection โœ… +**Files**: `src/services/graph_service.py`, `src/services/vector_service.py` + +- [x] `find_documents_without_vectors()` - graph nodes with no vectors +- [x] `find_chunks_without_graph_nodes()` - vectors with no graph node +- [x] Updated maintenance endpoints to use bidirectional checks +- [x] Added `chunks_without_graph` and `docs_without_vectors` to response models + +#### 1.3 Scheduler Configuration โœ… +**Scheduler-side task definition:** +```json +{ + "task_name": "library_reconcile_index", + "schedule": "0 4 * * *", + "endpoint": "POST /maintenance/reconcile-index?user=jpmschweitzer", + "description": "Daily index reconciliation - cleanup + reindex missing" +} +``` + +- [x] Documented in `LIBRARIAN_INTEGRATION.md` +- [x] Added `reconcile-index` endpoint (cleanup + reindex missing) +- [x] Lightweight health check mode for uptime monitoring +- [x] Detailed health check mode for dashboards + +--- + +## Phase 2: Volatile Memory System + +### Architecture + +``` +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ Library-Desk โ”‚โ—„โ”€โ”€โ”€โ”‚ Scheduler โ”‚โ”€โ”€โ”€โ–บโ”‚ External APIs โ”‚ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ (weather, news) โ”‚ +โ”‚ VolatileCache โ”‚ โ”‚ Refresh โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +โ”‚ Service โ”‚ โ”‚ Jobs โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ + โ–ผ +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ Redis โ”‚ +โ”‚ (DB 4, TTL) โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +### Data Model + +```python +class VolatileRecord(BaseModel): + key: str # e.g., "weather:rotterdam" + namespace: str # e.g., "weather", "news", "financial" + data: dict # Actual content + source: Optional[str] # Origin API/service + created_at: datetime + updated_at: datetime + ttl: int # Seconds until expiration + refresh_schedule: Optional[str] # Cron expression, if repeating + user: str # Multi-tenant isolation +``` + +**Key pattern**: `{user}:volatile:{namespace}:{key_hash}` + +### Implementation Order: Integration-First + +1. **Start with Consolidation Hook** - Understand data flow through existing system +2. **Build Service Layer** - VolatileCacheService with Redis operations +3. **Add API Endpoints** - REST interface for volatile data +4. **Biographer Integration** - Query user preferences for relevance + +### Tasks + +#### 2.1 Integrate with Consolidation (FIRST) +**New file**: `src/services/volatile_service.py` + +```python +class VolatileCacheService: + async def get(user, namespace, key) -> Optional[VolatileRecord] + async def set(user, namespace, key, data, ttl, refresh_schedule=None) + async def delete(user, namespace, key) + async def list_namespace(user, namespace) -> List[str] + async def get_scheduled(user) -> List[VolatileRecord] # For scheduler +``` + +#### 2.2 Create Volatile API Router +**New file**: `src/routers/volatile.py` + +| Endpoint | Method | Purpose | +|----------|--------|---------| +| `/volatile/{namespace}/{key}` | GET | Retrieve record | +| `/volatile/{namespace}/{key}` | POST | Store/update record | +| `/volatile/{namespace}/{key}` | DELETE | Remove record | +| `/volatile/{namespace}` | GET | List keys in namespace | +| `/volatile/scheduled` | GET | List records needing refresh | +| `/volatile/stats` | GET | Cache statistics | + +#### 2.3 Integrate with Consolidation +**File**: `src/services/consolidation_service.py` + +Add relevance trigger detection: +1. During consolidation, analyze search results for location/interest patterns +2. Query tatlock's Biographer collection for user preferences +3. If match found, create/update volatile refresh schedule + +#### 2.4 Biographer Integration +**File**: `src/core/dependencies.py` + +```python +def get_biographer_qdrant() -> QdrantClientWrapper: + """Direct access to tatlock's Biographer collection.""" + # Configure to connect to tatlock's Qdrant +``` + +#### 2.5 Scheduler-Side Configuration +Document required scheduler tasks: +```json +{ + "task_name": "volatile_refresh", + "schedule": "*/15 * * * *", + "endpoint": "GET /volatile/scheduled", + "follow_up": "For each record, call refresh endpoint with record.refresh_schedule" +} +``` + +--- + +## Phase 3: Document Storage (Research + Implementation) + +### Research Scope + +Evaluate FOSS self-hosted options for: +- Git repository mirroring +- PDF/document storage with metadata +- Image/video blob storage +- Full-text search capability + +**Constraints**: +- Must be self-hosted, Docker-deployable +- Performance is priority (can wrap complexity in API) +- No cloud dependencies + +**Candidates to evaluate**: +1. MinIO (S3-compatible object storage) + metadata in Neo4j +2. Paperless-ngx (document management with OCR) +3. SeaweedFS (distributed file system) +4. Custom: filesystem + Neo4j metadata + +### Category Descriptors + +**Wiki page structure for document collections**: +```markdown +# FastAPI Documentation + +## Overview +[LLM-generated summary from web search about FastAPI] + +## Collection Statistics +- **Documents**: 342 files +- **Last Sync**: 2025-12-24 03:30 UTC +- **Source**: github.com/tiangolo/fastapi +- **Coverage**: API reference, tutorials, deployment guides + +## What's Included +[LLM summary of collection contents based on document analysis] + +## Related Topics +- [[Python Web Frameworks]] +- [[REST API Design]] +``` + +### Tasks + +#### 3.1 Storage Research +**Deliverable**: Evaluation document comparing options + +#### 3.2 Storage Service Implementation +**New file**: `src/services/document_store_service.py` +(Details pending research results) + +#### 3.3 Category Descriptor Generation +**File**: `src/services/consolidation_service.py` + +Add LLM-powered category descriptor generation: +1. Web search for topic overview +2. Analyze collection contents +3. Generate/update wiki page with template + +--- + +## Files to Modify/Create + +### Phase 1 (Cleanup) +- `src/routers/maintenance.py` - Add timestamp tracking +- `src/services/graph_service.py` - Bidirectional validation +- `src/services/vector_service.py` - Cross-reference checks +- `LIBRARIAN_INTEGRATION.md` - Scheduler config docs + +### Phase 2 (Volatile) +- `src/services/volatile_service.py` - **NEW** +- `src/routers/volatile.py` - **NEW** +- `src/models/volatile.py` - **NEW** +- `src/core/dependencies.py` - Add Biographer client +- `src/services/consolidation_service.py` - Relevance triggers +- `tests/test_volatile.py` - **NEW** + +### Phase 3 (Documents) +- `docs/DOCUMENT_STORAGE_RESEARCH.md` - **NEW** +- `src/services/document_store_service.py` - **NEW** (post-research) +- `src/routers/documents.py` - **NEW** (post-research) + +--- + +## Resolved Design Decisions + +1. **Biographer Qdrant**: Same Qdrant instance, different collection. Library-Desk queries directly. +2. **Scheduler API**: Has REST API for task registration. Library-Desk can programmatically create refresh schedules. +3. **External API calls**: Library-Desk routes through SearXNG for web search. Consider dedicated API integrations for high-value volatiles (weather, financial) for consistent quality. + +--- + +## Future Consideration: Dedicated API Integrations + +For volatile data where quality/consistency matters (weather, financial), consider: +- OpenWeatherMap API for weather (daily refresh cycle) +- Financial data API (Alpha Vantage, Yahoo Finance) +- News APIs (NewsAPI, GDELT) +- **NOS.nl** - Explicit source for Dutch news + +This would live in a new `src/clients/` module with: +- `weather_client.py` - Daily refresh cycle +- `financial_client.py` +- `news_client.py` - Include NOS.nl scraper/API for Dutch coverage + +These provide structured, reliable data vs. SearXNG web scraping. Implementation deferred to later phase. + +--- + +## Refresh Schedules + +**Note:** TTL should be longer than refresh interval to prevent data gaps. + +| Volatile Type | TTL | Refresh Cycle | Refresh Interval | Sources | +|---------------|-----|---------------|------------------|---------| +| Weather | 86400s (24hr) | Daily | Every 24hr | OpenWeatherMap | +| Dutch News | 28800s (8hr) | 4x daily | Every 6hr | NOS.nl | +| Global News | 28800s (8hr) | 4x daily | Every 6hr | NewsAPI, GDELT | +| Financial | 600s (10min) | On-demand | N/A | Alpha Vantage | + +**TTL Logic:** +- TTL = Refresh Interval ร— 1.5 (buffer for failed refreshes) +- On-demand data gets shorter TTL since it's fetched when needed