docs: add scheduler integration and memory system plan
Documentation updates: - AGENTS.md: Added wakeup.sh usage and local testing instructions - LIBRARIAN_INTEGRATION.md: Complete maintenance scheduler docs - Scheduled task configuration for reconcile-index - Endpoint specifications and response formats - docs/MEMORY_SYSTEM_PLAN.md: Three-tier memory architecture - Volatile (Redis TTL) for ephemeral context - Documents (TBD) for git mirrors, PDFs, images - Knowledge (Wiki + Neo4j) for permanent research 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.5
@@ -0,0 +1,283 @@
|
||||
# Memory Management System - Implementation Plan
|
||||
|
||||
## Overview
|
||||
|
||||
A three-tier memory architecture for Library Desk with intelligent orchestration:
|
||||
|
||||
| Tier | Storage | Purpose | TTL |
|
||||
|------|---------|---------|-----|
|
||||
| **Volatile** | Redis | Weather, news, financial, ephemeral context | 5min - 2hr |
|
||||
| **Documents** | TBD (research) | Git mirrors, PDFs, video, images | Permanent |
|
||||
| **Knowledge** | Wiki + Neo4j | Personal dossiers, research, summaries | Permanent |
|
||||
|
||||
**Implementation Priority**: Cleanup → Volatile → Documents
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Cleanup System Completion
|
||||
|
||||
### Current State
|
||||
- **COMPLETE** - All Phase 1 tasks implemented
|
||||
- Redis timestamp tracking for last cleanup
|
||||
- Bidirectional orphan detection between vectors and graph
|
||||
- Scheduler integration endpoints ready
|
||||
|
||||
### Tasks
|
||||
|
||||
#### 1.1 Add Scheduler Integration Points ✅
|
||||
**Files**: `src/routers/maintenance.py`
|
||||
|
||||
- [x] Add `last_cleanup` timestamp tracking in Redis
|
||||
- [x] Return cleanup stats in format scheduler can log
|
||||
- [x] Added `RedisDep` to cleanup endpoints
|
||||
|
||||
#### 1.2 Bidirectional Orphan Detection ✅
|
||||
**Files**: `src/services/graph_service.py`, `src/services/vector_service.py`
|
||||
|
||||
- [x] `find_documents_without_vectors()` - graph nodes with no vectors
|
||||
- [x] `find_chunks_without_graph_nodes()` - vectors with no graph node
|
||||
- [x] Updated maintenance endpoints to use bidirectional checks
|
||||
- [x] Added `chunks_without_graph` and `docs_without_vectors` to response models
|
||||
|
||||
#### 1.3 Scheduler Configuration ✅
|
||||
**Scheduler-side task definition:**
|
||||
```json
|
||||
{
|
||||
"task_name": "library_reconcile_index",
|
||||
"schedule": "0 4 * * *",
|
||||
"endpoint": "POST /maintenance/reconcile-index?user=jpmschweitzer",
|
||||
"description": "Daily index reconciliation - cleanup + reindex missing"
|
||||
}
|
||||
```
|
||||
|
||||
- [x] Documented in `LIBRARIAN_INTEGRATION.md`
|
||||
- [x] Added `reconcile-index` endpoint (cleanup + reindex missing)
|
||||
- [x] Lightweight health check mode for uptime monitoring
|
||||
- [x] Detailed health check mode for dashboards
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Volatile Memory System
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌──────────────┐ ┌─────────────────┐
|
||||
│ Library-Desk │◄───│ Scheduler │───►│ External APIs │
|
||||
│ │ │ │ │ (weather, news) │
|
||||
│ VolatileCache │ │ Refresh │ └─────────────────┘
|
||||
│ Service │ │ Jobs │
|
||||
└────────┬────────┘ └──────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Redis │
|
||||
│ (DB 4, TTL) │
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
### Data Model
|
||||
|
||||
```python
|
||||
class VolatileRecord(BaseModel):
|
||||
key: str # e.g., "weather:rotterdam"
|
||||
namespace: str # e.g., "weather", "news", "financial"
|
||||
data: dict # Actual content
|
||||
source: Optional[str] # Origin API/service
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
ttl: int # Seconds until expiration
|
||||
refresh_schedule: Optional[str] # Cron expression, if repeating
|
||||
user: str # Multi-tenant isolation
|
||||
```
|
||||
|
||||
**Key pattern**: `{user}:volatile:{namespace}:{key_hash}`
|
||||
|
||||
### Implementation Order: Integration-First
|
||||
|
||||
1. **Start with Consolidation Hook** - Understand data flow through existing system
|
||||
2. **Build Service Layer** - VolatileCacheService with Redis operations
|
||||
3. **Add API Endpoints** - REST interface for volatile data
|
||||
4. **Biographer Integration** - Query user preferences for relevance
|
||||
|
||||
### Tasks
|
||||
|
||||
#### 2.1 Integrate with Consolidation (FIRST)
|
||||
**New file**: `src/services/volatile_service.py`
|
||||
|
||||
```python
|
||||
class VolatileCacheService:
|
||||
async def get(user, namespace, key) -> Optional[VolatileRecord]
|
||||
async def set(user, namespace, key, data, ttl, refresh_schedule=None)
|
||||
async def delete(user, namespace, key)
|
||||
async def list_namespace(user, namespace) -> List[str]
|
||||
async def get_scheduled(user) -> List[VolatileRecord] # For scheduler
|
||||
```
|
||||
|
||||
#### 2.2 Create Volatile API Router
|
||||
**New file**: `src/routers/volatile.py`
|
||||
|
||||
| Endpoint | Method | Purpose |
|
||||
|----------|--------|---------|
|
||||
| `/volatile/{namespace}/{key}` | GET | Retrieve record |
|
||||
| `/volatile/{namespace}/{key}` | POST | Store/update record |
|
||||
| `/volatile/{namespace}/{key}` | DELETE | Remove record |
|
||||
| `/volatile/{namespace}` | GET | List keys in namespace |
|
||||
| `/volatile/scheduled` | GET | List records needing refresh |
|
||||
| `/volatile/stats` | GET | Cache statistics |
|
||||
|
||||
#### 2.3 Integrate with Consolidation
|
||||
**File**: `src/services/consolidation_service.py`
|
||||
|
||||
Add relevance trigger detection:
|
||||
1. During consolidation, analyze search results for location/interest patterns
|
||||
2. Query tatlock's Biographer collection for user preferences
|
||||
3. If match found, create/update volatile refresh schedule
|
||||
|
||||
#### 2.4 Biographer Integration
|
||||
**File**: `src/core/dependencies.py`
|
||||
|
||||
```python
|
||||
def get_biographer_qdrant() -> QdrantClientWrapper:
|
||||
"""Direct access to tatlock's Biographer collection."""
|
||||
# Configure to connect to tatlock's Qdrant
|
||||
```
|
||||
|
||||
#### 2.5 Scheduler-Side Configuration
|
||||
Document required scheduler tasks:
|
||||
```json
|
||||
{
|
||||
"task_name": "volatile_refresh",
|
||||
"schedule": "*/15 * * * *",
|
||||
"endpoint": "GET /volatile/scheduled",
|
||||
"follow_up": "For each record, call refresh endpoint with record.refresh_schedule"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Document Storage (Research + Implementation)
|
||||
|
||||
### Research Scope
|
||||
|
||||
Evaluate FOSS self-hosted options for:
|
||||
- Git repository mirroring
|
||||
- PDF/document storage with metadata
|
||||
- Image/video blob storage
|
||||
- Full-text search capability
|
||||
|
||||
**Constraints**:
|
||||
- Must be self-hosted, Docker-deployable
|
||||
- Performance is priority (can wrap complexity in API)
|
||||
- No cloud dependencies
|
||||
|
||||
**Candidates to evaluate**:
|
||||
1. MinIO (S3-compatible object storage) + metadata in Neo4j
|
||||
2. Paperless-ngx (document management with OCR)
|
||||
3. SeaweedFS (distributed file system)
|
||||
4. Custom: filesystem + Neo4j metadata
|
||||
|
||||
### Category Descriptors
|
||||
|
||||
**Wiki page structure for document collections**:
|
||||
```markdown
|
||||
# FastAPI Documentation
|
||||
|
||||
## Overview
|
||||
[LLM-generated summary from web search about FastAPI]
|
||||
|
||||
## Collection Statistics
|
||||
- **Documents**: 342 files
|
||||
- **Last Sync**: 2025-12-24 03:30 UTC
|
||||
- **Source**: github.com/tiangolo/fastapi
|
||||
- **Coverage**: API reference, tutorials, deployment guides
|
||||
|
||||
## What's Included
|
||||
[LLM summary of collection contents based on document analysis]
|
||||
|
||||
## Related Topics
|
||||
- [[Python Web Frameworks]]
|
||||
- [[REST API Design]]
|
||||
```
|
||||
|
||||
### Tasks
|
||||
|
||||
#### 3.1 Storage Research
|
||||
**Deliverable**: Evaluation document comparing options
|
||||
|
||||
#### 3.2 Storage Service Implementation
|
||||
**New file**: `src/services/document_store_service.py`
|
||||
(Details pending research results)
|
||||
|
||||
#### 3.3 Category Descriptor Generation
|
||||
**File**: `src/services/consolidation_service.py`
|
||||
|
||||
Add LLM-powered category descriptor generation:
|
||||
1. Web search for topic overview
|
||||
2. Analyze collection contents
|
||||
3. Generate/update wiki page with template
|
||||
|
||||
---
|
||||
|
||||
## Files to Modify/Create
|
||||
|
||||
### Phase 1 (Cleanup)
|
||||
- `src/routers/maintenance.py` - Add timestamp tracking
|
||||
- `src/services/graph_service.py` - Bidirectional validation
|
||||
- `src/services/vector_service.py` - Cross-reference checks
|
||||
- `LIBRARIAN_INTEGRATION.md` - Scheduler config docs
|
||||
|
||||
### Phase 2 (Volatile)
|
||||
- `src/services/volatile_service.py` - **NEW**
|
||||
- `src/routers/volatile.py` - **NEW**
|
||||
- `src/models/volatile.py` - **NEW**
|
||||
- `src/core/dependencies.py` - Add Biographer client
|
||||
- `src/services/consolidation_service.py` - Relevance triggers
|
||||
- `tests/test_volatile.py` - **NEW**
|
||||
|
||||
### Phase 3 (Documents)
|
||||
- `docs/DOCUMENT_STORAGE_RESEARCH.md` - **NEW**
|
||||
- `src/services/document_store_service.py` - **NEW** (post-research)
|
||||
- `src/routers/documents.py` - **NEW** (post-research)
|
||||
|
||||
---
|
||||
|
||||
## Resolved Design Decisions
|
||||
|
||||
1. **Biographer Qdrant**: Same Qdrant instance, different collection. Library-Desk queries directly.
|
||||
2. **Scheduler API**: Has REST API for task registration. Library-Desk can programmatically create refresh schedules.
|
||||
3. **External API calls**: Library-Desk routes through SearXNG for web search. Consider dedicated API integrations for high-value volatiles (weather, financial) for consistent quality.
|
||||
|
||||
---
|
||||
|
||||
## Future Consideration: Dedicated API Integrations
|
||||
|
||||
For volatile data where quality/consistency matters (weather, financial), consider:
|
||||
- OpenWeatherMap API for weather (daily refresh cycle)
|
||||
- Financial data API (Alpha Vantage, Yahoo Finance)
|
||||
- News APIs (NewsAPI, GDELT)
|
||||
- **NOS.nl** - Explicit source for Dutch news
|
||||
|
||||
This would live in a new `src/clients/` module with:
|
||||
- `weather_client.py` - Daily refresh cycle
|
||||
- `financial_client.py`
|
||||
- `news_client.py` - Include NOS.nl scraper/API for Dutch coverage
|
||||
|
||||
These provide structured, reliable data vs. SearXNG web scraping. Implementation deferred to later phase.
|
||||
|
||||
---
|
||||
|
||||
## Refresh Schedules
|
||||
|
||||
**Note:** TTL should be longer than refresh interval to prevent data gaps.
|
||||
|
||||
| Volatile Type | TTL | Refresh Cycle | Refresh Interval | Sources |
|
||||
|---------------|-----|---------------|------------------|---------|
|
||||
| Weather | 86400s (24hr) | Daily | Every 24hr | OpenWeatherMap |
|
||||
| Dutch News | 28800s (8hr) | 4x daily | Every 6hr | NOS.nl |
|
||||
| Global News | 28800s (8hr) | 4x daily | Every 6hr | NewsAPI, GDELT |
|
||||
| Financial | 600s (10min) | On-demand | N/A | Alpha Vantage |
|
||||
|
||||
**TTL Logic:**
|
||||
- TTL = Refresh Interval × 1.5 (buffer for failed refreshes)
|
||||
- On-demand data gets shorter TTL since it's fetched when needed
|
||||
Reference in New Issue
Block a user