library-desk/docs/MEMORY_SYSTEM_PLAN.md

# Memory Management System - Implementation Plan

## Overview

A three-tier memory architecture for Library Desk with intelligent orchestration:

| Tier | Storage | Purpose | TTL |
|------|---------|---------|-----|
| **Volatile** | Redis | Weather, news, financial, ephemeral context | 5min - 2hr |
| **Documents** | TBD (research) | Git mirrors, PDFs, video, images | Permanent |
| **Knowledge** | Wiki + Neo4j | Personal dossiers, research, summaries | Permanent |

**Implementation Priority**: Cleanup → Volatile → Documents

---

## Phase 1: Cleanup System Completion

### Current State
- **COMPLETE** - All Phase 1 tasks implemented
- Redis timestamp tracking for last cleanup
- Bidirectional orphan detection between vectors and graph
- Scheduler integration endpoints ready

### Tasks

#### 1.1 Add Scheduler Integration Points ✅
**Files**: `src/routers/maintenance.py`

- [x] Add `last_cleanup` timestamp tracking in Redis
- [x] Return cleanup stats in format scheduler can log
- [x] Added `RedisDep` to cleanup endpoints

#### 1.2 Bidirectional Orphan Detection ✅
**Files**: `src/services/graph_service.py`, `src/services/vector_service.py`

- [x] `find_documents_without_vectors()` - graph nodes with no vectors
- [x] `find_chunks_without_graph_nodes()` - vectors with no graph node
- [x] Updated maintenance endpoints to use bidirectional checks
- [x] Added `chunks_without_graph` and `docs_without_vectors` to response models

#### 1.3 Scheduler Configuration ✅
**Scheduler-side task definition:**
```json
{
  "task_name": "library_reconcile_index",
  "schedule": "0 4 * * *",
  "endpoint": "POST /maintenance/reconcile-index?user=jpmschweitzer",
  "description": "Daily index reconciliation - cleanup + reindex missing"
}
```

- [x] Documented in `LIBRARIAN_INTEGRATION.md`
- [x] Added `reconcile-index` endpoint (cleanup + reindex missing)
- [x] Lightweight health check mode for uptime monitoring
- [x] Detailed health check mode for dashboards

---

## Phase 2: Volatile Memory System

### Architecture

```
┌─────────────────┐    ┌──────────────┐    ┌─────────────────┐
│  Library-Desk   │◄───│   Scheduler  │───►│ External APIs   │
│                 │    │              │    │ (weather, news) │
│  VolatileCache  │    │  Refresh     │    └─────────────────┘
│  Service        │    │  Jobs        │
└────────┬────────┘    └──────────────┘
         │
         ▼
┌─────────────────┐
│     Redis       │
│  (DB 4, TTL)    │
└─────────────────┘
```

### Data Model

```python
class VolatileRecord(BaseModel):
    key: str                           # e.g., "weather:rotterdam"
    namespace: str                     # e.g., "weather", "news", "financial"
    data: dict                         # Actual content
    source: Optional[str]              # Origin API/service
    created_at: datetime
    updated_at: datetime
    ttl: int                           # Seconds until expiration
    refresh_schedule: Optional[str]    # Cron expression, if repeating
    user: str                          # Multi-tenant isolation
```

**Key pattern**: `{user}:volatile:{namespace}:{key_hash}`

### Implementation Order: Integration-First

1. **Start with Consolidation Hook** - Understand data flow through existing system
2. **Build Service Layer** - VolatileCacheService with Redis operations
3. **Add API Endpoints** - REST interface for volatile data
4. **Biographer Integration** - Query user preferences for relevance

### Tasks

#### 2.1 Integrate with Consolidation (FIRST)
**New file**: `src/services/volatile_service.py`

```python
class VolatileCacheService:
    async def get(user, namespace, key) -> Optional[VolatileRecord]
    async def set(user, namespace, key, data, ttl, refresh_schedule=None)
    async def delete(user, namespace, key)
    async def list_namespace(user, namespace) -> List[str]
    async def get_scheduled(user) -> List[VolatileRecord]  # For scheduler
```

#### 2.2 Create Volatile API Router
**New file**: `src/routers/volatile.py`

| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/volatile/{namespace}/{key}` | GET | Retrieve record |
| `/volatile/{namespace}/{key}` | POST | Store/update record |
| `/volatile/{namespace}/{key}` | DELETE | Remove record |
| `/volatile/{namespace}` | GET | List keys in namespace |
| `/volatile/scheduled` | GET | List records needing refresh |
| `/volatile/stats` | GET | Cache statistics |

#### 2.3 Integrate with Consolidation
**File**: `src/services/consolidation_service.py`

Add relevance trigger detection:
1. During consolidation, analyze search results for location/interest patterns
2. Query tatlock's Biographer collection for user preferences
3. If match found, create/update volatile refresh schedule

#### 2.4 Biographer Integration
**File**: `src/core/dependencies.py`

```python
def get_biographer_qdrant() -> QdrantClientWrapper:
    """Direct access to tatlock's Biographer collection."""
    # Configure to connect to tatlock's Qdrant
```

#### 2.5 Scheduler-Side Configuration
Document required scheduler tasks:
```json
{
  "task_name": "volatile_refresh",
  "schedule": "*/15 * * * *",
  "endpoint": "GET /volatile/scheduled",
  "follow_up": "For each record, call refresh endpoint with record.refresh_schedule"
}
```

---

## Phase 3: Document Storage (Research + Implementation)

### Research Scope

Evaluate FOSS self-hosted options for:
- Git repository mirroring
- PDF/document storage with metadata
- Image/video blob storage
- Full-text search capability

**Constraints**:
- Must be self-hosted, Docker-deployable
- Performance is priority (can wrap complexity in API)
- No cloud dependencies

**Candidates to evaluate**:
1. MinIO (S3-compatible object storage) + metadata in Neo4j
2. Paperless-ngx (document management with OCR)
3. SeaweedFS (distributed file system)
4. Custom: filesystem + Neo4j metadata

### Category Descriptors

**Wiki page structure for document collections**:
```markdown
# FastAPI Documentation

## Overview
[LLM-generated summary from web search about FastAPI]

## Collection Statistics
- **Documents**: 342 files
- **Last Sync**: 2025-12-24 03:30 UTC
- **Source**: github.com/tiangolo/fastapi
- **Coverage**: API reference, tutorials, deployment guides

## What's Included
[LLM summary of collection contents based on document analysis]

## Related Topics
- [[Python Web Frameworks]]
- [[REST API Design]]
```

### Tasks

#### 3.1 Storage Research
**Deliverable**: Evaluation document comparing options

#### 3.2 Storage Service Implementation
**New file**: `src/services/document_store_service.py`
(Details pending research results)

#### 3.3 Category Descriptor Generation
**File**: `src/services/consolidation_service.py`

Add LLM-powered category descriptor generation:
1. Web search for topic overview
2. Analyze collection contents
3. Generate/update wiki page with template

---

## Files to Modify/Create

### Phase 1 (Cleanup)
- `src/routers/maintenance.py` - Add timestamp tracking
- `src/services/graph_service.py` - Bidirectional validation
- `src/services/vector_service.py` - Cross-reference checks
- `LIBRARIAN_INTEGRATION.md` - Scheduler config docs

### Phase 2 (Volatile)
- `src/services/volatile_service.py` - **NEW**
- `src/routers/volatile.py` - **NEW**
- `src/models/volatile.py` - **NEW**
- `src/core/dependencies.py` - Add Biographer client
- `src/services/consolidation_service.py` - Relevance triggers
- `tests/test_volatile.py` - **NEW**

### Phase 3 (Documents)
- `docs/DOCUMENT_STORAGE_RESEARCH.md` - **NEW**
- `src/services/document_store_service.py` - **NEW** (post-research)
- `src/routers/documents.py` - **NEW** (post-research)

---

## Resolved Design Decisions

1. **Biographer Qdrant**: Same Qdrant instance, different collection. Library-Desk queries directly.
2. **Scheduler API**: Has REST API for task registration. Library-Desk can programmatically create refresh schedules.
3. **External API calls**: Library-Desk routes through SearXNG for web search. Consider dedicated API integrations for high-value volatiles (weather, financial) for consistent quality.

---

## Future Consideration: Dedicated API Integrations

For volatile data where quality/consistency matters (weather, financial), consider:
- OpenWeatherMap API for weather (daily refresh cycle)
- Financial data API (Alpha Vantage, Yahoo Finance)
- News APIs (NewsAPI, GDELT)
- **NOS.nl** - Explicit source for Dutch news

This would live in a new `src/clients/` module with:
- `weather_client.py` - Daily refresh cycle
- `financial_client.py`
- `news_client.py` - Include NOS.nl scraper/API for Dutch coverage

These provide structured, reliable data vs. SearXNG web scraping. Implementation deferred to later phase.

---

## Refresh Schedules

**Note:** TTL should be longer than refresh interval to prevent data gaps.

| Volatile Type | TTL | Refresh Cycle | Refresh Interval | Sources |
|---------------|-----|---------------|------------------|---------|
| Weather | 86400s (24hr) | Daily | Every 24hr | OpenWeatherMap |
| Dutch News | 28800s (8hr) | 4x daily | Every 6hr | NOS.nl |
| Global News | 28800s (8hr) | 4x daily | Every 6hr | NewsAPI, GDELT |
| Financial | 600s (10min) | On-demand | N/A | Alpha Vantage |

**TTL Logic:**
- TTL = Refresh Interval × 1.5 (buffer for failed refreshes)
- On-demand data gets shorter TTL since it's fetched when needed