library-desk/docs/MEMORY_SYSTEM_PLAN.md

# Memory Management System - Implementation Plan

## Overview

A three-tier memory architecture for Library Desk with intelligent orchestration:

| Tier | Storage | Purpose | TTL |
|------|---------|---------|-----|
| **Volatile** | Qdrant (vectors) | Weather, news, financial, ephemeral context | 5min - 2hr |
| **Documents** | Paperless-ngx + ClamAV (host) | Git mirrors, PDFs, video, images | Permanent |
| **Knowledge** | Wiki + Neo4j | Personal dossiers, research, summaries | Permanent |

**Implementation Priority**: Cleanup → Volatile → Documents → Test Data Cleanup

### Phase Status

| Phase | Status | Version |
|-------|--------|---------|
| Phase 1: Cleanup System | ✅ Complete | v1.4.0 |
| Phase 2: Volatile Memory | ✅ Complete | v1.4.3 |
| Phase 3: Document Storage | ✅ Planned | See [DOCUMENT_STORAGE_PLAN.md](DOCUMENT_STORAGE_PLAN.md) |
| Phase 4: Test Data Cleanup | ✅ Complete | v1.4.4 |

---

## Phase 1: Cleanup System Completion ✅

### Current State
- **COMPLETE** - All Phase 1 tasks implemented
- Redis timestamp tracking for last cleanup
- Bidirectional orphan detection between vectors and graph
- Scheduler integration endpoints ready

### Tasks

#### 1.1 Add Scheduler Integration Points ✅
**Files**: `src/routers/maintenance.py`

- [x] Add `last_cleanup` timestamp tracking in Redis
- [x] Return cleanup stats in format scheduler can log
- [x] Added `RedisDep` to cleanup endpoints

#### 1.2 Bidirectional Orphan Detection ✅
**Files**: `src/services/graph_service.py`, `src/services/vector_service.py`

- [x] `find_documents_without_vectors()` - graph nodes with no vectors
- [x] `find_chunks_without_graph_nodes()` - vectors with no graph node
- [x] Updated maintenance endpoints to use bidirectional checks
- [x] Added `chunks_without_graph` and `docs_without_vectors` to response models

#### 1.3 Scheduler Configuration ✅
**Scheduler-side task definition:**
```json
{
  "task_name": "library_reconcile_index",
  "schedule": "0 4 * * *",
  "endpoint": "POST /maintenance/reconcile-index?user=jpmschweitzer",
  "description": "Daily index reconciliation - cleanup + reindex missing"
}
```

- [x] Documented in `LIBRARIAN_INTEGRATION.md`
- [x] Added `reconcile-index` endpoint (cleanup + reindex missing)
- [x] Lightweight health check mode for uptime monitoring
- [x] Detailed health check mode for dashboards

---

## Phase 2: Volatile Memory System ✅

### Architecture (Final Implementation)

```
┌─────────────────┐    ┌──────────────┐    ┌─────────────────┐
│  Library-Desk   │◄───│   Scheduler  │───►│ External APIs   │
│                 │    │              │    │ (weather, news) │
│  VolatileCache  │    │  Refresh     │    └─────────────────┘
│  Service        │    │  Jobs        │
└────────┬────────┘    └──────────────┘
         │
         ▼
┌─────────────────┐
│     Qdrant      │
│ (volatile_{user})│
└─────────────────┘
```

**Key design decisions:**
- Vector storage in Qdrant (not Redis) for semantic search
- Collection per user: `volatile_{user}`
- TTL via `ttl_expiry` timestamp in payload
- Natural language conversion for embedding structured data
- Integrated into HybridRAG with priority boost

### Endpoints (Implemented)

| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/volatile/search?q=...` | GET | Semantic search across volatile data |
| `/volatile/store?namespace=...&key=...` | POST | Store/update record |
| `/volatile/{namespace}/{key}` | GET | Retrieve specific record |
| `/volatile/{namespace}/{key}` | DELETE | Remove record |
| `/volatile/stats` | GET | Cache statistics |
| `/volatile/scheduled` | GET | Records needing refresh |
| `/volatile/namespaces` | GET | List available namespaces |
| `/maintenance/cleanup/volatile` | POST | Purge expired records |

### Namespaces

| Namespace | Default TTL | Use Case |
|-----------|-------------|----------|
| weather | 30 min | Current conditions, forecasts |
| news | 1 hour | Headlines, breaking news |
| financial | 5 min | Stock prices, exchange rates |
| transit | 5 min | Train/bus schedules, delays |
| traffic | 10 min | Commute times, road conditions |
| air_quality | 1 hour | Pollution, pollen counts |
| sports | 1 min | Live scores, matches |
| social | 10 min | Social notifications |
| system | 1 min | Service health status |
| context | 1 hour | Session state |
| custom | 1 hour | User-defined data |

---

## Phase 3: Document Storage (Research + Implementation)

### Research Scope

Evaluate FOSS self-hosted options for:
- Git repository mirroring
- PDF/document storage with metadata
- Image/video blob storage
- Full-text search capability

**Constraints**:
- Must be self-hosted, Docker-deployable
- Performance is priority (can wrap complexity in API)
- No cloud dependencies

**Candidates to evaluate**:
1. MinIO (S3-compatible object storage) + metadata in Neo4j
2. Paperless-ngx (document management with OCR)
3. SeaweedFS (distributed file system)
4. Custom: filesystem + Neo4j metadata

### Category Descriptors

**Wiki page structure for document collections**:
```markdown
# FastAPI Documentation

## Overview
[LLM-generated summary from web search about FastAPI]

## Collection Statistics
- **Documents**: 342 files
- **Last Sync**: 2025-12-24 03:30 UTC
- **Source**: github.com/tiangolo/fastapi
- **Coverage**: API reference, tutorials, deployment guides

## What's Included
[LLM summary of collection contents based on document analysis]

## Related Topics
- [[Python Web Frameworks]]
- [[REST API Design]]
```

### Tasks

#### 3.1 Storage Research
**Deliverable**: Evaluation document comparing options

#### 3.2 Storage Service Implementation
**New file**: `src/services/document_store_service.py`
(Details pending research results)

#### 3.3 Category Descriptor Generation
**File**: `src/services/consolidation_service.py`

Add LLM-powered category descriptor generation:
1. Web search for topic overview
2. Analyze collection contents
3. Generate/update wiki page with template

---

## Phase 4: LLM Tester Data Cleanup ✅

### Problem

LLM testing creates accumulated cruft across the system:
- Wiki.js pages under `llm-tester/` and `llm_tester/` paths
- Graph nodes (Document, Entity) linked to test pages
- Vector chunks in Qdrant for test content

This data accumulates over time and clutters Wiki.js visually (no separate tenant scope for tests).

### Solution

Add a maintenance endpoint to purge all LLM tester artifacts across wiki, graph, and vectors.

### Tasks

#### 4.1 Identify Test Data Patterns ✅
**Patterns matched** (security-restricted to test user namespace):
- `users/llm-tester/*`
- `users/llm_tester/*`

#### 4.2 Add Cleanup Endpoint ✅
**File**: `src/routers/maintenance.py`

```python
@router.post("/cleanup/test-data")
async def cleanup_test_data(
    dry_run: bool = Query(default=True),
    wiki: WikiJSDep = None,
    vector_service: VectorServiceDep = None,
    graph_service: GraphServiceDep = None,
    api_key: str = Depends(verify_api_key)
):
    """
    Purge LLM tester data from wiki, graph, and vectors.

    **Security**: Only deletes pages in the test user namespace:
    - users/llm-tester/*
    - users/llm_tester/*

    Use dry_run=true to preview what would be deleted.
    """
```

#### 4.3 Implementation Steps ✅

1. **Wiki cleanup**: Delete pages via GraphQL mutation
2. **Graph cleanup**: Delete Document nodes using `delete_page()` method
3. **Vector cleanup**: Delete chunks using `delete_page_chunks()` method

#### 4.4 Scheduler Integration ✅
**Recommended schedule**: Weekly (Sunday 3:00 AM)

```json
{
  "task_name": "test_data_cleanup",
  "schedule": "0 3 * * 0",
  "endpoint": "POST /maintenance/cleanup/test-data?dry_run=false",
  "description": "Weekly cleanup of LLM test data"
}
```

### Files to Modify

- `src/routers/maintenance.py` - Add cleanup endpoint
- `src/services/wiki_service.py` - Add bulk delete by path pattern (if needed)
- `src/services/graph_service.py` - May need pattern-based node deletion
- `src/services/vector_service.py` - Add pattern-based chunk deletion

---

## Files Modified/Created

### Phase 1 (Cleanup) ✅
- `src/routers/maintenance.py` - Timestamp tracking, cleanup endpoints
- `src/services/graph_service.py` - Bidirectional validation
- `src/services/vector_service.py` - Cross-reference checks
- `LIBRARIAN_INTEGRATION.md` - Scheduler config docs

### Phase 2 (Volatile) ✅
- `src/services/volatile_service.py` - Qdrant-based volatile cache
- `src/routers/volatile.py` - Simplified endpoints
- `src/models/volatile.py` - Namespaces and models
- `src/models/hybrid_rag.py` - Volatile config options
- `src/services/hybrid_rag_service.py` - Volatile integration
- `src/clients/qdrant_client.py` - Expiry filter methods
- `tests/test_volatile.py` - 37 tests

### Phase 3 (Documents)
- `docs/DOCUMENT_STORAGE_RESEARCH.md` - **NEW**
- `src/services/document_store_service.py` - **NEW** (post-research)
- `src/routers/documents.py` - **NEW** (post-research)

### Phase 4 (Test Data Cleanup)
- `src/routers/maintenance.py` - Add cleanup endpoint
- `src/services/wiki_service.py` - Bulk delete by path pattern
- `src/services/graph_service.py` - Pattern-based node deletion
- `src/services/vector_service.py` - Pattern-based chunk deletion

---

## Resolved Design Decisions

1. **Volatile Storage**: Qdrant vectors (not Redis) for semantic search capability
2. **Collection Naming**: `volatile_{user}` for per-user isolation
3. **TTL Mechanism**: `ttl_expiry` timestamp in payload, background cleanup job
4. **HybridRAG Integration**: Volatile as third source with RRF priority boost
5. **Biographer Qdrant**: Same Qdrant instance, different collection
6. **Scheduler API**: Has REST API for task registration

---

## Future Consideration: Dedicated API Integrations

For volatile data where quality/consistency matters (weather, financial), consider:
- OpenWeatherMap API for weather (daily refresh cycle)
- Financial data API (Alpha Vantage, Yahoo Finance)
- News APIs (NewsAPI, GDELT)
- **NOS.nl** - Explicit source for Dutch news

This would live in a new `src/clients/` module with:
- `weather_client.py` - Daily refresh cycle
- `financial_client.py`
- `news_client.py` - Include NOS.nl scraper/API for Dutch coverage

These provide structured, reliable data vs. SearXNG web scraping. Implementation deferred to later phase.

---

## Refresh Schedules

**Note:** TTL should be longer than refresh interval to prevent data gaps.

| Volatile Type | TTL | Refresh Cycle | Refresh Interval | Sources |
|---------------|-----|---------------|------------------|---------|
| Weather | 86400s (24hr) | Daily | Every 24hr | OpenWeatherMap |
| Dutch News | 28800s (8hr) | 4x daily | Every 6hr | NOS.nl |
| Global News | 28800s (8hr) | 4x daily | Every 6hr | NewsAPI, GDELT |
| Financial | 600s (10min) | On-demand | N/A | Alpha Vantage |

**TTL Logic:**
- TTL = Refresh Interval × 1.5 (buffer for failed refreshes)
- On-demand data gets shorter TTL since it's fetched when needed