library-desk/docs/MEMORY_SYSTEM_PLAN.md

Memory Management System - Implementation Plan

Overview

A three-tier memory architecture for Library Desk with intelligent orchestration:

Tier	Storage	Purpose	TTL
Volatile	Redis	Weather, news, financial, ephemeral context	5min - 2hr
Documents	TBD (research)	Git mirrors, PDFs, video, images	Permanent
Knowledge	Wiki + Neo4j	Personal dossiers, research, summaries	Permanent

Implementation Priority: Cleanup → Volatile → Documents

---

Phase 1: Cleanup System Completion

Current State

• COMPLETE - All Phase 1 tasks implemented
• Redis timestamp tracking for last cleanup
• Bidirectional orphan detection between vectors and graph
• Scheduler integration endpoints ready

Tasks

1.1 Add Scheduler Integration Points ✅

Files: src/routers/maintenance.py

• Add last_cleanup timestamp tracking in Redis
• Return cleanup stats in format scheduler can log
• Added RedisDep to cleanup endpoints

1.2 Bidirectional Orphan Detection ✅

Files: src/services/graph_service.py, src/services/vector_service.py

• find_documents_without_vectors() - graph nodes with no vectors
• find_chunks_without_graph_nodes() - vectors with no graph node
• Updated maintenance endpoints to use bidirectional checks
• Added chunks_without_graph and docs_without_vectors to response models

1.3 Scheduler Configuration ✅

Scheduler-side task definition:

{
  "task_name": "library_reconcile_index",
  "schedule": "0 4 * * *",
  "endpoint": "POST /maintenance/reconcile-index?user=jpmschweitzer",
  "description": "Daily index reconciliation - cleanup + reindex missing"
}

• Documented in LIBRARIAN_INTEGRATION.md
• Added reconcile-index endpoint (cleanup + reindex missing)
• Lightweight health check mode for uptime monitoring
• Detailed health check mode for dashboards

---

Phase 2: Volatile Memory System

Architecture

┌─────────────────┐    ┌──────────────┐    ┌─────────────────┐
│  Library-Desk   │◄───│   Scheduler  │───►│ External APIs   │
│                 │    │              │    │ (weather, news) │
│  VolatileCache  │    │  Refresh     │    └─────────────────┘
│  Service        │    │  Jobs        │
└────────┬────────┘    └──────────────┘
         │
         ▼
┌─────────────────┐
│     Redis       │
│  (DB 4, TTL)    │
└─────────────────┘

Data Model

class VolatileRecord(BaseModel):
    key: str                           # e.g., "weather:rotterdam"
    namespace: str                     # e.g., "weather", "news", "financial"
    data: dict                         # Actual content
    source: Optional[str]              # Origin API/service
    created_at: datetime
    updated_at: datetime
    ttl: int                           # Seconds until expiration
    refresh_schedule: Optional[str]    # Cron expression, if repeating
    user: str                          # Multi-tenant isolation

Key pattern: {user}:volatile:{namespace}:{key_hash}

Implementation Order: Integration-First

1. Start with Consolidation Hook - Understand data flow through existing system
2. Build Service Layer - VolatileCacheService with Redis operations
3. Add API Endpoints - REST interface for volatile data
4. Biographer Integration - Query user preferences for relevance

Tasks

2.1 Integrate with Consolidation (FIRST)

New file: src/services/volatile_service.py

class VolatileCacheService:
    async def get(user, namespace, key) -> Optional[VolatileRecord]
    async def set(user, namespace, key, data, ttl, refresh_schedule=None)
    async def delete(user, namespace, key)
    async def list_namespace(user, namespace) -> List[str]
    async def get_scheduled(user) -> List[VolatileRecord]  # For scheduler

2.2 Create Volatile API Router

New file: src/routers/volatile.py

Endpoint	Method	Purpose
/volatile/{namespace}/{key}	GET	Retrieve record
/volatile/{namespace}/{key}	POST	Store/update record
/volatile/{namespace}/{key}	DELETE	Remove record
/volatile/{namespace}	GET	List keys in namespace
/volatile/scheduled	GET	List records needing refresh
/volatile/stats	GET	Cache statistics

2.3 Integrate with Consolidation

File: src/services/consolidation_service.py

Add relevance trigger detection:

1. During consolidation, analyze search results for location/interest patterns
2. Query tatlock's Biographer collection for user preferences
3. If match found, create/update volatile refresh schedule

2.4 Biographer Integration

File: src/core/dependencies.py

def get_biographer_qdrant() -> QdrantClientWrapper:
    """Direct access to tatlock's Biographer collection."""
    # Configure to connect to tatlock's Qdrant

2.5 Scheduler-Side Configuration

Document required scheduler tasks:

{
  "task_name": "volatile_refresh",
  "schedule": "*/15 * * * *",
  "endpoint": "GET /volatile/scheduled",
  "follow_up": "For each record, call refresh endpoint with record.refresh_schedule"
}

---

Phase 3: Document Storage (Research + Implementation)

Research Scope

Evaluate FOSS self-hosted options for:

• Git repository mirroring
• PDF/document storage with metadata
• Image/video blob storage
• Full-text search capability

Constraints:

• Must be self-hosted, Docker-deployable
• Performance is priority (can wrap complexity in API)
• No cloud dependencies

Candidates to evaluate:

1. MinIO (S3-compatible object storage) + metadata in Neo4j
2. Paperless-ngx (document management with OCR)
3. SeaweedFS (distributed file system)
4. Custom: filesystem + Neo4j metadata

Category Descriptors

Wiki page structure for document collections:

# FastAPI Documentation

## Overview
[LLM-generated summary from web search about FastAPI]

## Collection Statistics
- **Documents**: 342 files
- **Last Sync**: 2025-12-24 03:30 UTC
- **Source**: github.com/tiangolo/fastapi
- **Coverage**: API reference, tutorials, deployment guides

## What's Included
[LLM summary of collection contents based on document analysis]

## Related Topics
- [[Python Web Frameworks]]
- [[REST API Design]]

Tasks

3.1 Storage Research

Deliverable: Evaluation document comparing options

3.2 Storage Service Implementation

New file: src/services/document_store_service.py (Details pending research results)

3.3 Category Descriptor Generation

File: src/services/consolidation_service.py

Add LLM-powered category descriptor generation:

1. Web search for topic overview
2. Analyze collection contents
3. Generate/update wiki page with template

---

Files to Modify/Create

Phase 1 (Cleanup)

• src/routers/maintenance.py - Add timestamp tracking
• src/services/graph_service.py - Bidirectional validation
• src/services/vector_service.py - Cross-reference checks
• LIBRARIAN_INTEGRATION.md - Scheduler config docs

Phase 2 (Volatile)

• src/services/volatile_service.py - NEW
• src/routers/volatile.py - NEW
• src/models/volatile.py - NEW
• src/core/dependencies.py - Add Biographer client
• src/services/consolidation_service.py - Relevance triggers
• tests/test_volatile.py - NEW

Phase 3 (Documents)

• docs/DOCUMENT_STORAGE_RESEARCH.md - NEW
• src/services/document_store_service.py - NEW (post-research)
• src/routers/documents.py - NEW (post-research)

---

Resolved Design Decisions

1. Biographer Qdrant: Same Qdrant instance, different collection. Library-Desk queries directly.
2. Scheduler API: Has REST API for task registration. Library-Desk can programmatically create refresh schedules.
3. External API calls: Library-Desk routes through SearXNG for web search. Consider dedicated API integrations for high-value volatiles (weather, financial) for consistent quality.

---

Future Consideration: Dedicated API Integrations

For volatile data where quality/consistency matters (weather, financial), consider:

• OpenWeatherMap API for weather (daily refresh cycle)
• Financial data API (Alpha Vantage, Yahoo Finance)
• News APIs (NewsAPI, GDELT)
• NOS.nl - Explicit source for Dutch news

This would live in a new src/clients/ module with:

• weather_client.py - Daily refresh cycle
• financial_client.py
• news_client.py - Include NOS.nl scraper/API for Dutch coverage

These provide structured, reliable data vs. SearXNG web scraping. Implementation deferred to later phase.

---

Refresh Schedules

Note: TTL should be longer than refresh interval to prevent data gaps.

Volatile Type	TTL	Refresh Cycle	Refresh Interval	Sources
Weather	86400s (24hr)	Daily	Every 24hr	OpenWeatherMap
Dutch News	28800s (8hr)	4x daily	Every 6hr	NOS.nl
Global News	28800s (8hr)	4x daily	Every 6hr	NewsAPI, GDELT
Financial	600s (10min)	On-demand	N/A	Alpha Vantage

TTL Logic:

• TTL = Refresh Interval × 1.5 (buffer for failed refreshes)
• On-demand data gets shorter TTL since it's fetched when needed