library-desk/docs/MEMORY_SYSTEM_PLAN.md

Memory Management System - Implementation Plan

Overview

A three-tier memory architecture for Library Desk with intelligent orchestration:

Tier	Storage	Purpose	TTL
Volatile	Qdrant (vectors)	Weather, news, financial, ephemeral context	5min - 2hr
Documents	Paperless-ngx + ClamAV (host)	Git mirrors, PDFs, video, images	Permanent
Knowledge	Wiki + Neo4j	Personal dossiers, research, summaries	Permanent

Implementation Priority: Cleanup → Volatile → Documents → Test Data Cleanup

Phase Status

Phase	Status	Version
Phase 1: Cleanup System	✅ Complete	v1.4.0
Phase 2: Volatile Memory	✅ Complete	v1.4.3
Phase 3: Document Storage	✅ Planned	See DOCUMENT_STORAGE_PLAN.md
Phase 4: Test Data Cleanup	✅ Complete	v1.4.4

---

Phase 1: Cleanup System Completion ✅

Current State

• COMPLETE - All Phase 1 tasks implemented
• Redis timestamp tracking for last cleanup
• Bidirectional orphan detection between vectors and graph
• Scheduler integration endpoints ready

Tasks

1.1 Add Scheduler Integration Points ✅

Files: src/routers/maintenance.py

• Add last_cleanup timestamp tracking in Redis
• Return cleanup stats in format scheduler can log
• Added RedisDep to cleanup endpoints

1.2 Bidirectional Orphan Detection ✅

Files: src/services/graph_service.py, src/services/vector_service.py

• find_documents_without_vectors() - graph nodes with no vectors
• find_chunks_without_graph_nodes() - vectors with no graph node
• Updated maintenance endpoints to use bidirectional checks
• Added chunks_without_graph and docs_without_vectors to response models

1.3 Scheduler Configuration ✅

Scheduler-side task definition:

{
  "task_name": "library_reconcile_index",
  "schedule": "0 4 * * *",
  "endpoint": "POST /maintenance/reconcile-index?user=jpmschweitzer",
  "description": "Daily index reconciliation - cleanup + reindex missing"
}

• Documented in LIBRARIAN_INTEGRATION.md
• Added reconcile-index endpoint (cleanup + reindex missing)
• Lightweight health check mode for uptime monitoring
• Detailed health check mode for dashboards

---

Phase 2: Volatile Memory System ✅

Architecture (Final Implementation)

┌─────────────────┐    ┌──────────────┐    ┌─────────────────┐
│  Library-Desk   │◄───│   Scheduler  │───►│ External APIs   │
│                 │    │              │    │ (weather, news) │
│  VolatileCache  │    │  Refresh     │    └─────────────────┘
│  Service        │    │  Jobs        │
└────────┬────────┘    └──────────────┘
         │
         ▼
┌─────────────────┐
│     Qdrant      │
│ (volatile_{user})│
└─────────────────┘

Key design decisions:

• Vector storage in Qdrant (not Redis) for semantic search
• Collection per user: volatile_{user}
• TTL via ttl_expiry timestamp in payload
• Natural language conversion for embedding structured data
• Integrated into HybridRAG with priority boost

Endpoints (Implemented)

Endpoint	Method	Purpose
/volatile/search?q=...	GET	Semantic search across volatile data
/volatile/store?namespace=...&key=...	POST	Store/update record
/volatile/{namespace}/{key}	GET	Retrieve specific record
/volatile/{namespace}/{key}	DELETE	Remove record
/volatile/stats	GET	Cache statistics
/volatile/scheduled	GET	Records needing refresh
/volatile/namespaces	GET	List available namespaces
/maintenance/cleanup/volatile	POST	Purge expired records

Namespaces

Namespace	Default TTL	Use Case
weather	30 min	Current conditions, forecasts
news	1 hour	Headlines, breaking news
financial	5 min	Stock prices, exchange rates
transit	5 min	Train/bus schedules, delays
traffic	10 min	Commute times, road conditions
air_quality	1 hour	Pollution, pollen counts
sports	1 min	Live scores, matches
social	10 min	Social notifications
system	1 min	Service health status
context	1 hour	Session state
custom	1 hour	User-defined data

---

Phase 3: Document Storage (Research + Implementation)

Research Scope

Evaluate FOSS self-hosted options for:

• Git repository mirroring
• PDF/document storage with metadata
• Image/video blob storage
• Full-text search capability

Constraints:

• Must be self-hosted, Docker-deployable
• Performance is priority (can wrap complexity in API)
• No cloud dependencies

Candidates to evaluate:

1. MinIO (S3-compatible object storage) + metadata in Neo4j
2. Paperless-ngx (document management with OCR)
3. SeaweedFS (distributed file system)
4. Custom: filesystem + Neo4j metadata

Category Descriptors

Wiki page structure for document collections:

# FastAPI Documentation

## Overview
[LLM-generated summary from web search about FastAPI]

## Collection Statistics
- **Documents**: 342 files
- **Last Sync**: 2025-12-24 03:30 UTC
- **Source**: github.com/tiangolo/fastapi
- **Coverage**: API reference, tutorials, deployment guides

## What's Included
[LLM summary of collection contents based on document analysis]

## Related Topics
- [[Python Web Frameworks]]
- [[REST API Design]]

Tasks

3.1 Storage Research

Deliverable: Evaluation document comparing options

3.2 Storage Service Implementation

New file: src/services/document_store_service.py (Details pending research results)

3.3 Category Descriptor Generation

File: src/services/consolidation_service.py

Add LLM-powered category descriptor generation:

1. Web search for topic overview
2. Analyze collection contents
3. Generate/update wiki page with template

---

Phase 4: LLM Tester Data Cleanup ✅

Problem

LLM testing creates accumulated cruft across the system:

• Wiki.js pages under llm-tester/ and llm_tester/ paths
• Graph nodes (Document, Entity) linked to test pages
• Vector chunks in Qdrant for test content

This data accumulates over time and clutters Wiki.js visually (no separate tenant scope for tests).

Solution

Add a maintenance endpoint to purge all LLM tester artifacts across wiki, graph, and vectors.

Tasks

4.1 Identify Test Data Patterns ✅

Patterns matched (security-restricted to test user namespace):

• users/llm-tester/*
• users/llm_tester/*

4.2 Add Cleanup Endpoint ✅

File: src/routers/maintenance.py

@router.post("/cleanup/test-data")
async def cleanup_test_data(
    dry_run: bool = Query(default=True),
    wiki: WikiJSDep = None,
    vector_service: VectorServiceDep = None,
    graph_service: GraphServiceDep = None,
    api_key: str = Depends(verify_api_key)
):
    """
    Purge LLM tester data from wiki, graph, and vectors.

    **Security**: Only deletes pages in the test user namespace:
    - users/llm-tester/*
    - users/llm_tester/*

    Use dry_run=true to preview what would be deleted.
    """

4.3 Implementation Steps ✅

1. Wiki cleanup: Delete pages via GraphQL mutation
2. Graph cleanup: Delete Document nodes using delete_page() method
3. Vector cleanup: Delete chunks using delete_page_chunks() method

4.4 Scheduler Integration ✅

Recommended schedule: Weekly (Sunday 3:00 AM)

{
  "task_name": "test_data_cleanup",
  "schedule": "0 3 * * 0",
  "endpoint": "POST /maintenance/cleanup/test-data?dry_run=false",
  "description": "Weekly cleanup of LLM test data"
}

Files to Modify

• src/routers/maintenance.py - Add cleanup endpoint
• src/services/wiki_service.py - Add bulk delete by path pattern (if needed)
• src/services/graph_service.py - May need pattern-based node deletion
• src/services/vector_service.py - Add pattern-based chunk deletion

---

Files Modified/Created

Phase 1 (Cleanup) ✅

• src/routers/maintenance.py - Timestamp tracking, cleanup endpoints
• src/services/graph_service.py - Bidirectional validation
• src/services/vector_service.py - Cross-reference checks
• LIBRARIAN_INTEGRATION.md - Scheduler config docs

Phase 2 (Volatile) ✅

• src/services/volatile_service.py - Qdrant-based volatile cache
• src/routers/volatile.py - Simplified endpoints
• src/models/volatile.py - Namespaces and models
• src/models/hybrid_rag.py - Volatile config options
• src/services/hybrid_rag_service.py - Volatile integration
• src/clients/qdrant_client.py - Expiry filter methods
• tests/test_volatile.py - 37 tests

Phase 3 (Documents)

• docs/DOCUMENT_STORAGE_RESEARCH.md - NEW
• src/services/document_store_service.py - NEW (post-research)
• src/routers/documents.py - NEW (post-research)

Phase 4 (Test Data Cleanup)

• src/routers/maintenance.py - Add cleanup endpoint
• src/services/wiki_service.py - Bulk delete by path pattern
• src/services/graph_service.py - Pattern-based node deletion
• src/services/vector_service.py - Pattern-based chunk deletion

---

Resolved Design Decisions

1. Volatile Storage: Qdrant vectors (not Redis) for semantic search capability
2. Collection Naming: volatile_{user} for per-user isolation
3. TTL Mechanism: ttl_expiry timestamp in payload, background cleanup job
4. HybridRAG Integration: Volatile as third source with RRF priority boost
5. Biographer Qdrant: Same Qdrant instance, different collection
6. Scheduler API: Has REST API for task registration

---

Future Consideration: Dedicated API Integrations

For volatile data where quality/consistency matters (weather, financial), consider:

• OpenWeatherMap API for weather (daily refresh cycle)
• Financial data API (Alpha Vantage, Yahoo Finance)
• News APIs (NewsAPI, GDELT)
• NOS.nl - Explicit source for Dutch news

This would live in a new src/clients/ module with:

• weather_client.py - Daily refresh cycle
• financial_client.py
• news_client.py - Include NOS.nl scraper/API for Dutch coverage

These provide structured, reliable data vs. SearXNG web scraping. Implementation deferred to later phase.

---

Refresh Schedules

Note: TTL should be longer than refresh interval to prevent data gaps.

Volatile Type	TTL	Refresh Cycle	Refresh Interval	Sources
Weather	86400s (24hr)	Daily	Every 24hr	OpenWeatherMap
Dutch News	28800s (8hr)	4x daily	Every 6hr	NOS.nl
Global News	28800s (8hr)	4x daily	Every 6hr	NewsAPI, GDELT
Financial	600s (10min)	On-demand	N/A	Alpha Vantage

TTL Logic:

• TTL = Refresh Interval × 1.5 (buffer for failed refreshes)
• On-demand data gets shorter TTL since it's fetched when needed