Add POST /maintenance/cleanup/test-data endpoint to purge LLM test data from wiki, graph, and vectors. Security-restricted to test user namespace only (users/llm-tester/*, users/llm_tester/*). - Supports dry_run=true (default) to preview before deleting - Cleans vectors, graph nodes, and wiki pages - Scheduler task configured for weekly cleanup (Sunday 3:00 AM) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.5
+138
-89
@@ -6,15 +6,24 @@ A three-tier memory architecture for Library Desk with intelligent orchestration
|
||||
|
||||
| Tier | Storage | Purpose | TTL |
|
||||
|------|---------|---------|-----|
|
||||
| **Volatile** | Redis | Weather, news, financial, ephemeral context | 5min - 2hr |
|
||||
| **Volatile** | Qdrant (vectors) | Weather, news, financial, ephemeral context | 5min - 2hr |
|
||||
| **Documents** | TBD (research) | Git mirrors, PDFs, video, images | Permanent |
|
||||
| **Knowledge** | Wiki + Neo4j | Personal dossiers, research, summaries | Permanent |
|
||||
|
||||
**Implementation Priority**: Cleanup → Volatile → Documents
|
||||
**Implementation Priority**: Cleanup → Volatile → Documents → Test Data Cleanup
|
||||
|
||||
### Phase Status
|
||||
|
||||
| Phase | Status | Version |
|
||||
|-------|--------|---------|
|
||||
| Phase 1: Cleanup System | ✅ Complete | v1.4.0 |
|
||||
| Phase 2: Volatile Memory | ✅ Complete | v1.4.3 |
|
||||
| Phase 3: Document Storage | ⏳ Pending | - |
|
||||
| Phase 4: Test Data Cleanup | ✅ Complete | v1.4.4 |
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Cleanup System Completion
|
||||
## Phase 1: Cleanup System Completion ✅
|
||||
|
||||
### Current State
|
||||
- **COMPLETE** - All Phase 1 tasks implemented
|
||||
@@ -57,9 +66,9 @@ A three-tier memory architecture for Library Desk with intelligent orchestration
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Volatile Memory System
|
||||
## Phase 2: Volatile Memory System ✅
|
||||
|
||||
### Architecture
|
||||
### Architecture (Final Implementation)
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌──────────────┐ ┌─────────────────┐
|
||||
@@ -71,88 +80,46 @@ A three-tier memory architecture for Library Desk with intelligent orchestration
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Redis │
|
||||
│ (DB 4, TTL) │
|
||||
│ Qdrant │
|
||||
│ (volatile_{user})│
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
### Data Model
|
||||
**Key design decisions:**
|
||||
- Vector storage in Qdrant (not Redis) for semantic search
|
||||
- Collection per user: `volatile_{user}`
|
||||
- TTL via `ttl_expiry` timestamp in payload
|
||||
- Natural language conversion for embedding structured data
|
||||
- Integrated into HybridRAG with priority boost
|
||||
|
||||
```python
|
||||
class VolatileRecord(BaseModel):
|
||||
key: str # e.g., "weather:rotterdam"
|
||||
namespace: str # e.g., "weather", "news", "financial"
|
||||
data: dict # Actual content
|
||||
source: Optional[str] # Origin API/service
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
ttl: int # Seconds until expiration
|
||||
refresh_schedule: Optional[str] # Cron expression, if repeating
|
||||
user: str # Multi-tenant isolation
|
||||
```
|
||||
|
||||
**Key pattern**: `{user}:volatile:{namespace}:{key_hash}`
|
||||
|
||||
### Implementation Order: Integration-First
|
||||
|
||||
1. **Start with Consolidation Hook** - Understand data flow through existing system
|
||||
2. **Build Service Layer** - VolatileCacheService with Redis operations
|
||||
3. **Add API Endpoints** - REST interface for volatile data
|
||||
4. **Biographer Integration** - Query user preferences for relevance
|
||||
|
||||
### Tasks
|
||||
|
||||
#### 2.1 Integrate with Consolidation (FIRST)
|
||||
**New file**: `src/services/volatile_service.py`
|
||||
|
||||
```python
|
||||
class VolatileCacheService:
|
||||
async def get(user, namespace, key) -> Optional[VolatileRecord]
|
||||
async def set(user, namespace, key, data, ttl, refresh_schedule=None)
|
||||
async def delete(user, namespace, key)
|
||||
async def list_namespace(user, namespace) -> List[str]
|
||||
async def get_scheduled(user) -> List[VolatileRecord] # For scheduler
|
||||
```
|
||||
|
||||
#### 2.2 Create Volatile API Router
|
||||
**New file**: `src/routers/volatile.py`
|
||||
### Endpoints (Implemented)
|
||||
|
||||
| Endpoint | Method | Purpose |
|
||||
|----------|--------|---------|
|
||||
| `/volatile/{namespace}/{key}` | GET | Retrieve record |
|
||||
| `/volatile/{namespace}/{key}` | POST | Store/update record |
|
||||
| `/volatile/search?q=...` | GET | Semantic search across volatile data |
|
||||
| `/volatile/store?namespace=...&key=...` | POST | Store/update record |
|
||||
| `/volatile/{namespace}/{key}` | GET | Retrieve specific record |
|
||||
| `/volatile/{namespace}/{key}` | DELETE | Remove record |
|
||||
| `/volatile/{namespace}` | GET | List keys in namespace |
|
||||
| `/volatile/scheduled` | GET | List records needing refresh |
|
||||
| `/volatile/stats` | GET | Cache statistics |
|
||||
| `/volatile/scheduled` | GET | Records needing refresh |
|
||||
| `/volatile/namespaces` | GET | List available namespaces |
|
||||
| `/maintenance/cleanup/volatile` | POST | Purge expired records |
|
||||
|
||||
#### 2.3 Integrate with Consolidation
|
||||
**File**: `src/services/consolidation_service.py`
|
||||
### Namespaces
|
||||
|
||||
Add relevance trigger detection:
|
||||
1. During consolidation, analyze search results for location/interest patterns
|
||||
2. Query tatlock's Biographer collection for user preferences
|
||||
3. If match found, create/update volatile refresh schedule
|
||||
|
||||
#### 2.4 Biographer Integration
|
||||
**File**: `src/core/dependencies.py`
|
||||
|
||||
```python
|
||||
def get_biographer_qdrant() -> QdrantClientWrapper:
|
||||
"""Direct access to tatlock's Biographer collection."""
|
||||
# Configure to connect to tatlock's Qdrant
|
||||
```
|
||||
|
||||
#### 2.5 Scheduler-Side Configuration
|
||||
Document required scheduler tasks:
|
||||
```json
|
||||
{
|
||||
"task_name": "volatile_refresh",
|
||||
"schedule": "*/15 * * * *",
|
||||
"endpoint": "GET /volatile/scheduled",
|
||||
"follow_up": "For each record, call refresh endpoint with record.refresh_schedule"
|
||||
}
|
||||
```
|
||||
| Namespace | Default TTL | Use Case |
|
||||
|-----------|-------------|----------|
|
||||
| weather | 30 min | Current conditions, forecasts |
|
||||
| news | 1 hour | Headlines, breaking news |
|
||||
| financial | 5 min | Stock prices, exchange rates |
|
||||
| transit | 5 min | Train/bus schedules, delays |
|
||||
| traffic | 10 min | Commute times, road conditions |
|
||||
| air_quality | 1 hour | Pollution, pollen counts |
|
||||
| sports | 1 min | Live scores, matches |
|
||||
| social | 10 min | Social notifications |
|
||||
| system | 1 min | Service health status |
|
||||
| context | 1 hour | Session state |
|
||||
| custom | 1 hour | User-defined data |
|
||||
|
||||
---
|
||||
|
||||
@@ -219,34 +186,116 @@ Add LLM-powered category descriptor generation:
|
||||
|
||||
---
|
||||
|
||||
## Files to Modify/Create
|
||||
## Phase 4: LLM Tester Data Cleanup ✅
|
||||
|
||||
### Phase 1 (Cleanup)
|
||||
- `src/routers/maintenance.py` - Add timestamp tracking
|
||||
### Problem
|
||||
|
||||
LLM testing creates accumulated cruft across the system:
|
||||
- Wiki.js pages under `llm-tester/` and `llm_tester/` paths
|
||||
- Graph nodes (Document, Entity) linked to test pages
|
||||
- Vector chunks in Qdrant for test content
|
||||
|
||||
This data accumulates over time and clutters Wiki.js visually (no separate tenant scope for tests).
|
||||
|
||||
### Solution
|
||||
|
||||
Add a maintenance endpoint to purge all LLM tester artifacts across wiki, graph, and vectors.
|
||||
|
||||
### Tasks
|
||||
|
||||
#### 4.1 Identify Test Data Patterns ✅
|
||||
**Patterns matched** (security-restricted to test user namespace):
|
||||
- `users/llm-tester/*`
|
||||
- `users/llm_tester/*`
|
||||
|
||||
#### 4.2 Add Cleanup Endpoint ✅
|
||||
**File**: `src/routers/maintenance.py`
|
||||
|
||||
```python
|
||||
@router.post("/cleanup/test-data")
|
||||
async def cleanup_test_data(
|
||||
dry_run: bool = Query(default=True),
|
||||
wiki: WikiJSDep = None,
|
||||
vector_service: VectorServiceDep = None,
|
||||
graph_service: GraphServiceDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Purge LLM tester data from wiki, graph, and vectors.
|
||||
|
||||
**Security**: Only deletes pages in the test user namespace:
|
||||
- users/llm-tester/*
|
||||
- users/llm_tester/*
|
||||
|
||||
Use dry_run=true to preview what would be deleted.
|
||||
"""
|
||||
```
|
||||
|
||||
#### 4.3 Implementation Steps ✅
|
||||
|
||||
1. **Wiki cleanup**: Delete pages via GraphQL mutation
|
||||
2. **Graph cleanup**: Delete Document nodes using `delete_page()` method
|
||||
3. **Vector cleanup**: Delete chunks using `delete_page_chunks()` method
|
||||
|
||||
#### 4.4 Scheduler Integration ✅
|
||||
**Recommended schedule**: Weekly (Sunday 3:00 AM)
|
||||
|
||||
```json
|
||||
{
|
||||
"task_name": "test_data_cleanup",
|
||||
"schedule": "0 3 * * 0",
|
||||
"endpoint": "POST /maintenance/cleanup/test-data?dry_run=false",
|
||||
"description": "Weekly cleanup of LLM test data"
|
||||
}
|
||||
```
|
||||
|
||||
### Files to Modify
|
||||
|
||||
- `src/routers/maintenance.py` - Add cleanup endpoint
|
||||
- `src/services/wiki_service.py` - Add bulk delete by path pattern (if needed)
|
||||
- `src/services/graph_service.py` - May need pattern-based node deletion
|
||||
- `src/services/vector_service.py` - Add pattern-based chunk deletion
|
||||
|
||||
---
|
||||
|
||||
## Files Modified/Created
|
||||
|
||||
### Phase 1 (Cleanup) ✅
|
||||
- `src/routers/maintenance.py` - Timestamp tracking, cleanup endpoints
|
||||
- `src/services/graph_service.py` - Bidirectional validation
|
||||
- `src/services/vector_service.py` - Cross-reference checks
|
||||
- `LIBRARIAN_INTEGRATION.md` - Scheduler config docs
|
||||
|
||||
### Phase 2 (Volatile)
|
||||
- `src/services/volatile_service.py` - **NEW**
|
||||
- `src/routers/volatile.py` - **NEW**
|
||||
- `src/models/volatile.py` - **NEW**
|
||||
- `src/core/dependencies.py` - Add Biographer client
|
||||
- `src/services/consolidation_service.py` - Relevance triggers
|
||||
- `tests/test_volatile.py` - **NEW**
|
||||
### Phase 2 (Volatile) ✅
|
||||
- `src/services/volatile_service.py` - Qdrant-based volatile cache
|
||||
- `src/routers/volatile.py` - Simplified endpoints
|
||||
- `src/models/volatile.py` - Namespaces and models
|
||||
- `src/models/hybrid_rag.py` - Volatile config options
|
||||
- `src/services/hybrid_rag_service.py` - Volatile integration
|
||||
- `src/clients/qdrant_client.py` - Expiry filter methods
|
||||
- `tests/test_volatile.py` - 37 tests
|
||||
|
||||
### Phase 3 (Documents)
|
||||
- `docs/DOCUMENT_STORAGE_RESEARCH.md` - **NEW**
|
||||
- `src/services/document_store_service.py` - **NEW** (post-research)
|
||||
- `src/routers/documents.py` - **NEW** (post-research)
|
||||
|
||||
### Phase 4 (Test Data Cleanup)
|
||||
- `src/routers/maintenance.py` - Add cleanup endpoint
|
||||
- `src/services/wiki_service.py` - Bulk delete by path pattern
|
||||
- `src/services/graph_service.py` - Pattern-based node deletion
|
||||
- `src/services/vector_service.py` - Pattern-based chunk deletion
|
||||
|
||||
---
|
||||
|
||||
## Resolved Design Decisions
|
||||
|
||||
1. **Biographer Qdrant**: Same Qdrant instance, different collection. Library-Desk queries directly.
|
||||
2. **Scheduler API**: Has REST API for task registration. Library-Desk can programmatically create refresh schedules.
|
||||
3. **External API calls**: Library-Desk routes through SearXNG for web search. Consider dedicated API integrations for high-value volatiles (weather, financial) for consistent quality.
|
||||
1. **Volatile Storage**: Qdrant vectors (not Redis) for semantic search capability
|
||||
2. **Collection Naming**: `volatile_{user}` for per-user isolation
|
||||
3. **TTL Mechanism**: `ttl_expiry` timestamp in payload, background cleanup job
|
||||
4. **HybridRAG Integration**: Volatile as third source with RRF priority boost
|
||||
5. **Biographer Qdrant**: Same Qdrant instance, different collection
|
||||
6. **Scheduler API**: Has REST API for task registration
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user