Compare commits
@@ -0,0 +1,23 @@
|
||||
# Service URLs for local dev (pointing to your server)
|
||||
TEST_HOST=192.168.86.149
|
||||
WIKIJS_URL=http://192.168.86.149:8088
|
||||
NEO4J_URI=bolt://192.168.86.149:7687
|
||||
QDRANT_HOST=192.168.86.149
|
||||
QDRANT_PORT=6333
|
||||
OLLAMA_URL=http://192.168.86.149:11434
|
||||
SEARXNG_URL=http://192.168.86.149:8080
|
||||
REDIS_HOST=192.168.86.149
|
||||
|
||||
OLLAMA_MODEL=mistral-nemo-large:latest
|
||||
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
|
||||
|
||||
# Wiki.js auth
|
||||
WIKIJS_USERNAME=librarian@schweitz.net
|
||||
WIKIJS_PASSWORD=key_here
|
||||
# Wiki.js GraphQL API token (generate from Admin → API Access)
|
||||
WIKI_GRAPHQL_API=your_jwt_token_here
|
||||
|
||||
LIBRARY_API_KEY=key_here
|
||||
NEO4J_PASSWORD=key_here
|
||||
WIKIJS_DB_PASSWORD=key_here
|
||||
SCHEDULER_API_KEY=key_here
|
||||
@@ -23,6 +23,47 @@
|
||||
* **Update `CHANGELOG.md`** with every user-facing change.
|
||||
* Format: `## [Unreleased] - YYYY-MM-DD` followed by `### Added`, `### Changed`, or `### Fixed`.
|
||||
|
||||
### 🚀 Release Flow
|
||||
When changes are ready for deployment:
|
||||
|
||||
1. **Ask user if deploy cycle is desired **
|
||||
|
||||
2. **Update version** in `pyproject.toml`:
|
||||
- Bug fixes: bump patch version (1.8.3 → 1.8.4)
|
||||
- New features: bump minor version (1.8.4 → 1.9.0)
|
||||
|
||||
3. **Update CHANGELOG.md**:
|
||||
- Move items from `[Unreleased]` to new version section
|
||||
- Add release date: `## [1.8.4] - 2025-12-16`
|
||||
|
||||
4. **Commit and tag**:
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "fix: description of changes"
|
||||
git tag v1.8.4
|
||||
git push origin main --tags
|
||||
```
|
||||
|
||||
5. **CI/CD triggers automatically**:
|
||||
- Gitea CI builds Docker image on new tag
|
||||
- Watchtower pulls and deploys to production
|
||||
- Verify deployment: `curl http://192.168.86.149:8000/health`
|
||||
|
||||
---
|
||||
|
||||
### 🧪 Local Development Setup
|
||||
* **Always test locally first** before committing and deploying. The build-deploy loop is slow.
|
||||
* **Start the local server** with `./wakeup.sh` - logs are written to `logs/server.log` for easy tailing
|
||||
* **Auto-reload**: The wakeup script runs uvicorn in reload mode - code changes are picked up automatically without restart (except for requirements.txt changes)
|
||||
* **Test REST endpoints** against `http://localhost:8778` using curl or similar tools
|
||||
* **Only deploy** when a phase or feature is complete and tested locally
|
||||
* **Environment**: Copy `.env.example` to `.env` and configure for your local setup (Ollama, Redis, Neo4j, Qdrant, Wiki.js hosts)
|
||||
* **Running tests**: Always use the venv explicitly to avoid environment mismatches:
|
||||
```bash
|
||||
.venv/bin/python -m pytest tests/ # All tests
|
||||
.venv/bin/python -m pytest tests/ -v # Verbose output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. FastAPI Architecture & Best Practices
|
||||
|
||||
+164
@@ -5,6 +5,170 @@ All notable changes to Library Desk will be documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [1.4.3] - 2025-12-24
|
||||
|
||||
### Changed
|
||||
|
||||
- **Volatile Cache System Refactored to Vector Storage**
|
||||
- Backend migrated from Redis to Qdrant for semantic search capability
|
||||
- Data converted to natural language for embedding and semantic retrieval
|
||||
- Collection naming: `volatile_{user}` for per-user isolation
|
||||
- TTL implemented via `ttl_expiry` timestamp in vector payload
|
||||
- Simplified endpoints:
|
||||
- `GET /volatile/search?q=...` - Semantic search across volatile data
|
||||
- `POST /volatile/store?namespace=...&key=...` - Store with query params
|
||||
- `GET /volatile/{namespace}/{key}` - Get specific record
|
||||
- `DELETE /volatile/{namespace}/{key}` - Delete record
|
||||
- Removed namespace-specific URL patterns (simpler API for LLM tool use)
|
||||
|
||||
### Added
|
||||
|
||||
- **HybridRAG Volatile Integration** - Volatile cache now included in multi-source search
|
||||
- Volatile results get priority boost in RRF fusion (current data ranks higher)
|
||||
- New config options: `enable_volatile`, `volatile_limit` (default 1), `volatile_threshold`
|
||||
- Timing breakdown includes `volatile_ms`
|
||||
- **Volatile Cleanup Endpoint** - `POST /maintenance/cleanup/volatile`
|
||||
- Purges expired records across all `volatile_*` collections
|
||||
- Scheduler task for every 10 minutes recommended
|
||||
- Returns per-collection cleanup counts
|
||||
- **Natural Language Conversion** - Structured data converted for embedding
|
||||
- Template-based conversion for each namespace (weather, news, financial, etc.)
|
||||
- Fallback for custom namespaces
|
||||
|
||||
## [1.4.2] - 2025-12-24
|
||||
|
||||
### Added
|
||||
|
||||
- **Volatile Cache System** - Ephemeral data storage with TTL
|
||||
- `GET /volatile/{namespace}/{key}` - Retrieve cached record
|
||||
- `POST /volatile/{namespace}/{key}` - Store/update record with TTL
|
||||
- `DELETE /volatile/{namespace}/{key}` - Remove record
|
||||
- `GET /volatile/{namespace}` - List keys in namespace
|
||||
- `DELETE /volatile/{namespace}` - Clear all records in namespace
|
||||
- `GET /volatile/stats` - Cache statistics by namespace
|
||||
- `GET /volatile/scheduled` - Records needing refresh (for scheduler)
|
||||
- `GET /volatile/namespaces` - List available namespaces with default TTLs
|
||||
- **Volatile Namespaces** - Predefined categories with appropriate TTLs:
|
||||
- `weather` (30min) - Weather conditions and forecasts
|
||||
- `news` (1hr) - Headlines and breaking news
|
||||
- `financial` (5min) - Stock prices, exchange rates
|
||||
- `transit` (5min) - Train/bus schedules, delays
|
||||
- `traffic` (10min) - Commute times, road conditions
|
||||
- `air_quality` (1hr) - Pollution, pollen counts
|
||||
- `sports` (1min) - Live scores, matches
|
||||
- `social` (10min) - Social notifications
|
||||
- `system` (1min) - Service health status
|
||||
- `context` (1hr) - Session state
|
||||
- `custom` (1hr) - User-defined data
|
||||
- **Refresh Schedule Support** - Optional cron expressions for scheduler integration
|
||||
|
||||
## [1.4.1] - 2025-12-24
|
||||
|
||||
### Fixed
|
||||
|
||||
- Wiki.js API token now optional - GraphQL API works without authentication
|
||||
- Container startup failure when `WIKI_GRAPHQL_API` env var not set
|
||||
|
||||
## [1.4.0] - 2025-12-24
|
||||
|
||||
### Added
|
||||
|
||||
- **Maintenance Router** - New `/maintenance` endpoints for system health and cleanup
|
||||
- `GET /maintenance/health` - Lightweight health check (detailed mode available)
|
||||
- `POST /maintenance/cleanup/all` - Full orphan cleanup (vectors + graph)
|
||||
- `POST /maintenance/cleanup/vectors` - Purge orphan vector chunks
|
||||
- `POST /maintenance/cleanup/graph` - Purge orphan graph nodes
|
||||
- `POST /maintenance/reconcile-index` - Combined cleanup + reindex missing pages
|
||||
- **Bidirectional Orphan Detection** - Cross-validate vectors and graph nodes
|
||||
- `find_documents_without_vectors()` - Graph nodes missing vector chunks
|
||||
- `find_chunks_without_graph_nodes()` - Vector chunks missing graph nodes
|
||||
- **Qdrant Client Methods** - Bulk operations for maintenance
|
||||
- `scroll_all_points()` - Iterate all points with pagination
|
||||
- `delete_by_ids()` - Batch delete by point IDs
|
||||
- **Graph Service Cleanup** - Node deletion methods
|
||||
- `delete_document_node()` - Remove document and relationships
|
||||
- `delete_collection_node()` - Remove collection and contained documents
|
||||
- `get_all_document_references()` - Get all document references for validation
|
||||
- **Redis Timestamp Tracking** - `last_cleanup` timestamp for scheduler integration
|
||||
- **Memory System Plan** - Documented three-tier architecture (volatile/documents/knowledge)
|
||||
|
||||
### Changed
|
||||
|
||||
- **Wiki.js Authentication** - Switched from username/password to API token
|
||||
- New `WIKI_GRAPHQL_API` environment variable for JWT token
|
||||
- Deprecated `WIKIJS_USERNAME` and `WIKIJS_PASSWORD` (kept for backwards compatibility)
|
||||
- **Service Dependencies** - Added `VectorServiceDep` and `GraphServiceDep` type aliases
|
||||
|
||||
### Fixed
|
||||
|
||||
- Wiki.js client now properly handles API token auth without login flow
|
||||
|
||||
## [1.3.3] - 2025-12-23
|
||||
|
||||
### Added
|
||||
|
||||
- Temperature parameter to `OllamaClient.generate_text()` for controlling output determinism
|
||||
- `TODO.md` tracking remaining stub endpoints to implement
|
||||
- Wired `/query/semantic` endpoint to VectorService
|
||||
- Wired `/query/graph` endpoint to GraphService
|
||||
|
||||
### Changed
|
||||
|
||||
- **Improved LLM prompts** based on llm-findings.md recommendations:
|
||||
- Keyword extraction: temperature 0.0, negative constraints
|
||||
- LLM re-ranking: temperature 0.0, explicit rules
|
||||
- Conflict detection: temperature 0.0, analysis steps (CoT)
|
||||
- Wiki page creation: temperature 0.3, anti-hallucination constraints
|
||||
- Page reconstruction: temperature 0.2, preservation constraints
|
||||
- Web results analysis: temperature 0.0, conservative approach
|
||||
- Test fixtures now use configurable host (TEST_HOST) instead of Docker hostnames
|
||||
|
||||
### Removed
|
||||
|
||||
- Dead code: unused `get_default_user()` function
|
||||
- Unused imports from routers (wiki.py, graph.py, hybrid_rag.py)
|
||||
- Stub endpoints shadowed by real implementations (/stats, /ingest/document, /ingest/batch)
|
||||
|
||||
## [1.3.2] - 2025-12-22
|
||||
|
||||
### Changed
|
||||
|
||||
- **Consolidated Ollama model configuration** - All LLM operations now use single `OLLAMA_MODEL` environment variable
|
||||
- Removed separate `reranker_model` setting
|
||||
- HybridRAG re-ranking, consolidation analysis, and wiki page writing all use the same model
|
||||
- Improves VRAM efficiency by keeping one model hot
|
||||
- Added `OLLAMA_EMBEDDING_MODEL` environment variable for embedding model (previously overloaded `OLLAMA_MODEL`)
|
||||
- Updated WikiPageWriter to accept settings instead of hardcoded model name
|
||||
|
||||
## [1.3.1] - 2025-12-16
|
||||
|
||||
### Fixed
|
||||
|
||||
- Smart create endpoint missing `content_extractor` dependency causing 500 errors on `POST /wiki/pages/smart-create`
|
||||
|
||||
## [1.3.0] - 2025-12-15
|
||||
|
||||
### Changed
|
||||
|
||||
- **Two-Stage RRF Architecture** - Major refactor to level the playing field between wiki and web results
|
||||
- Stage 1: Vector and graph results merged into single "wiki" ranking using mini-RRF
|
||||
- Stage 2: Final RRF between wiki (single source) and web (single source)
|
||||
- Wiki pages no longer get 2x advantage from appearing in both vector and graph searches
|
||||
- Multi-source confirmation still determines wiki internal ranking
|
||||
|
||||
- **Skip synonyms in graph search** - LLM-generated synonyms (e.g., "author") no longer match unrelated graph entities (e.g., "author2000")
|
||||
- Vector search still uses synonyms for semantic similarity
|
||||
- Graph search uses only core keywords for exact entity matching
|
||||
|
||||
### Added
|
||||
|
||||
- `VECTOR_SIMILARITY_THRESHOLD` config setting (default: 0.7) to filter weak vector matches
|
||||
- Deduplication in graph search to prevent same document appearing multiple times
|
||||
|
||||
### Fixed
|
||||
|
||||
- Graph search duplicate entity bug where same document could appear twice if entity linked multiple times
|
||||
|
||||
## [1.2.1] - 2025-12-15
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
# Claude Code Instructions
|
||||
|
||||
**MANDATORY: Read AGENTS.md instead of this file.**
|
||||
|
||||
This project uses a unified configuration file for all LLM coding agents.
|
||||
|
||||
## Instructions
|
||||
|
||||
1. **Read and follow AGENTS.md** - All project guidelines are located there
|
||||
2. **Do not modify this file** - Only update AGENTS.md
|
||||
3. **Do not create or modify other agent-specific files** - Use AGENTS.md as the single source of truth
|
||||
|
||||
This approach ensures consistent behavior across all LLM coding agents without managing separate configuration files.
|
||||
|
||||
---
|
||||
|
||||
If you need to update project guidelines, edit AGENTS.md, not this file.
|
||||
@@ -455,6 +455,154 @@ LIBRARY_BATCH_SIZE=50
|
||||
LIBRARY_SYNC_ENABLED=true
|
||||
```
|
||||
|
||||
## Maintenance Tasks
|
||||
|
||||
### Index Reconciliation (Daily)
|
||||
|
||||
The `reconcile-index` endpoint performs full index maintenance:
|
||||
|
||||
1. **Cleanup Phase**: Remove orphaned data
|
||||
- Vector chunks without wiki source
|
||||
- Graph nodes without vectors (bidirectional)
|
||||
- Vectors without graph nodes (bidirectional)
|
||||
- Orphan entities (no MENTIONS relationships)
|
||||
- Broken relationships
|
||||
|
||||
2. **Reindex Phase**: Index missing pages
|
||||
- Wiki pages without vector embeddings
|
||||
- Wiki pages without graph Document nodes
|
||||
|
||||
**Scheduler Task: `library_reconcile_index`**
|
||||
|
||||
```yaml
|
||||
Task Name: library_reconcile_index
|
||||
Description: Daily index reconciliation - cleanup orphans + reindex missing pages
|
||||
Schedule: Daily at 04:00 (after library_sync at 03:30)
|
||||
Priority: 10 (system maintenance)
|
||||
Service: library
|
||||
Executor: POST /maintenance/reconcile-index
|
||||
Configuration:
|
||||
- LIBRARY_DESK_URL: http://library-desk:8089
|
||||
- LIBRARY_API_KEY: ${LIBRARY_API_KEY}
|
||||
Parameters:
|
||||
- user: jpmschweitzer
|
||||
- dry_run: false
|
||||
Outputs:
|
||||
- Vector orphans purged
|
||||
- Entity orphans purged
|
||||
- Missing pages reindexed
|
||||
```
|
||||
|
||||
### Maintenance Endpoints
|
||||
|
||||
| Endpoint | Method | Purpose |
|
||||
|----------|--------|---------|
|
||||
| `/maintenance/reconcile-index` | POST | **Recommended**: Full cleanup + reindex missing |
|
||||
| `/maintenance/cleanup/all` | POST | Cleanup only (orphan removal) |
|
||||
| `/maintenance/cleanup/vectors` | POST | Clean orphan vector chunks only |
|
||||
| `/maintenance/cleanup/graph` | POST | Clean orphan entities & stale docs only |
|
||||
| `/maintenance/health` | GET | Lightweight health check (for uptime monitoring) |
|
||||
| `/maintenance/health?detailed=true` | GET | Full analysis with orphan counts |
|
||||
| `/maintenance/reindex/{page_id}` | POST | Force re-index a specific page |
|
||||
|
||||
### Health Check Modes
|
||||
|
||||
**Lightweight (default)** - Use for frequent uptime checks (every 30s):
|
||||
```bash
|
||||
curl "http://library-desk:8089/maintenance/health?user=jpmschweitzer" \
|
||||
-H "Authorization: Bearer ${LIBRARY_API_KEY}"
|
||||
```
|
||||
|
||||
Returns only last cleanup timestamp and basic status (no database queries).
|
||||
|
||||
**Detailed** - Use for dashboards or before reconciliation:
|
||||
```bash
|
||||
curl "http://library-desk:8089/maintenance/health?user=jpmschweitzer&detailed=true" \
|
||||
-H "Authorization: Bearer ${LIBRARY_API_KEY}"
|
||||
```
|
||||
|
||||
Returns full orphan analysis (runs database queries).
|
||||
|
||||
### Example Reconcile Request
|
||||
|
||||
```bash
|
||||
curl -X POST "http://library-desk:8089/maintenance/reconcile-index?user=jpmschweitzer" \
|
||||
-H "Authorization: Bearer ${LIBRARY_API_KEY}"
|
||||
```
|
||||
|
||||
### Example Response
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"cleanup": {
|
||||
"success": true,
|
||||
"vector_cleanup": {
|
||||
"wiki_chunks": {"orphans_found": 5, "orphans_purged": 5},
|
||||
"document_chunks": {"orphans_found": 0, "orphans_purged": 0},
|
||||
"chunks_without_graph": {"orphans_found": 2, "orphans_purged": 2},
|
||||
"total_chunks_scanned": 1250,
|
||||
"total_orphans_purged": 7
|
||||
},
|
||||
"graph_cleanup": {
|
||||
"orphan_entities": {"orphans_found": 3, "orphans_purged": 3},
|
||||
"stale_wiki_documents": {"orphans_found": 1, "orphans_purged": 1},
|
||||
"stale_store_documents": {"orphans_found": 0, "orphans_purged": 0},
|
||||
"docs_without_vectors": {"orphans_found": 0, "orphans_purged": 0},
|
||||
"broken_relationships_cleaned": 0
|
||||
},
|
||||
"total_duration_ms": 1523.5
|
||||
},
|
||||
"reindex_missing": {
|
||||
"pages_without_vectors": 2,
|
||||
"pages_without_graph": 1,
|
||||
"pages_reindexed": 2,
|
||||
"pages_failed": 0,
|
||||
"failed_page_ids": [],
|
||||
"duration_ms": 3421.2
|
||||
},
|
||||
"total_duration_ms": 4944.7
|
||||
}
|
||||
```
|
||||
|
||||
### Scheduler Integration Code
|
||||
|
||||
```python
|
||||
# scheduler/src/tasks/library_maintenance.py
|
||||
|
||||
async def library_reconcile_index_task(user: str = "jpmschweitzer"):
|
||||
"""Run daily Library Desk index reconciliation."""
|
||||
|
||||
async with httpx.AsyncClient() as client:
|
||||
# Run reconcile-index (cleanup + reindex missing)
|
||||
result = await client.post(
|
||||
f"{LIBRARY_DESK_URL}/maintenance/reconcile-index",
|
||||
params={"user": user, "dry_run": False},
|
||||
headers={"Authorization": f"Bearer {LIBRARY_API_KEY}"},
|
||||
timeout=600.0 # 10 minutes for large indexes
|
||||
)
|
||||
|
||||
data = result.json()
|
||||
|
||||
# Log summary
|
||||
cleanup = data["cleanup"]
|
||||
reindex = data["reindex_missing"]
|
||||
|
||||
logger.info(
|
||||
f"Reconcile complete: "
|
||||
f"{cleanup['vector_cleanup']['total_orphans_purged']} vector orphans, "
|
||||
f"{cleanup['graph_cleanup']['orphan_entities']['orphans_purged']} entity orphans, "
|
||||
f"{reindex['pages_reindexed']} pages reindexed"
|
||||
)
|
||||
|
||||
if reindex["pages_failed"] > 0:
|
||||
logger.warning(f"Failed to reindex pages: {reindex['failed_page_ids']}")
|
||||
|
||||
return data
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Implement ingestion endpoints in Library Desk
|
||||
|
||||
@@ -49,7 +49,8 @@ QDRANT_PORT=6333
|
||||
WIKIJS_URL=http://wiki:3000
|
||||
SEARXNG_URL=http://searxng:8080
|
||||
OLLAMA_URL=http://ollama:11434
|
||||
OLLAMA_MODEL=nomic-embed-text
|
||||
OLLAMA_MODEL=mistral-nemo-large:latest
|
||||
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
|
||||
REDIS_HOST=redis-shared
|
||||
REDIS_PORT=6379
|
||||
REDIS_DB=2
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
# TODO
|
||||
|
||||
Outstanding work items for Library Desk.
|
||||
|
||||
## Stub Endpoints to Implement
|
||||
|
||||
The following endpoints in `src/main.py` return stub responses and need real implementations:
|
||||
|
||||
### Ingestion Status Endpoints
|
||||
|
||||
#### `POST /ingest/check-updates`
|
||||
Check which documents need updating based on content hashes. Used by Scheduler to determine what changed since last sync.
|
||||
|
||||
**Implementation needed:**
|
||||
1. Query existing documents by path
|
||||
2. Compare content hashes
|
||||
3. Return list of updates needed
|
||||
|
||||
#### `GET /ingest/status/{document_id}`
|
||||
Get processing status for a document.
|
||||
|
||||
**Implementation needed:**
|
||||
- Status tracking system (Redis or database)
|
||||
- Track ingestion progress per document
|
||||
|
||||
#### `GET /ingest/repo-status/{repository}`
|
||||
Get indexing status for an entire repository.
|
||||
|
||||
**Implementation needed:**
|
||||
- Repository-level statistics
|
||||
- Track which documents from a repo are indexed
|
||||
|
||||
### Deduplication
|
||||
|
||||
#### `POST /deduplicate/check`
|
||||
Check for duplicate or highly similar documents using vector similarity and graph analysis.
|
||||
|
||||
**Implementation needed:**
|
||||
1. Get document embedding from Qdrant
|
||||
2. Find similar vectors above threshold
|
||||
3. Check graph relationships
|
||||
4. Return candidates with similarity scores
|
||||
|
||||
## System Statistics
|
||||
|
||||
#### `GET /stats`
|
||||
Get system statistics (wiki pages, neo4j nodes, qdrant vectors).
|
||||
|
||||
**Implementation needed:**
|
||||
- Query Neo4j for node count
|
||||
- Query Qdrant for vector count
|
||||
- Query Wiki.js for page count
|
||||
@@ -0,0 +1,58 @@
|
||||
# HybridRAG Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
HybridRAG combines three search sources to provide comprehensive results:
|
||||
- **Vector search** (Qdrant) - Semantic similarity via embeddings
|
||||
- **Graph search** (Neo4j) - Entity relationships in knowledge graph
|
||||
- **Web search** (SearXNG) - External web results via Trafilatura extraction
|
||||
|
||||
## Two-Stage RRF Fusion (v1.3.0+)
|
||||
|
||||
To ensure fair ranking between wiki and web results, we use a two-stage Reciprocal Rank Fusion:
|
||||
|
||||
```
|
||||
Stage 1: Wiki Merge
|
||||
vector results ─┬─→ Mini-RRF ─→ Unified wiki ranking
|
||||
graph results ─┘
|
||||
|
||||
Stage 2: Final RRF
|
||||
wiki (merged) ─┬─→ Final RRF ─→ Combined results
|
||||
web results ─┘
|
||||
```
|
||||
|
||||
**Why two stages?**
|
||||
|
||||
Previously, wiki pages found by BOTH vector and graph received double RRF contribution, giving them an unfair 2x advantage over web results. The two-stage approach:
|
||||
1. Merges vector+graph into a single "wiki" source
|
||||
2. Wiki's internal ranking still benefits from multi-source confirmation
|
||||
3. Wiki and web compete as equals in final ranking
|
||||
|
||||
## Configuration
|
||||
|
||||
| Setting | Default | Description |
|
||||
|---------|---------|-------------|
|
||||
| `VECTOR_SIMILARITY_THRESHOLD` | 0.7 | Minimum similarity score for vector results |
|
||||
| `HYBRID_RAG_VECTOR_LIMIT` | 10 | Max vector results |
|
||||
| `HYBRID_RAG_GRAPH_LIMIT` | 10 | Max graph results |
|
||||
| `HYBRID_RAG_WEB_LIMIT` | 5 | Max web results |
|
||||
|
||||
## Known Limitations & Future Improvements
|
||||
|
||||
### Vector Search Noise
|
||||
|
||||
**Status:** Open for improvement if needed after observation period.
|
||||
|
||||
Vector search may return generic category/index pages (e.g., "Reference", "Projects", "Places") with high similarity scores (~0.86). These pages often have similar boilerplate content leading to uniform scores.
|
||||
|
||||
**Potential solutions if this becomes problematic:**
|
||||
1. **Raise threshold** - Increase `VECTOR_SIMILARITY_THRESHOLD` to 0.85+
|
||||
2. **Page-type filtering** - Exclude pages tagged as category/index/stub
|
||||
3. **Content length signal** - Penalize pages with minimal content
|
||||
4. **Duplicate score detection** - Flag results with suspiciously identical scores
|
||||
|
||||
The LLM re-ranking phase typically demotes these low-quality results, so this may not require immediate action.
|
||||
|
||||
### Graph Search
|
||||
|
||||
Graph search uses only core keywords (no LLM-generated synonyms) to avoid false matches like "author" → "author2000". This is intentional - vector search handles semantic similarity via embeddings.
|
||||
@@ -0,0 +1,283 @@
|
||||
# Memory Management System - Implementation Plan
|
||||
|
||||
## Overview
|
||||
|
||||
A three-tier memory architecture for Library Desk with intelligent orchestration:
|
||||
|
||||
| Tier | Storage | Purpose | TTL |
|
||||
|------|---------|---------|-----|
|
||||
| **Volatile** | Redis | Weather, news, financial, ephemeral context | 5min - 2hr |
|
||||
| **Documents** | TBD (research) | Git mirrors, PDFs, video, images | Permanent |
|
||||
| **Knowledge** | Wiki + Neo4j | Personal dossiers, research, summaries | Permanent |
|
||||
|
||||
**Implementation Priority**: Cleanup → Volatile → Documents
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Cleanup System Completion
|
||||
|
||||
### Current State
|
||||
- **COMPLETE** - All Phase 1 tasks implemented
|
||||
- Redis timestamp tracking for last cleanup
|
||||
- Bidirectional orphan detection between vectors and graph
|
||||
- Scheduler integration endpoints ready
|
||||
|
||||
### Tasks
|
||||
|
||||
#### 1.1 Add Scheduler Integration Points ✅
|
||||
**Files**: `src/routers/maintenance.py`
|
||||
|
||||
- [x] Add `last_cleanup` timestamp tracking in Redis
|
||||
- [x] Return cleanup stats in format scheduler can log
|
||||
- [x] Added `RedisDep` to cleanup endpoints
|
||||
|
||||
#### 1.2 Bidirectional Orphan Detection ✅
|
||||
**Files**: `src/services/graph_service.py`, `src/services/vector_service.py`
|
||||
|
||||
- [x] `find_documents_without_vectors()` - graph nodes with no vectors
|
||||
- [x] `find_chunks_without_graph_nodes()` - vectors with no graph node
|
||||
- [x] Updated maintenance endpoints to use bidirectional checks
|
||||
- [x] Added `chunks_without_graph` and `docs_without_vectors` to response models
|
||||
|
||||
#### 1.3 Scheduler Configuration ✅
|
||||
**Scheduler-side task definition:**
|
||||
```json
|
||||
{
|
||||
"task_name": "library_reconcile_index",
|
||||
"schedule": "0 4 * * *",
|
||||
"endpoint": "POST /maintenance/reconcile-index?user=jpmschweitzer",
|
||||
"description": "Daily index reconciliation - cleanup + reindex missing"
|
||||
}
|
||||
```
|
||||
|
||||
- [x] Documented in `LIBRARIAN_INTEGRATION.md`
|
||||
- [x] Added `reconcile-index` endpoint (cleanup + reindex missing)
|
||||
- [x] Lightweight health check mode for uptime monitoring
|
||||
- [x] Detailed health check mode for dashboards
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Volatile Memory System
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌──────────────┐ ┌─────────────────┐
|
||||
│ Library-Desk │◄───│ Scheduler │───►│ External APIs │
|
||||
│ │ │ │ │ (weather, news) │
|
||||
│ VolatileCache │ │ Refresh │ └─────────────────┘
|
||||
│ Service │ │ Jobs │
|
||||
└────────┬────────┘ └──────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Redis │
|
||||
│ (DB 4, TTL) │
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
### Data Model
|
||||
|
||||
```python
|
||||
class VolatileRecord(BaseModel):
|
||||
key: str # e.g., "weather:rotterdam"
|
||||
namespace: str # e.g., "weather", "news", "financial"
|
||||
data: dict # Actual content
|
||||
source: Optional[str] # Origin API/service
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
ttl: int # Seconds until expiration
|
||||
refresh_schedule: Optional[str] # Cron expression, if repeating
|
||||
user: str # Multi-tenant isolation
|
||||
```
|
||||
|
||||
**Key pattern**: `{user}:volatile:{namespace}:{key_hash}`
|
||||
|
||||
### Implementation Order: Integration-First
|
||||
|
||||
1. **Start with Consolidation Hook** - Understand data flow through existing system
|
||||
2. **Build Service Layer** - VolatileCacheService with Redis operations
|
||||
3. **Add API Endpoints** - REST interface for volatile data
|
||||
4. **Biographer Integration** - Query user preferences for relevance
|
||||
|
||||
### Tasks
|
||||
|
||||
#### 2.1 Integrate with Consolidation (FIRST)
|
||||
**New file**: `src/services/volatile_service.py`
|
||||
|
||||
```python
|
||||
class VolatileCacheService:
|
||||
async def get(user, namespace, key) -> Optional[VolatileRecord]
|
||||
async def set(user, namespace, key, data, ttl, refresh_schedule=None)
|
||||
async def delete(user, namespace, key)
|
||||
async def list_namespace(user, namespace) -> List[str]
|
||||
async def get_scheduled(user) -> List[VolatileRecord] # For scheduler
|
||||
```
|
||||
|
||||
#### 2.2 Create Volatile API Router
|
||||
**New file**: `src/routers/volatile.py`
|
||||
|
||||
| Endpoint | Method | Purpose |
|
||||
|----------|--------|---------|
|
||||
| `/volatile/{namespace}/{key}` | GET | Retrieve record |
|
||||
| `/volatile/{namespace}/{key}` | POST | Store/update record |
|
||||
| `/volatile/{namespace}/{key}` | DELETE | Remove record |
|
||||
| `/volatile/{namespace}` | GET | List keys in namespace |
|
||||
| `/volatile/scheduled` | GET | List records needing refresh |
|
||||
| `/volatile/stats` | GET | Cache statistics |
|
||||
|
||||
#### 2.3 Integrate with Consolidation
|
||||
**File**: `src/services/consolidation_service.py`
|
||||
|
||||
Add relevance trigger detection:
|
||||
1. During consolidation, analyze search results for location/interest patterns
|
||||
2. Query tatlock's Biographer collection for user preferences
|
||||
3. If match found, create/update volatile refresh schedule
|
||||
|
||||
#### 2.4 Biographer Integration
|
||||
**File**: `src/core/dependencies.py`
|
||||
|
||||
```python
|
||||
def get_biographer_qdrant() -> QdrantClientWrapper:
|
||||
"""Direct access to tatlock's Biographer collection."""
|
||||
# Configure to connect to tatlock's Qdrant
|
||||
```
|
||||
|
||||
#### 2.5 Scheduler-Side Configuration
|
||||
Document required scheduler tasks:
|
||||
```json
|
||||
{
|
||||
"task_name": "volatile_refresh",
|
||||
"schedule": "*/15 * * * *",
|
||||
"endpoint": "GET /volatile/scheduled",
|
||||
"follow_up": "For each record, call refresh endpoint with record.refresh_schedule"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Document Storage (Research + Implementation)
|
||||
|
||||
### Research Scope
|
||||
|
||||
Evaluate FOSS self-hosted options for:
|
||||
- Git repository mirroring
|
||||
- PDF/document storage with metadata
|
||||
- Image/video blob storage
|
||||
- Full-text search capability
|
||||
|
||||
**Constraints**:
|
||||
- Must be self-hosted, Docker-deployable
|
||||
- Performance is priority (can wrap complexity in API)
|
||||
- No cloud dependencies
|
||||
|
||||
**Candidates to evaluate**:
|
||||
1. MinIO (S3-compatible object storage) + metadata in Neo4j
|
||||
2. Paperless-ngx (document management with OCR)
|
||||
3. SeaweedFS (distributed file system)
|
||||
4. Custom: filesystem + Neo4j metadata
|
||||
|
||||
### Category Descriptors
|
||||
|
||||
**Wiki page structure for document collections**:
|
||||
```markdown
|
||||
# FastAPI Documentation
|
||||
|
||||
## Overview
|
||||
[LLM-generated summary from web search about FastAPI]
|
||||
|
||||
## Collection Statistics
|
||||
- **Documents**: 342 files
|
||||
- **Last Sync**: 2025-12-24 03:30 UTC
|
||||
- **Source**: github.com/tiangolo/fastapi
|
||||
- **Coverage**: API reference, tutorials, deployment guides
|
||||
|
||||
## What's Included
|
||||
[LLM summary of collection contents based on document analysis]
|
||||
|
||||
## Related Topics
|
||||
- [[Python Web Frameworks]]
|
||||
- [[REST API Design]]
|
||||
```
|
||||
|
||||
### Tasks
|
||||
|
||||
#### 3.1 Storage Research
|
||||
**Deliverable**: Evaluation document comparing options
|
||||
|
||||
#### 3.2 Storage Service Implementation
|
||||
**New file**: `src/services/document_store_service.py`
|
||||
(Details pending research results)
|
||||
|
||||
#### 3.3 Category Descriptor Generation
|
||||
**File**: `src/services/consolidation_service.py`
|
||||
|
||||
Add LLM-powered category descriptor generation:
|
||||
1. Web search for topic overview
|
||||
2. Analyze collection contents
|
||||
3. Generate/update wiki page with template
|
||||
|
||||
---
|
||||
|
||||
## Files to Modify/Create
|
||||
|
||||
### Phase 1 (Cleanup)
|
||||
- `src/routers/maintenance.py` - Add timestamp tracking
|
||||
- `src/services/graph_service.py` - Bidirectional validation
|
||||
- `src/services/vector_service.py` - Cross-reference checks
|
||||
- `LIBRARIAN_INTEGRATION.md` - Scheduler config docs
|
||||
|
||||
### Phase 2 (Volatile)
|
||||
- `src/services/volatile_service.py` - **NEW**
|
||||
- `src/routers/volatile.py` - **NEW**
|
||||
- `src/models/volatile.py` - **NEW**
|
||||
- `src/core/dependencies.py` - Add Biographer client
|
||||
- `src/services/consolidation_service.py` - Relevance triggers
|
||||
- `tests/test_volatile.py` - **NEW**
|
||||
|
||||
### Phase 3 (Documents)
|
||||
- `docs/DOCUMENT_STORAGE_RESEARCH.md` - **NEW**
|
||||
- `src/services/document_store_service.py` - **NEW** (post-research)
|
||||
- `src/routers/documents.py` - **NEW** (post-research)
|
||||
|
||||
---
|
||||
|
||||
## Resolved Design Decisions
|
||||
|
||||
1. **Biographer Qdrant**: Same Qdrant instance, different collection. Library-Desk queries directly.
|
||||
2. **Scheduler API**: Has REST API for task registration. Library-Desk can programmatically create refresh schedules.
|
||||
3. **External API calls**: Library-Desk routes through SearXNG for web search. Consider dedicated API integrations for high-value volatiles (weather, financial) for consistent quality.
|
||||
|
||||
---
|
||||
|
||||
## Future Consideration: Dedicated API Integrations
|
||||
|
||||
For volatile data where quality/consistency matters (weather, financial), consider:
|
||||
- OpenWeatherMap API for weather (daily refresh cycle)
|
||||
- Financial data API (Alpha Vantage, Yahoo Finance)
|
||||
- News APIs (NewsAPI, GDELT)
|
||||
- **NOS.nl** - Explicit source for Dutch news
|
||||
|
||||
This would live in a new `src/clients/` module with:
|
||||
- `weather_client.py` - Daily refresh cycle
|
||||
- `financial_client.py`
|
||||
- `news_client.py` - Include NOS.nl scraper/API for Dutch coverage
|
||||
|
||||
These provide structured, reliable data vs. SearXNG web scraping. Implementation deferred to later phase.
|
||||
|
||||
---
|
||||
|
||||
## Refresh Schedules
|
||||
|
||||
**Note:** TTL should be longer than refresh interval to prevent data gaps.
|
||||
|
||||
| Volatile Type | TTL | Refresh Cycle | Refresh Interval | Sources |
|
||||
|---------------|-----|---------------|------------------|---------|
|
||||
| Weather | 86400s (24hr) | Daily | Every 24hr | OpenWeatherMap |
|
||||
| Dutch News | 28800s (8hr) | 4x daily | Every 6hr | NOS.nl |
|
||||
| Global News | 28800s (8hr) | 4x daily | Every 6hr | NewsAPI, GDELT |
|
||||
| Financial | 600s (10min) | On-demand | N/A | Alpha Vantage |
|
||||
|
||||
**TTL Logic:**
|
||||
- TTL = Refresh Interval × 1.5 (buffer for failed refreshes)
|
||||
- On-demand data gets shorter TTL since it's fetched when needed
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "library-desk"
|
||||
version = "1.2.1"
|
||||
version = "1.4.3"
|
||||
description = "Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and knowledge consolidation"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
|
||||
@@ -249,7 +249,8 @@ class OllamaClient:
|
||||
self,
|
||||
prompt: str,
|
||||
model: Optional[str] = None,
|
||||
stream: bool = False
|
||||
stream: bool = False,
|
||||
temperature: Optional[float] = None
|
||||
) -> Optional[str]:
|
||||
"""
|
||||
Generate text completion (for non-embedding use cases).
|
||||
@@ -258,12 +259,15 @@ class OllamaClient:
|
||||
prompt: Input prompt
|
||||
model: Model name (defaults to self.model)
|
||||
stream: Enable streaming response
|
||||
temperature: Sampling temperature (0.0 = deterministic, higher = more creative)
|
||||
None uses model default (~0.7 for mistral-nemo)
|
||||
|
||||
Returns:
|
||||
Generated text or None on failure
|
||||
|
||||
Note: This is primarily for debugging/testing. Use specialized
|
||||
LLM services for production text generation.
|
||||
Note: Use temperature=0.0 for deterministic outputs like JSON parsing,
|
||||
ranking, and factual extraction. Use higher values (0.3-0.7) for
|
||||
creative content generation.
|
||||
"""
|
||||
try:
|
||||
payload = {
|
||||
@@ -272,6 +276,10 @@ class OllamaClient:
|
||||
"stream": stream
|
||||
}
|
||||
|
||||
# Add temperature to options if specified
|
||||
if temperature is not None:
|
||||
payload["options"] = {"temperature": temperature}
|
||||
|
||||
response = await self.client.post(
|
||||
self.generate_url,
|
||||
json=payload
|
||||
|
||||
@@ -11,7 +11,7 @@ Provides async vector operations with:
|
||||
from qdrant_client import QdrantClient
|
||||
from qdrant_client.models import (
|
||||
Distance, VectorParams, PointStruct,
|
||||
Filter, FieldCondition, MatchValue
|
||||
Filter, FieldCondition, MatchValue, Range
|
||||
)
|
||||
from typing import List, Dict, Any, Optional
|
||||
import uuid
|
||||
@@ -551,6 +551,97 @@ class QdrantClientWrapper:
|
||||
logger.error(f"Search failed: {e}", exc_info=True)
|
||||
return []
|
||||
|
||||
async def scroll_all_points(
|
||||
self,
|
||||
collection_name: str,
|
||||
batch_size: int = 100,
|
||||
with_payload: bool = True,
|
||||
with_vectors: bool = False,
|
||||
filter_conditions: Optional[Dict[str, Any]] = None
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Scroll through all points in a collection.
|
||||
|
||||
Args:
|
||||
collection_name: Collection name
|
||||
batch_size: Number of points per batch
|
||||
with_payload: Include payload in results
|
||||
with_vectors: Include vectors in results
|
||||
filter_conditions: Optional filter conditions
|
||||
|
||||
Returns:
|
||||
List of all points with id and payload
|
||||
"""
|
||||
all_points = []
|
||||
offset = None
|
||||
|
||||
# Build filter if provided
|
||||
scroll_filter = None
|
||||
if filter_conditions:
|
||||
conditions = []
|
||||
for key, value in filter_conditions.items():
|
||||
conditions.append(
|
||||
FieldCondition(key=key, match=MatchValue(value=value))
|
||||
)
|
||||
scroll_filter = Filter(must=conditions)
|
||||
|
||||
try:
|
||||
while True:
|
||||
points, next_offset = self.client.scroll(
|
||||
collection_name=collection_name,
|
||||
scroll_filter=scroll_filter,
|
||||
limit=batch_size,
|
||||
offset=offset,
|
||||
with_payload=with_payload,
|
||||
with_vectors=with_vectors
|
||||
)
|
||||
|
||||
for point in points:
|
||||
all_points.append({
|
||||
"id": str(point.id),
|
||||
"payload": dict(point.payload) if point.payload else {}
|
||||
})
|
||||
|
||||
if next_offset is None:
|
||||
break
|
||||
offset = next_offset
|
||||
|
||||
return all_points
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to scroll collection {collection_name}: {e}", exc_info=True)
|
||||
return []
|
||||
|
||||
async def delete_by_ids(
|
||||
self,
|
||||
collection_name: str,
|
||||
point_ids: List[str]
|
||||
) -> int:
|
||||
"""
|
||||
Delete points by their IDs.
|
||||
|
||||
Args:
|
||||
collection_name: Collection name
|
||||
point_ids: List of point IDs to delete
|
||||
|
||||
Returns:
|
||||
Number of points deleted
|
||||
"""
|
||||
if not point_ids:
|
||||
return 0
|
||||
|
||||
try:
|
||||
self.client.delete(
|
||||
collection_name=collection_name,
|
||||
points_selector=point_ids
|
||||
)
|
||||
logger.info(f"Deleted {len(point_ids)} points from {collection_name}")
|
||||
return len(point_ids)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to delete points by IDs: {e}", exc_info=True)
|
||||
return 0
|
||||
|
||||
async def list_collections(self) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
List all collections with stats.
|
||||
@@ -585,4 +676,134 @@ class QdrantClientWrapper:
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to list collections: {e}", exc_info=True)
|
||||
return []
|
||||
|
||||
# ========== Volatile Data Methods ==========
|
||||
|
||||
async def search_with_expiry_filter(
|
||||
self,
|
||||
collection_name: str,
|
||||
query_vector: List[float],
|
||||
current_timestamp: int,
|
||||
limit: int = 10,
|
||||
score_threshold: float = 0.7
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Search vectors filtering out expired records.
|
||||
|
||||
Args:
|
||||
collection_name: Collection name
|
||||
query_vector: Query embedding vector
|
||||
current_timestamp: Current time in milliseconds
|
||||
limit: Maximum results
|
||||
score_threshold: Minimum similarity score
|
||||
|
||||
Returns:
|
||||
List of non-expired search results
|
||||
"""
|
||||
# Filter: ttl_expiry > current_timestamp (not expired)
|
||||
expiry_filter = Filter(
|
||||
must=[
|
||||
FieldCondition(
|
||||
key="ttl_expiry",
|
||||
range=Range(gt=current_timestamp)
|
||||
)
|
||||
]
|
||||
)
|
||||
|
||||
try:
|
||||
response = self.client.query_points(
|
||||
collection_name=collection_name,
|
||||
query=query_vector,
|
||||
limit=limit,
|
||||
score_threshold=score_threshold,
|
||||
query_filter=expiry_filter,
|
||||
with_payload=True
|
||||
)
|
||||
|
||||
return [
|
||||
{
|
||||
"id": str(point.id),
|
||||
"score": point.score,
|
||||
"payload": dict(point.payload)
|
||||
}
|
||||
for point in response.points
|
||||
]
|
||||
except Exception as e:
|
||||
logger.error(f"Volatile search failed: {e}", exc_info=True)
|
||||
return []
|
||||
|
||||
async def delete_expired_vectors(
|
||||
self,
|
||||
collection_name: str,
|
||||
current_timestamp: int
|
||||
) -> int:
|
||||
"""
|
||||
Delete all vectors where ttl_expiry < current_timestamp.
|
||||
|
||||
Args:
|
||||
collection_name: Collection name
|
||||
current_timestamp: Current time in milliseconds
|
||||
|
||||
Returns:
|
||||
Number of points deleted (approximate)
|
||||
"""
|
||||
# Filter: ttl_expiry < current_timestamp (expired)
|
||||
expiry_filter = Filter(
|
||||
must=[
|
||||
FieldCondition(
|
||||
key="ttl_expiry",
|
||||
range=Range(lt=current_timestamp)
|
||||
)
|
||||
]
|
||||
)
|
||||
|
||||
try:
|
||||
# First count how many will be deleted (scroll to count)
|
||||
count = 0
|
||||
offset = None
|
||||
while True:
|
||||
points, next_offset = self.client.scroll(
|
||||
collection_name=collection_name,
|
||||
scroll_filter=expiry_filter,
|
||||
limit=100,
|
||||
offset=offset,
|
||||
with_payload=False
|
||||
)
|
||||
count += len(points)
|
||||
if next_offset is None:
|
||||
break
|
||||
offset = next_offset
|
||||
|
||||
if count == 0:
|
||||
return 0
|
||||
|
||||
# Delete expired points
|
||||
self.client.delete(
|
||||
collection_name=collection_name,
|
||||
points_selector=expiry_filter
|
||||
)
|
||||
|
||||
logger.info(f"Deleted {count} expired vectors from {collection_name}")
|
||||
return count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to delete expired vectors: {e}", exc_info=True)
|
||||
return 0
|
||||
|
||||
async def get_volatile_collections(self) -> List[str]:
|
||||
"""
|
||||
Get all volatile collections (prefixed with 'volatile_').
|
||||
|
||||
Returns:
|
||||
List of volatile collection names
|
||||
"""
|
||||
try:
|
||||
collections = self.client.get_collections()
|
||||
return [
|
||||
c.name for c in collections.collections
|
||||
if c.name.startswith("volatile_")
|
||||
]
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to list volatile collections: {e}", exc_info=True)
|
||||
return []
|
||||
@@ -20,96 +20,34 @@ class WikiJSClient:
|
||||
Wiki.js GraphQL API client.
|
||||
|
||||
Documentation: https://docs.requarks.io/dev/api
|
||||
Authentication: Username/password login to get user-specific JWT token
|
||||
Authentication: API token (JWT) generated from Wiki.js admin panel
|
||||
"""
|
||||
|
||||
def __init__(self, base_url: str, username: str, password: str):
|
||||
def __init__(self, base_url: str, api_token: str):
|
||||
"""
|
||||
Initialize Wiki.js client.
|
||||
|
||||
Args:
|
||||
base_url: Wiki.js base URL (e.g., "http://wiki:3000")
|
||||
username: Wiki.js username (e.g., "librarian@schweitz.net")
|
||||
password: Wiki.js password
|
||||
api_token: Wiki.js API token (JWT from admin panel)
|
||||
"""
|
||||
self.base_url = base_url.rstrip("/")
|
||||
self.graphql_url = f"{self.base_url}/graphql"
|
||||
self.username = username
|
||||
self.password = password
|
||||
self.jwt_token: Optional[str] = None
|
||||
self.api_token = api_token
|
||||
self.client = httpx.AsyncClient(timeout=30.0)
|
||||
logger.info(f"Initialized Wiki.js client: {base_url} (user: {username})")
|
||||
auth_mode = "with API token" if api_token else "without auth (open API)"
|
||||
logger.info(f"Initialized Wiki.js client: {base_url} ({auth_mode})")
|
||||
|
||||
async def close(self):
|
||||
"""Close HTTP client"""
|
||||
await self.client.aclose()
|
||||
|
||||
async def login(self) -> bool:
|
||||
"""
|
||||
Authenticate with Wiki.js using username/password.
|
||||
|
||||
Returns:
|
||||
True if login successful, False otherwise
|
||||
"""
|
||||
login_mutation = """
|
||||
mutation Login($username: String!, $password: String!, $strategy: String!) {
|
||||
authentication {
|
||||
login(username: $username, password: $password, strategy: $strategy) {
|
||||
responseResult {
|
||||
succeeded
|
||||
errorCode
|
||||
message
|
||||
}
|
||||
jwt
|
||||
}
|
||||
}
|
||||
}
|
||||
"""
|
||||
|
||||
variables = {
|
||||
"username": self.username,
|
||||
"password": self.password,
|
||||
"strategy": "local"
|
||||
}
|
||||
|
||||
try:
|
||||
response = await self.client.post(
|
||||
self.graphql_url,
|
||||
headers={"Content-Type": "application/json"},
|
||||
json={"query": login_mutation, "variables": variables}
|
||||
)
|
||||
response.raise_for_status()
|
||||
result = response.json()
|
||||
|
||||
if "errors" in result:
|
||||
logger.error(f"Login failed: {result['errors']}")
|
||||
return False
|
||||
|
||||
login_result = result.get("data", {}).get("authentication", {}).get("login", {})
|
||||
response_result = login_result.get("responseResult", {})
|
||||
|
||||
if not response_result.get("succeeded"):
|
||||
logger.error(f"Login failed: {response_result.get('message')}")
|
||||
return False
|
||||
|
||||
self.jwt_token = login_result.get("jwt")
|
||||
if not self.jwt_token:
|
||||
logger.error("Login succeeded but no JWT token received")
|
||||
return False
|
||||
|
||||
logger.info(f"Successfully authenticated as {self.username}")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Login failed: {e}", exc_info=True)
|
||||
return False
|
||||
|
||||
async def _ensure_authenticated(self):
|
||||
"""Ensure we have a valid JWT token, login if needed."""
|
||||
if not self.jwt_token:
|
||||
success = await self.login()
|
||||
if not success:
|
||||
raise Exception("Failed to authenticate with Wiki.js")
|
||||
def _get_headers(self) -> Dict[str, str]:
|
||||
"""Get request headers, optionally including auth token."""
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if self.api_token:
|
||||
headers["Authorization"] = f"Bearer {self.api_token}"
|
||||
return headers
|
||||
|
||||
async def _execute_query(
|
||||
self,
|
||||
@@ -129,18 +67,12 @@ class WikiJSClient:
|
||||
Raises:
|
||||
Exception: If query fails or returns errors
|
||||
"""
|
||||
# Ensure we're authenticated before making requests
|
||||
await self._ensure_authenticated()
|
||||
|
||||
payload = {
|
||||
"query": query,
|
||||
"variables": variables or {}
|
||||
}
|
||||
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.jwt_token}",
|
||||
"Content-Type": "application/json"
|
||||
}
|
||||
headers = self._get_headers()
|
||||
|
||||
try:
|
||||
response = await self.client.post(
|
||||
|
||||
+23
-5
@@ -43,8 +43,10 @@ class Settings(BaseSettings):
|
||||
|
||||
# Wiki.js Configuration
|
||||
wikijs_url: str = Field(default="http://wiki:3000", description="Wiki.js URL")
|
||||
wikijs_username: str = Field(..., description="Wiki.js username")
|
||||
wikijs_password: str = Field(..., description="Wiki.js password")
|
||||
wiki_graphql_api: str = Field(default="", description="Wiki.js GraphQL API token (optional - API may be open)")
|
||||
# Legacy auth fields - kept for backwards compatibility but deprecated
|
||||
wikijs_username: str = Field(default="", description="Wiki.js username (deprecated, use wiki_graphql_api)")
|
||||
wikijs_password: str = Field(default="", description="Wiki.js password (deprecated, use wiki_graphql_api)")
|
||||
|
||||
# Wiki.js Database Configuration (for change listener)
|
||||
wikijs_db_host: str = Field(default="postgres-shared", description="Wiki.js PostgreSQL host")
|
||||
@@ -62,16 +64,17 @@ class Settings(BaseSettings):
|
||||
# SearXNG Configuration
|
||||
searxng_url: str = Field(default="http://searxng:8080", description="SearXNG URL")
|
||||
|
||||
# Ollama Configuration (for embeddings)
|
||||
# Ollama Configuration
|
||||
ollama_url: str = Field(default="http://ollama:11434", description="Ollama URL")
|
||||
ollama_model: str = Field(default="nomic-embed-text", description="Ollama embedding model")
|
||||
ollama_model: str = Field(default="mistral-nemo-large:latest", description="Ollama LLM model")
|
||||
ollama_embedding_model: str = Field(default="nomic-embed-text", description="Ollama embedding model")
|
||||
|
||||
# HybridRAG Configuration
|
||||
reranker_model: str = Field(default="mistral-nemo", description="Model for LLM re-ranking")
|
||||
reranker_enabled: bool = Field(default=True, description="Enable LLM re-ranking")
|
||||
hybrid_rag_vector_limit: int = Field(default=10, ge=1, le=50, description="Vector search limit")
|
||||
hybrid_rag_graph_limit: int = Field(default=10, ge=1, le=50, description="Graph search limit")
|
||||
hybrid_rag_web_limit: int = Field(default=5, ge=1, le=20, description="Web search limit")
|
||||
vector_similarity_threshold: float = Field(default=0.7, ge=0.0, le=1.0, description="Minimum similarity score for vector results")
|
||||
|
||||
# Entity Linking Fuzzy Matching Configuration
|
||||
entity_linking_min_confidence: float = Field(default=0.70, ge=0.0, le=1.0, description="Minimum confidence for entity-document matching")
|
||||
@@ -98,6 +101,21 @@ class Settings(BaseSettings):
|
||||
content_extraction_timeout: int = Field(default=5, ge=1, le=30, description="Trafilatura per-URL timeout in seconds")
|
||||
content_max_length: int = Field(default=2000, ge=500, le=10000, description="Max extracted content length per result")
|
||||
|
||||
# Document Store Configuration
|
||||
document_store_enabled: bool = Field(default=True, description="Enable document store feature")
|
||||
document_catalog_path_prefix: str = Field(default="docs", description="Wiki path prefix for catalog pages")
|
||||
|
||||
# Volatile Cache Configuration
|
||||
volatile_cache_enabled: bool = Field(default=True, description="Enable volatile cache feature")
|
||||
volatile_default_ttl: int = Field(default=3600, ge=60, le=86400, description="Default TTL in seconds")
|
||||
volatile_weather_ttl: int = Field(default=1800, ge=60, le=7200, description="Weather data TTL in seconds")
|
||||
volatile_news_ttl: int = Field(default=7200, ge=300, le=86400, description="News data TTL in seconds")
|
||||
volatile_financial_ttl: int = Field(default=300, ge=60, le=3600, description="Financial data TTL in seconds")
|
||||
|
||||
# Maintenance Configuration
|
||||
maintenance_orphan_cleanup_enabled: bool = Field(default=True, description="Enable automatic orphan cleanup")
|
||||
maintenance_cleanup_batch_size: int = Field(default=100, ge=10, le=1000, description="Cleanup batch size")
|
||||
|
||||
@property
|
||||
def qdrant_url(self) -> str:
|
||||
"""Computed Qdrant URL."""
|
||||
|
||||
+12
-16
@@ -76,13 +76,12 @@ def get_wikijs_client() -> WikiJSClient:
|
||||
Get Wiki.js client singleton.
|
||||
|
||||
Returns:
|
||||
Initialized Wiki.js GraphQL client with username/password auth
|
||||
Initialized Wiki.js GraphQL client with API token auth
|
||||
"""
|
||||
settings = get_settings()
|
||||
client = WikiJSClient(
|
||||
base_url=settings.wikijs_url,
|
||||
username=settings.wikijs_username,
|
||||
password=settings.wikijs_password
|
||||
api_token=settings.wiki_graphql_api
|
||||
)
|
||||
logger.debug("Created Wiki.js client instance")
|
||||
return client
|
||||
@@ -113,7 +112,7 @@ def get_ollama_client() -> OllamaClient:
|
||||
settings = get_settings()
|
||||
client = OllamaClient(
|
||||
base_url=settings.ollama_url,
|
||||
model=settings.ollama_model
|
||||
model=settings.ollama_embedding_model
|
||||
)
|
||||
logger.debug("Created Ollama client instance")
|
||||
return client
|
||||
@@ -395,18 +394,6 @@ def get_rag_search_service() -> "RAGSearchService":
|
||||
)
|
||||
|
||||
|
||||
# Utility: Get default user from settings or multi_tenancy
|
||||
def get_default_user() -> str:
|
||||
"""
|
||||
Get default user for operations.
|
||||
|
||||
Returns:
|
||||
Default user identifier
|
||||
"""
|
||||
from src.core.multi_tenancy import DEFAULT_USER
|
||||
return DEFAULT_USER
|
||||
|
||||
|
||||
# Authentication
|
||||
from fastapi import Security, HTTPException
|
||||
from fastapi.security import HTTPBearer
|
||||
@@ -437,3 +424,12 @@ async def verify_api_key(
|
||||
detail="Invalid API key"
|
||||
)
|
||||
return credentials.credentials
|
||||
|
||||
|
||||
# Service type aliases for FastAPI endpoint dependencies
|
||||
# These are defined after the factory functions
|
||||
from src.services.vector_service import VectorService
|
||||
from src.services.graph_service import GraphService
|
||||
|
||||
VectorServiceDep = Annotated[VectorService, Depends(get_vector_service)]
|
||||
GraphServiceDep = Annotated[GraphService, Depends(get_graph_service)]
|
||||
|
||||
+73
-101
@@ -8,7 +8,7 @@ Following best practices:
|
||||
- OpenAPI documentation
|
||||
"""
|
||||
|
||||
from fastapi import FastAPI, HTTPException, Depends
|
||||
from fastapi import FastAPI, HTTPException, Depends, Query
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from pydantic import BaseModel
|
||||
@@ -17,7 +17,10 @@ import logging
|
||||
from pathlib import Path
|
||||
|
||||
from src.config import Settings, get_settings, __version__
|
||||
from src.core.dependencies import verify_api_key
|
||||
from src.core.dependencies import (
|
||||
verify_api_key, QdrantDep, WikiJSDep, OllamaDep, Neo4jDep
|
||||
)
|
||||
from src.core.multi_tenancy import DEFAULT_USER
|
||||
|
||||
# Configure logging
|
||||
logging.basicConfig(
|
||||
@@ -47,7 +50,8 @@ app.add_middleware(
|
||||
# Register routers
|
||||
from src.routers import (
|
||||
wiki, tools, graph, vector, hybrid_rag, consolidation,
|
||||
ingestion, entity_linking, webhooks, rag_search, content
|
||||
ingestion, entity_linking, webhooks, rag_search, content,
|
||||
maintenance, volatile
|
||||
)
|
||||
|
||||
app.include_router(wiki.router)
|
||||
@@ -61,6 +65,8 @@ app.include_router(entity_linking.router)
|
||||
app.include_router(webhooks.router)
|
||||
app.include_router(rag_search.router)
|
||||
app.include_router(content.router)
|
||||
app.include_router(maintenance.router)
|
||||
app.include_router(volatile.router)
|
||||
|
||||
# Mount static files directory for Wiki.js integration scripts
|
||||
static_dir = Path(__file__).parent.parent / "static"
|
||||
@@ -78,13 +84,6 @@ class HealthResponse(BaseModel):
|
||||
services: Dict[str, Any]
|
||||
|
||||
|
||||
class StatsResponse(BaseModel):
|
||||
"""Statistics response model."""
|
||||
wiki_pages: int
|
||||
neo4j_nodes: int
|
||||
qdrant_vectors: int
|
||||
|
||||
|
||||
# Routes
|
||||
@app.get("/", tags=["Root"])
|
||||
async def root() -> Dict[str, str]:
|
||||
@@ -141,77 +140,6 @@ async def health(settings: Settings = Depends(get_settings)) -> HealthResponse:
|
||||
)
|
||||
|
||||
|
||||
@app.get("/stats", response_model=StatsResponse, tags=["System"])
|
||||
async def stats(
|
||||
api_key: str = Depends(verify_api_key)
|
||||
) -> StatsResponse:
|
||||
"""
|
||||
Get system statistics.
|
||||
Protected endpoint - requires API key.
|
||||
|
||||
TODO: Implement actual stats gathering from:
|
||||
- Neo4j (node count)
|
||||
- Qdrant (vector count)
|
||||
- Wiki.js (page count)
|
||||
"""
|
||||
return StatsResponse(
|
||||
wiki_pages=0,
|
||||
neo4j_nodes=0,
|
||||
qdrant_vectors=0
|
||||
)
|
||||
|
||||
|
||||
# Ingestion endpoints (for Scheduler integration)
|
||||
@app.post("/ingest/document", tags=["Ingestion"])
|
||||
async def ingest_document(
|
||||
document: Dict[str, Any],
|
||||
api_key: str = Depends(verify_api_key)
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Ingest a single document for indexing.
|
||||
Used by The Scheduler to add mirrored documentation to the knowledge base.
|
||||
|
||||
Expected fields:
|
||||
- source: str (e.g., "github", "gitea")
|
||||
- repository: str (e.g., "anthropic-cookbook")
|
||||
- path: str (file path)
|
||||
- content: str (document content)
|
||||
- metadata: dict (commit, author, tags, etc.)
|
||||
|
||||
TODO: Implement document ingestion pipeline:
|
||||
1. Chunk content
|
||||
2. Generate embeddings (Ollama)
|
||||
3. Extract entities (NLP)
|
||||
4. Index in Qdrant
|
||||
5. Create graph nodes/relationships in Neo4j
|
||||
"""
|
||||
return {
|
||||
"message": "Document ingestion not yet implemented",
|
||||
"document_id": f"doc_{document.get('path', 'unknown')}",
|
||||
"status": "stub"
|
||||
}
|
||||
|
||||
|
||||
@app.post("/ingest/batch", tags=["Ingestion"])
|
||||
async def batch_ingest(
|
||||
batch: Dict[str, Any],
|
||||
api_key: str = Depends(verify_api_key)
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Ingest multiple documents in a batch.
|
||||
More efficient than individual ingestion for large syncs.
|
||||
|
||||
TODO: Implement batch processing with task queue
|
||||
"""
|
||||
document_count = len(batch.get("documents", []))
|
||||
return {
|
||||
"message": "Batch ingestion not yet implemented",
|
||||
"batch_id": "batch_stub",
|
||||
"total_documents": document_count,
|
||||
"status": "stub"
|
||||
}
|
||||
|
||||
|
||||
@app.post("/ingest/check-updates", tags=["Ingestion"])
|
||||
async def check_updates(
|
||||
documents: Dict[str, Any],
|
||||
@@ -269,41 +197,85 @@ async def get_repo_status(
|
||||
}
|
||||
|
||||
|
||||
# Query endpoints (stubs for future implementation)
|
||||
# NOTE: /query/hybrid is now implemented in routers/hybrid_rag.py
|
||||
# Query endpoints
|
||||
# NOTE: /query/hybrid is implemented in routers/hybrid_rag.py
|
||||
|
||||
@app.post("/query/semantic", tags=["Query"])
|
||||
async def semantic_query(
|
||||
query: Dict[str, Any],
|
||||
query: str = Query(..., min_length=1, description="Search query text"),
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
limit: int = Query(default=10, ge=1, le=100, description="Maximum results"),
|
||||
score_threshold: float = Query(default=0.5, ge=0.0, le=1.0, description="Minimum similarity score"),
|
||||
qdrant_client: QdrantDep = None,
|
||||
wiki_client: WikiJSDep = None,
|
||||
ollama_client: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
) -> Dict[str, Any]:
|
||||
):
|
||||
"""
|
||||
Semantic search via Qdrant.
|
||||
Pure vector similarity search.
|
||||
Semantic search via Qdrant vector similarity.
|
||||
|
||||
TODO: Implement semantic search
|
||||
Searches document chunks using embedding similarity. Returns matching
|
||||
chunks with relevance scores, page titles, and paths.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
POST /query/semantic?query=docker%20configuration&user=jpmschweitzer&limit=10
|
||||
```
|
||||
|
||||
**Returns:** List of matching chunks with similarity scores (0-1)
|
||||
"""
|
||||
return {
|
||||
"message": "Semantic search not yet implemented",
|
||||
"query": query
|
||||
}
|
||||
from src.services.vector_service import VectorService
|
||||
|
||||
vector_service = VectorService(qdrant_client, wiki_client, ollama_client)
|
||||
try:
|
||||
return await vector_service.search(
|
||||
query=query,
|
||||
user=user,
|
||||
limit=limit,
|
||||
score_threshold=score_threshold
|
||||
)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.error(f"Semantic search failed: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail="Search failed")
|
||||
|
||||
|
||||
@app.post("/query/graph", tags=["Query"])
|
||||
async def graph_query(
|
||||
query: Dict[str, Any],
|
||||
query: str = Query(..., description="Cypher query to execute"),
|
||||
user: str = Query(default=DEFAULT_USER, description="User for scoping (auto-filters results)"),
|
||||
neo4j_client: Neo4jDep = None,
|
||||
wiki_client: WikiJSDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
) -> Dict[str, Any]:
|
||||
):
|
||||
"""
|
||||
Graph traversal via Neo4j.
|
||||
Execute Cypher queries.
|
||||
Execute a Cypher query against the Neo4j knowledge graph.
|
||||
|
||||
TODO: Implement graph queries
|
||||
Queries are automatically scoped to the user's data for security.
|
||||
Use this for custom graph traversals beyond what /graph/nodes provides.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
POST /query/graph?query=MATCH%20(d:Document)-[:MENTIONS]->(p:Person)%20RETURN%20d,p&user=jpmschweitzer
|
||||
```
|
||||
|
||||
**Security:** All queries are user-scoped to prevent cross-user data access.
|
||||
"""
|
||||
return {
|
||||
"message": "Graph query not yet implemented",
|
||||
"query": query
|
||||
}
|
||||
from src.services.graph_service import GraphService
|
||||
|
||||
graph_service = GraphService(neo4j_client, wiki_client)
|
||||
try:
|
||||
return await graph_service.execute_query(
|
||||
query=query,
|
||||
parameters={},
|
||||
user=user
|
||||
)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.error(f"Graph query failed: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail="Query execution failed")
|
||||
|
||||
|
||||
# Deduplication endpoints
|
||||
|
||||
@@ -14,13 +14,16 @@ class HybridRAGConfig(BaseModel):
|
||||
vector_limit: int = Field(default=10, ge=1, le=50, description="Max vector results")
|
||||
graph_limit: int = Field(default=10, ge=1, le=50, description="Max graph results")
|
||||
web_limit: int = Field(default=5, ge=1, le=20, description="Max web results")
|
||||
volatile_limit: int = Field(default=1, ge=1, le=5, description="Max volatile results (typically 1)")
|
||||
enable_vector: bool = Field(default=True, description="Enable vector search")
|
||||
enable_graph: bool = Field(default=True, description="Enable graph search")
|
||||
enable_web: bool = Field(default=True, description="Enable web search")
|
||||
enable_volatile: bool = Field(default=True, description="Enable volatile cache search")
|
||||
enable_reranking: bool = Field(default=True, description="Enable LLM re-ranking")
|
||||
enable_enrichment: bool = Field(default=True, description="Enable graph enrichment")
|
||||
final_result_count: int = Field(default=10, ge=1, le=50, description="Final results to return")
|
||||
rrf_k: int = Field(default=60, ge=1, le=100, description="RRF constant")
|
||||
volatile_threshold: float = Field(default=0.8, ge=0.5, le=1.0, description="Volatile similarity threshold")
|
||||
|
||||
|
||||
class RelatedDossier(BaseModel):
|
||||
@@ -34,7 +37,7 @@ class RelatedDossier(BaseModel):
|
||||
|
||||
class HybridRAGResult(BaseModel):
|
||||
"""Single result from HybridRAG query."""
|
||||
source_type: str = Field(..., description="Source: 'vector', 'graph', 'web'")
|
||||
source_type: str = Field(..., description="Source: 'wiki', 'web', 'volatile'")
|
||||
title: str
|
||||
content: str
|
||||
url: Optional[str] = Field(None, description="URL for web results")
|
||||
@@ -53,6 +56,7 @@ class TimingBreakdown(BaseModel):
|
||||
vector_ms: float = Field(..., description="Phase 1: Vector search")
|
||||
graph_ms: float = Field(..., description="Phase 1: Graph search")
|
||||
web_ms: float = Field(..., description="Phase 1: Web search")
|
||||
volatile_ms: float = Field(default=0, description="Phase 1: Volatile cache search")
|
||||
fusion_ms: float = Field(..., description="Phase 2: RRF fusion")
|
||||
enrichment_ms: float = Field(..., description="Phase 3: Graph enrichment")
|
||||
reranking_ms: float = Field(..., description="Phase 4: LLM re-ranking")
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
"""
|
||||
Volatile memory models for Library Desk.
|
||||
|
||||
Provides models for ephemeral cached data with TTL - weather, news, financial data,
|
||||
transit schedules, and other time-sensitive external information.
|
||||
"""
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
from typing import Dict, Any, Optional, List
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
|
||||
|
||||
class VolatileNamespace(str, Enum):
|
||||
"""
|
||||
Predefined namespaces for volatile data.
|
||||
|
||||
Each namespace can have different default TTLs and refresh schedules.
|
||||
"""
|
||||
# Real-time external data
|
||||
WEATHER = "weather" # Current conditions, forecasts
|
||||
NEWS = "news" # Headlines, breaking news
|
||||
FINANCIAL = "financial" # Stock prices, exchange rates, crypto
|
||||
TRANSIT = "transit" # Train/bus schedules, delays, disruptions
|
||||
TRAFFIC = "traffic" # Commute times, road conditions
|
||||
AIR_QUALITY = "air_quality" # Pollution levels, pollen counts
|
||||
SPORTS = "sports" # Live scores, upcoming matches
|
||||
|
||||
# System/integration data
|
||||
SOCIAL = "social" # Social media mentions, notifications
|
||||
SYSTEM = "system" # Service health, infrastructure status
|
||||
|
||||
# Ephemeral context
|
||||
CONTEXT = "context" # Conversation context, session state
|
||||
CUSTOM = "custom" # User-defined volatile data
|
||||
|
||||
|
||||
# Default TTLs per namespace (in seconds)
|
||||
NAMESPACE_DEFAULT_TTL: Dict[str, int] = {
|
||||
VolatileNamespace.WEATHER: 1800, # 30 min - weather changes slowly
|
||||
VolatileNamespace.NEWS: 3600, # 1 hour - news cycles
|
||||
VolatileNamespace.FINANCIAL: 300, # 5 min - markets move fast
|
||||
VolatileNamespace.TRANSIT: 300, # 5 min - schedules update frequently
|
||||
VolatileNamespace.TRAFFIC: 600, # 10 min - traffic patterns
|
||||
VolatileNamespace.AIR_QUALITY: 3600, # 1 hour - air quality stable
|
||||
VolatileNamespace.SPORTS: 60, # 1 min - live scores
|
||||
VolatileNamespace.SOCIAL: 600, # 10 min - social notifications
|
||||
VolatileNamespace.SYSTEM: 60, # 1 min - system health
|
||||
VolatileNamespace.CONTEXT: 3600, # 1 hour - session context
|
||||
VolatileNamespace.CUSTOM: 3600, # 1 hour - default for custom
|
||||
}
|
||||
|
||||
|
||||
class VolatileRecord(BaseModel):
|
||||
"""
|
||||
A volatile cache record with TTL.
|
||||
|
||||
Volatile records are ephemeral data stored in Redis with automatic expiration.
|
||||
Used for weather, news, financial data, and other time-sensitive information.
|
||||
"""
|
||||
key: str = Field(..., description="Record key (e.g., 'rotterdam', 'nos-headlines')")
|
||||
namespace: str = Field(..., description="Namespace (e.g., 'weather', 'news', 'financial')")
|
||||
data: Dict[str, Any] = Field(..., description="Actual content/payload")
|
||||
source: Optional[str] = Field(None, description="Origin API/service (e.g., 'openweathermap', 'nos.nl')")
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow, description="When record was created")
|
||||
updated_at: datetime = Field(default_factory=datetime.utcnow, description="When record was last updated")
|
||||
ttl: int = Field(..., ge=60, le=604800, description="Time-to-live in seconds (max 7 days)")
|
||||
refresh_schedule: Optional[str] = Field(None, description="Cron expression for scheduled refresh")
|
||||
user: str = Field(..., description="User identifier for multi-tenancy")
|
||||
|
||||
|
||||
class VolatileRecordCreate(BaseModel):
|
||||
"""Request model for creating/updating a volatile record."""
|
||||
data: Dict[str, Any] = Field(..., description="Content to store")
|
||||
source: Optional[str] = Field(None, description="Origin API/service")
|
||||
ttl: Optional[int] = Field(None, ge=60, le=604800, description="TTL in seconds (uses namespace default if not set)")
|
||||
refresh_schedule: Optional[str] = Field(None, description="Cron expression for scheduled refresh")
|
||||
|
||||
|
||||
class VolatileRecordResponse(BaseModel):
|
||||
"""Response model for a volatile record."""
|
||||
key: str = Field(..., description="Record key")
|
||||
namespace: str = Field(..., description="Namespace")
|
||||
data: Dict[str, Any] = Field(..., description="Stored content")
|
||||
source: Optional[str] = Field(None, description="Origin API/service")
|
||||
created_at: datetime = Field(..., description="Creation timestamp")
|
||||
updated_at: datetime = Field(..., description="Last update timestamp")
|
||||
ttl: int = Field(..., description="TTL in seconds")
|
||||
ttl_remaining: int = Field(..., description="Seconds until expiration")
|
||||
refresh_schedule: Optional[str] = Field(None, description="Cron expression if scheduled")
|
||||
user: str = Field(..., description="User identifier")
|
||||
|
||||
|
||||
class VolatileListResponse(BaseModel):
|
||||
"""Response model for listing volatile records."""
|
||||
namespace: str = Field(..., description="Namespace queried")
|
||||
keys: List[str] = Field(..., description="List of keys in namespace")
|
||||
count: int = Field(..., description="Number of keys")
|
||||
user: str = Field(..., description="User identifier")
|
||||
|
||||
|
||||
class VolatileScheduledResponse(BaseModel):
|
||||
"""Response model for records needing refresh."""
|
||||
records: List[VolatileRecordResponse] = Field(..., description="Records with refresh schedules")
|
||||
count: int = Field(..., description="Number of scheduled records")
|
||||
user: str = Field(..., description="User identifier")
|
||||
|
||||
|
||||
class VolatileStatsResponse(BaseModel):
|
||||
"""Response model for volatile cache statistics."""
|
||||
total_records: int = Field(..., description="Total volatile records for user")
|
||||
by_namespace: Dict[str, int] = Field(..., description="Record count per namespace")
|
||||
scheduled_count: int = Field(..., description="Records with refresh schedules")
|
||||
total_memory_bytes: Optional[int] = Field(None, description="Approximate memory usage")
|
||||
user: str = Field(..., description="User identifier")
|
||||
|
||||
|
||||
class VolatileDeleteResponse(BaseModel):
|
||||
"""Response model for delete operation."""
|
||||
key: str = Field(..., description="Deleted key")
|
||||
namespace: str = Field(..., description="Namespace")
|
||||
deleted: bool = Field(..., description="Whether record was found and deleted")
|
||||
user: str = Field(..., description="User identifier")
|
||||
|
||||
|
||||
class VolatileBulkDeleteResponse(BaseModel):
|
||||
"""Response model for bulk delete operations."""
|
||||
namespace: Optional[str] = Field(None, description="Namespace if namespace-wide delete")
|
||||
deleted_count: int = Field(..., description="Number of records deleted")
|
||||
user: str = Field(..., description="User identifier")
|
||||
@@ -15,8 +15,6 @@ from src.models.graph import (
|
||||
MindMapResponse
|
||||
)
|
||||
from src.services.graph_service import GraphService
|
||||
from src.clients.neo4j_client import Neo4jClient
|
||||
from src.clients.wikijs_client import WikiJSClient
|
||||
from src.core.dependencies import Neo4jDep, WikiJSDep, verify_api_key
|
||||
from src.core.multi_tenancy import DEFAULT_USER
|
||||
|
||||
|
||||
+13
-12
@@ -1,7 +1,7 @@
|
||||
"""
|
||||
HybridRAG router for multi-source search API.
|
||||
|
||||
Provides endpoint for combining vector, graph, and web search
|
||||
Provides endpoint for combining vector, graph, volatile cache, and web search
|
||||
with RRF fusion and LLM re-ranking.
|
||||
"""
|
||||
|
||||
@@ -10,10 +10,6 @@ import logging
|
||||
|
||||
from src.models.hybrid_rag import HybridRAGRequest, HybridRAGResponse
|
||||
from src.services.hybrid_rag_service import HybridRAGService
|
||||
from src.services.vector_service import VectorService
|
||||
from src.services.graph_service import GraphService
|
||||
from src.clients.searxng_client import SearXNGClient
|
||||
from src.clients.ollama_client import OllamaClient
|
||||
from src.core.dependencies import (
|
||||
Neo4jDep, WikiJSDep, QdrantDep, OllamaDep,
|
||||
SearXNGDep, ContentExtractorDep, verify_api_key, get_settings
|
||||
@@ -38,10 +34,12 @@ def get_hybrid_rag_service(
|
||||
"""Get HybridRAG service instance with all dependencies."""
|
||||
from src.services.vector_service import VectorService
|
||||
from src.services.graph_service import GraphService
|
||||
from src.services.volatile_service import VolatileCacheService
|
||||
|
||||
# Create component services
|
||||
vector_service = VectorService(qdrant_client, wiki_client, ollama_client)
|
||||
graph_service = GraphService(neo4j_client, wiki_client)
|
||||
volatile_service = VolatileCacheService(qdrant_client, ollama_client, settings)
|
||||
|
||||
# Create HybridRAG service
|
||||
return HybridRAGService(
|
||||
@@ -50,7 +48,8 @@ def get_hybrid_rag_service(
|
||||
searxng_client=searxng_client,
|
||||
ollama_client=ollama_client,
|
||||
content_extractor=content_extractor,
|
||||
settings=settings
|
||||
settings=settings,
|
||||
volatile_service=volatile_service
|
||||
)
|
||||
|
||||
|
||||
@@ -62,26 +61,28 @@ async def hybrid_search(
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Execute HybridRAG query combining vector, graph, and web search.
|
||||
Execute HybridRAG query combining vector, graph, volatile cache, and web search.
|
||||
|
||||
**6-Phase Pipeline:**
|
||||
1. **Query Enhancement**: Extract keywords/synonyms with LLM
|
||||
2. **Parallel Retrieval**: Search vector (Qdrant), graph (Neo4j), web (SearXNG)
|
||||
3. **RRF Fusion**: Merge results with Reciprocal Rank Fusion
|
||||
2. **Parallel Retrieval**: Search vector (Qdrant), graph (Neo4j), volatile cache, web (SearXNG)
|
||||
3. **RRF Fusion**: Merge results with Reciprocal Rank Fusion (volatile gets priority boost)
|
||||
4. **Enrichment**: Add related documents via shared entities
|
||||
5. **LLM Re-ranking**: Re-rank with mistral-nemo for relevance
|
||||
5. **LLM Re-ranking**: Re-rank with configured model for relevance
|
||||
6. **Context Formatting**: Format for LLM consumption
|
||||
7. **Persistence**: Store for Librarian knowledge consolidation
|
||||
|
||||
**Example Request:**
|
||||
```json
|
||||
{
|
||||
"query": "How does Docker orchestration work with Kubernetes?",
|
||||
"query": "What's the weather in Rotterdam?",
|
||||
"user": "jpmschweitzer",
|
||||
"config": {
|
||||
"vector_limit": 10,
|
||||
"graph_limit": 10,
|
||||
"web_limit": 5,
|
||||
"volatile_limit": 5,
|
||||
"enable_volatile": true,
|
||||
"enable_reranking": true,
|
||||
"final_result_count": 10
|
||||
}
|
||||
@@ -89,7 +90,7 @@ async def hybrid_search(
|
||||
```
|
||||
|
||||
**Returns:**
|
||||
- Ranked results from all sources
|
||||
- Ranked results from all sources (wiki, volatile, web)
|
||||
- Extracted keywords/synonyms
|
||||
- Related dossiers (via graph)
|
||||
- Formatted context for LLM
|
||||
|
||||
@@ -0,0 +1,860 @@
|
||||
"""
|
||||
Maintenance router for Library Desk cleanup operations.
|
||||
|
||||
Provides endpoints to clean up orphaned data in vectors and graph:
|
||||
- Orphan vector chunks (no matching page/document in graph)
|
||||
- Orphan entities (no MENTIONS relationships)
|
||||
- Stale documents (graph nodes with no matching wiki page)
|
||||
- Broken relationships
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Depends, Query
|
||||
from pydantic import BaseModel, Field
|
||||
from typing import Optional, List, Dict, Any
|
||||
import logging
|
||||
import time
|
||||
|
||||
from src.services.vector_service import VectorService
|
||||
from src.services.graph_service import GraphService
|
||||
from src.services.volatile_service import VolatileCacheService
|
||||
from src.core.dependencies import (
|
||||
VectorServiceDep, GraphServiceDep, WikiJSDep, RedisDep,
|
||||
QdrantDep, OllamaDep, verify_api_key
|
||||
)
|
||||
from src.config import get_settings
|
||||
from datetime import datetime, timezone
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/maintenance", tags=["Maintenance"])
|
||||
|
||||
# Redis key for tracking last cleanup timestamp
|
||||
LAST_CLEANUP_KEY = "library:maintenance:last_cleanup:{user}"
|
||||
|
||||
|
||||
async def _get_last_cleanup(redis, user: str) -> Optional[str]:
|
||||
"""Get last cleanup timestamp from Redis."""
|
||||
try:
|
||||
key = LAST_CLEANUP_KEY.format(user=user)
|
||||
return await redis.get(key)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to get last cleanup timestamp: {e}")
|
||||
return None
|
||||
|
||||
|
||||
async def _set_last_cleanup(redis, user: str) -> None:
|
||||
"""Store current timestamp as last cleanup time."""
|
||||
try:
|
||||
key = LAST_CLEANUP_KEY.format(user=user)
|
||||
timestamp = datetime.now(timezone.utc).isoformat()
|
||||
# Keep for 30 days
|
||||
await redis.setex(key, 86400 * 30, timestamp)
|
||||
logger.info(f"Recorded cleanup timestamp: {timestamp}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to store cleanup timestamp: {e}")
|
||||
|
||||
|
||||
async def _find_unindexed_pages(
|
||||
wiki_pages: List[Dict],
|
||||
chunk_refs: List[Dict],
|
||||
graph_docs: List[Dict]
|
||||
) -> tuple[List[int], List[int]]:
|
||||
"""
|
||||
Find wiki pages that are missing from vectors or graph.
|
||||
|
||||
Returns:
|
||||
Tuple of (pages_without_vectors, pages_without_graph)
|
||||
"""
|
||||
# Build sets of indexed page IDs
|
||||
vectorized_page_ids = {
|
||||
ref.get("page_id") for ref in chunk_refs
|
||||
if ref.get("doc_type") == "wiki" and ref.get("page_id")
|
||||
}
|
||||
graphed_page_ids = {
|
||||
doc.get("page_id") for doc in graph_docs
|
||||
if doc.get("doc_type") == "wiki" and doc.get("page_id")
|
||||
}
|
||||
|
||||
# Find wiki pages missing from each store
|
||||
pages_without_vectors = []
|
||||
pages_without_graph = []
|
||||
|
||||
for page in wiki_pages:
|
||||
page_id = page.get("id")
|
||||
if not page_id:
|
||||
continue
|
||||
|
||||
if page_id not in vectorized_page_ids:
|
||||
pages_without_vectors.append(page_id)
|
||||
if page_id not in graphed_page_ids:
|
||||
pages_without_graph.append(page_id)
|
||||
|
||||
return pages_without_vectors, pages_without_graph
|
||||
|
||||
|
||||
async def _reindex_missing_pages(
|
||||
page_ids: List[int],
|
||||
user: str,
|
||||
vector_service,
|
||||
graph_service
|
||||
) -> tuple[int, int, List[int]]:
|
||||
"""
|
||||
Reindex pages that are missing from vectors or graph.
|
||||
|
||||
Returns:
|
||||
Tuple of (pages_reindexed, pages_failed, failed_page_ids)
|
||||
"""
|
||||
reindexed = 0
|
||||
failed = 0
|
||||
failed_ids = []
|
||||
|
||||
for page_id in page_ids:
|
||||
try:
|
||||
# Index to both stores
|
||||
vector_result = await vector_service.update_from_page(page_id, user, force_refresh=True)
|
||||
graph_result = await graph_service.update_from_page(page_id, user, force_refresh=True)
|
||||
|
||||
if vector_result.success and graph_result.success:
|
||||
reindexed += 1
|
||||
logger.info(f"Reindexed missing page {page_id}")
|
||||
else:
|
||||
failed += 1
|
||||
failed_ids.append(page_id)
|
||||
logger.warning(f"Failed to reindex page {page_id}: vector={vector_result.success}, graph={graph_result.success}")
|
||||
|
||||
except Exception as e:
|
||||
failed += 1
|
||||
failed_ids.append(page_id)
|
||||
logger.error(f"Error reindexing page {page_id}: {e}")
|
||||
|
||||
return reindexed, failed, failed_ids
|
||||
|
||||
|
||||
# ========== Response Models ==========
|
||||
|
||||
class CleanupResult(BaseModel):
|
||||
"""Result of a cleanup operation."""
|
||||
orphans_found: int = Field(default=0, description="Number of orphans detected")
|
||||
orphans_purged: int = Field(default=0, description="Number of orphans deleted")
|
||||
duration_ms: float = Field(description="Operation duration in milliseconds")
|
||||
|
||||
|
||||
class VectorCleanupResponse(BaseModel):
|
||||
"""Response from vector cleanup operation."""
|
||||
success: bool
|
||||
wiki_chunks: CleanupResult
|
||||
document_chunks: CleanupResult
|
||||
chunks_without_graph: CleanupResult # Vectors with no graph node
|
||||
total_chunks_scanned: int
|
||||
total_orphans_purged: int
|
||||
duration_ms: float
|
||||
|
||||
|
||||
class GraphCleanupResponse(BaseModel):
|
||||
"""Response from graph cleanup operation."""
|
||||
success: bool
|
||||
orphan_entities: CleanupResult
|
||||
stale_wiki_documents: CleanupResult
|
||||
stale_store_documents: CleanupResult
|
||||
docs_without_vectors: CleanupResult # Graph nodes with no vectors
|
||||
broken_relationships_cleaned: int
|
||||
duration_ms: float
|
||||
|
||||
|
||||
class FullCleanupResponse(BaseModel):
|
||||
"""Response from full cleanup operation."""
|
||||
success: bool
|
||||
vector_cleanup: VectorCleanupResponse
|
||||
graph_cleanup: GraphCleanupResponse
|
||||
total_duration_ms: float
|
||||
|
||||
|
||||
class HealthCheckResponse(BaseModel):
|
||||
"""Response from maintenance health check."""
|
||||
status: str = Field(description="Health status: healthy, degraded, or unhealthy")
|
||||
orphan_vector_count: int = Field(description="Number of orphan vector chunks (no source)")
|
||||
orphan_entity_count: int = Field(description="Number of orphan entities")
|
||||
stale_document_count: int = Field(description="Number of stale document nodes")
|
||||
vectors_without_graph: int = Field(default=0, description="Vector chunks with no graph node")
|
||||
docs_without_vectors: int = Field(default=0, description="Graph docs with no vectors")
|
||||
unindexed_pages: int = Field(default=0, description="Wiki pages missing from indexes")
|
||||
last_cleanup: Optional[str] = Field(default=None, description="Timestamp of last cleanup")
|
||||
recommendations: List[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class ReindexResponse(BaseModel):
|
||||
"""Response from reindex operation."""
|
||||
success: bool
|
||||
page_id: int
|
||||
vectors_deleted: int
|
||||
vectors_created: int
|
||||
graph_updated: bool
|
||||
duration_ms: float
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
class ReindexMissingResult(BaseModel):
|
||||
"""Result of reindexing missing pages."""
|
||||
pages_without_vectors: int = Field(description="Wiki pages with no vector embeddings")
|
||||
pages_without_graph: int = Field(description="Wiki pages with no graph Document node")
|
||||
pages_reindexed: int = Field(description="Pages successfully reindexed")
|
||||
pages_failed: int = Field(description="Pages that failed to reindex")
|
||||
failed_page_ids: List[int] = Field(default_factory=list)
|
||||
duration_ms: float
|
||||
|
||||
|
||||
class ReconcileIndexResponse(BaseModel):
|
||||
"""Response from reconcile-index operation (cleanup + reindex-missing)."""
|
||||
success: bool
|
||||
cleanup: FullCleanupResponse
|
||||
reindex_missing: ReindexMissingResult
|
||||
total_duration_ms: float
|
||||
|
||||
|
||||
class VolatileCleanupResponse(BaseModel):
|
||||
"""Response from volatile cache cleanup operation."""
|
||||
success: bool
|
||||
collections_processed: int
|
||||
total_expired_purged: int
|
||||
by_collection: Dict[str, int] = Field(default_factory=dict)
|
||||
duration_ms: float
|
||||
|
||||
|
||||
# ========== Endpoints ==========
|
||||
|
||||
@router.post("/cleanup/vectors", response_model=VectorCleanupResponse)
|
||||
async def cleanup_vectors(
|
||||
user: str = Query(..., description="User identifier"),
|
||||
dry_run: bool = Query(False, description="If true, only count orphans without deleting"),
|
||||
vector_service: VectorServiceDep = None,
|
||||
graph_service: GraphServiceDep = None,
|
||||
wiki_client: WikiJSDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Find and purge orphan vector chunks.
|
||||
|
||||
Orphan chunks are vector embeddings that reference:
|
||||
- Wiki pages that no longer exist
|
||||
- Document Store documents that no longer exist
|
||||
- Chunks with no corresponding graph Document node (bidirectional check)
|
||||
|
||||
**Scheduler Task** - Recommended to run daily.
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
# Get all vector chunk references
|
||||
chunk_refs = await vector_service.get_all_chunk_references(user)
|
||||
total_scanned = len(chunk_refs)
|
||||
|
||||
# Get all valid page IDs from wiki
|
||||
wiki_pages = await wiki_client.list_all_pages()
|
||||
valid_page_ids = {p.get("id") for p in wiki_pages if p.get("id")}
|
||||
|
||||
# Get all valid document references from graph
|
||||
graph_docs = await graph_service.get_all_document_references(user)
|
||||
valid_doc_ids = {d["document_id"] for d in graph_docs if d.get("document_id")}
|
||||
|
||||
# Find orphan wiki chunks (page_id not in wiki)
|
||||
wiki_orphan_ids = []
|
||||
doc_orphan_ids = []
|
||||
|
||||
for ref in chunk_refs:
|
||||
doc_type = ref.get("doc_type", "wiki")
|
||||
|
||||
if doc_type == "wiki":
|
||||
page_id = ref.get("page_id")
|
||||
if page_id and page_id not in valid_page_ids:
|
||||
wiki_orphan_ids.append(ref["chunk_id"])
|
||||
else:
|
||||
document_id = ref.get("document_id")
|
||||
if document_id and document_id not in valid_doc_ids:
|
||||
doc_orphan_ids.append(ref["chunk_id"])
|
||||
|
||||
# Bidirectional check: chunks with no graph node
|
||||
chunks_without_graph = vector_service.find_chunks_without_graph_nodes(
|
||||
chunk_refs, graph_docs
|
||||
)
|
||||
|
||||
# Purge orphans if not dry run
|
||||
wiki_purged = 0
|
||||
doc_purged = 0
|
||||
graph_orphans_purged = 0
|
||||
|
||||
if not dry_run:
|
||||
if wiki_orphan_ids:
|
||||
wiki_purged = await vector_service.purge_chunks_by_ids(user, wiki_orphan_ids)
|
||||
if doc_orphan_ids:
|
||||
doc_purged = await vector_service.purge_chunks_by_ids(user, doc_orphan_ids)
|
||||
if chunks_without_graph:
|
||||
graph_orphans_purged = await vector_service.purge_chunks_by_ids(
|
||||
user, chunks_without_graph
|
||||
)
|
||||
|
||||
duration_ms = (time.time() - start_time) * 1000
|
||||
|
||||
return VectorCleanupResponse(
|
||||
success=True,
|
||||
wiki_chunks=CleanupResult(
|
||||
orphans_found=len(wiki_orphan_ids),
|
||||
orphans_purged=wiki_purged,
|
||||
duration_ms=duration_ms / 3
|
||||
),
|
||||
document_chunks=CleanupResult(
|
||||
orphans_found=len(doc_orphan_ids),
|
||||
orphans_purged=doc_purged,
|
||||
duration_ms=duration_ms / 3
|
||||
),
|
||||
chunks_without_graph=CleanupResult(
|
||||
orphans_found=len(chunks_without_graph),
|
||||
orphans_purged=graph_orphans_purged,
|
||||
duration_ms=duration_ms / 3
|
||||
),
|
||||
total_chunks_scanned=total_scanned,
|
||||
total_orphans_purged=wiki_purged + doc_purged + graph_orphans_purged,
|
||||
duration_ms=duration_ms
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Vector cleanup failed: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.post("/cleanup/graph", response_model=GraphCleanupResponse)
|
||||
async def cleanup_graph(
|
||||
user: str = Query(..., description="User identifier"),
|
||||
dry_run: bool = Query(False, description="If true, only count orphans without deleting"),
|
||||
vector_service: VectorServiceDep = None,
|
||||
graph_service: GraphServiceDep = None,
|
||||
wiki_client: WikiJSDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Find and purge orphan entities and stale documents from the graph.
|
||||
|
||||
Cleans up:
|
||||
- Orphan entities (no MENTIONS relationships)
|
||||
- Stale wiki Document nodes (page deleted from Wiki.js)
|
||||
- Stale Document Store nodes (document deleted)
|
||||
- Graph Document nodes with no corresponding vectors (bidirectional check)
|
||||
- Broken FOUND relationships from SearchQuery nodes
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
# 1. Find orphan entities
|
||||
orphan_entities = await graph_service.find_orphan_entities(user)
|
||||
entities_purged = 0
|
||||
|
||||
if not dry_run and orphan_entities:
|
||||
entities_purged = await graph_service.purge_orphan_entities(user)
|
||||
|
||||
# 2. Find stale wiki documents
|
||||
graph_docs = await graph_service.get_all_document_references(user)
|
||||
wiki_docs = [d for d in graph_docs if d.get("doc_type") == "wiki" and d.get("page_id")]
|
||||
|
||||
# Get valid wiki page IDs
|
||||
wiki_pages = await wiki_client.list_all_pages()
|
||||
valid_page_ids = {p.get("id") for p in wiki_pages if p.get("id")}
|
||||
|
||||
stale_wiki_ids = [d["page_id"] for d in wiki_docs if d["page_id"] not in valid_page_ids]
|
||||
wiki_docs_purged = 0
|
||||
|
||||
if not dry_run and stale_wiki_ids:
|
||||
wiki_docs_purged = await graph_service.purge_stale_documents_by_ids(
|
||||
user, page_ids=stale_wiki_ids
|
||||
)
|
||||
|
||||
# 3. Find stale Document Store documents (these would be detected differently)
|
||||
# For now, Document Store docs are only stale if the collection is deleted
|
||||
# This will be more relevant once DocumentService exists
|
||||
stale_store_docs = 0
|
||||
store_docs_purged = 0
|
||||
|
||||
# 4. Bidirectional check: graph docs with no vectors
|
||||
chunk_refs = await vector_service.get_all_chunk_references(user)
|
||||
docs_without_vectors = await graph_service.find_documents_without_vectors(
|
||||
user, chunk_refs
|
||||
)
|
||||
docs_without_vectors_purged = 0
|
||||
|
||||
if not dry_run and docs_without_vectors:
|
||||
# Purge wiki docs without vectors
|
||||
wiki_orphans = [d["page_id"] for d in docs_without_vectors
|
||||
if d.get("doc_type") == "wiki" and d.get("page_id")]
|
||||
doc_orphans = [d["document_id"] for d in docs_without_vectors
|
||||
if d.get("doc_type") != "wiki" and d.get("document_id")]
|
||||
|
||||
if wiki_orphans:
|
||||
docs_without_vectors_purged += await graph_service.purge_stale_documents_by_ids(
|
||||
user, page_ids=wiki_orphans
|
||||
)
|
||||
if doc_orphans:
|
||||
docs_without_vectors_purged += await graph_service.purge_stale_documents_by_ids(
|
||||
user, document_ids=doc_orphans
|
||||
)
|
||||
|
||||
# 5. Clean broken relationships
|
||||
broken_rels_cleaned = 0
|
||||
if not dry_run:
|
||||
broken_rels_cleaned = await graph_service.cleanup_broken_relationships(user)
|
||||
|
||||
duration_ms = (time.time() - start_time) * 1000
|
||||
|
||||
return GraphCleanupResponse(
|
||||
success=True,
|
||||
orphan_entities=CleanupResult(
|
||||
orphans_found=len(orphan_entities),
|
||||
orphans_purged=entities_purged,
|
||||
duration_ms=duration_ms / 5
|
||||
),
|
||||
stale_wiki_documents=CleanupResult(
|
||||
orphans_found=len(stale_wiki_ids),
|
||||
orphans_purged=wiki_docs_purged,
|
||||
duration_ms=duration_ms / 5
|
||||
),
|
||||
stale_store_documents=CleanupResult(
|
||||
orphans_found=stale_store_docs,
|
||||
orphans_purged=store_docs_purged,
|
||||
duration_ms=duration_ms / 5
|
||||
),
|
||||
docs_without_vectors=CleanupResult(
|
||||
orphans_found=len(docs_without_vectors),
|
||||
orphans_purged=docs_without_vectors_purged,
|
||||
duration_ms=duration_ms / 5
|
||||
),
|
||||
broken_relationships_cleaned=broken_rels_cleaned,
|
||||
duration_ms=duration_ms
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Graph cleanup failed: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.post("/cleanup/all", response_model=FullCleanupResponse)
|
||||
async def cleanup_all(
|
||||
user: str = Query(..., description="User identifier"),
|
||||
dry_run: bool = Query(False, description="If true, only count orphans without deleting"),
|
||||
vector_service: VectorServiceDep = None,
|
||||
graph_service: GraphServiceDep = None,
|
||||
wiki_client: WikiJSDep = None,
|
||||
redis: RedisDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Full cleanup of vectors and graph.
|
||||
|
||||
Runs both vector and graph cleanup in sequence.
|
||||
|
||||
**Scheduler Task** - Recommended to run daily at low-traffic time.
|
||||
|
||||
**Scheduler Integration:**
|
||||
```json
|
||||
{
|
||||
"task_name": "library_maintenance",
|
||||
"schedule": "0 4 * * *",
|
||||
"endpoint": "POST /maintenance/cleanup/all?user=jpmschweitzer",
|
||||
"description": "Daily cleanup of orphan vectors and graph nodes"
|
||||
}
|
||||
```
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
# Run vector cleanup
|
||||
vector_result = await cleanup_vectors(
|
||||
user=user,
|
||||
dry_run=dry_run,
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
wiki_client=wiki_client,
|
||||
api_key=api_key
|
||||
)
|
||||
|
||||
# Run graph cleanup
|
||||
graph_result = await cleanup_graph(
|
||||
user=user,
|
||||
dry_run=dry_run,
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
wiki_client=wiki_client,
|
||||
api_key=api_key
|
||||
)
|
||||
|
||||
total_duration_ms = (time.time() - start_time) * 1000
|
||||
|
||||
# Record cleanup timestamp (only if not dry run)
|
||||
if not dry_run and redis:
|
||||
await _set_last_cleanup(redis, user)
|
||||
|
||||
return FullCleanupResponse(
|
||||
success=True,
|
||||
vector_cleanup=vector_result,
|
||||
graph_cleanup=graph_result,
|
||||
total_duration_ms=total_duration_ms
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Full cleanup failed: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.post("/cleanup/volatile", response_model=VolatileCleanupResponse)
|
||||
async def cleanup_volatile(
|
||||
qdrant: QdrantDep = None,
|
||||
ollama: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Purge expired volatile cache records across all users.
|
||||
|
||||
Loops through all volatile_* collections and removes records where
|
||||
ttl_expiry < current_timestamp.
|
||||
|
||||
**Scheduler Task** - Recommended to run every 10 minutes.
|
||||
|
||||
**Scheduler Integration:**
|
||||
```json
|
||||
{
|
||||
"task_name": "volatile_cleanup",
|
||||
"schedule": "*/10 * * * *",
|
||||
"endpoint": "POST /maintenance/cleanup/volatile",
|
||||
"description": "Purge expired volatile cache records"
|
||||
}
|
||||
```
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
settings = get_settings()
|
||||
service = VolatileCacheService(
|
||||
qdrant_client=qdrant,
|
||||
ollama_client=ollama,
|
||||
settings=settings
|
||||
)
|
||||
|
||||
# Purge expired from all volatile collections
|
||||
results = await service.purge_all_expired()
|
||||
|
||||
total_purged = sum(results.values())
|
||||
duration_ms = (time.time() - start_time) * 1000
|
||||
|
||||
logger.info(f"Volatile cleanup complete: {total_purged} expired records purged from {len(results)} collections")
|
||||
|
||||
return VolatileCleanupResponse(
|
||||
success=True,
|
||||
collections_processed=len(results),
|
||||
total_expired_purged=total_purged,
|
||||
by_collection=results,
|
||||
duration_ms=duration_ms
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Volatile cleanup failed: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.get("/health", response_model=HealthCheckResponse)
|
||||
async def maintenance_health(
|
||||
user: str = Query(..., description="User identifier"),
|
||||
detailed: bool = Query(False, description="If true, run full orphan analysis (slower)"),
|
||||
vector_service: VectorServiceDep = None,
|
||||
graph_service: GraphServiceDep = None,
|
||||
wiki_client: WikiJSDep = None,
|
||||
redis: RedisDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Health check for maintenance status.
|
||||
|
||||
**Lightweight mode (default)**: Returns last cleanup timestamp and basic status.
|
||||
Use for frequent uptime checks (every 30s).
|
||||
|
||||
**Detailed mode (?detailed=true)**: Runs full orphan/unindexed analysis.
|
||||
Use for dashboards or before running reconcile-index.
|
||||
"""
|
||||
try:
|
||||
# Get last cleanup timestamp from Redis (lightweight)
|
||||
last_cleanup = None
|
||||
if redis:
|
||||
last_cleanup = await _get_last_cleanup(redis, user)
|
||||
|
||||
# Lightweight mode - just return basic status
|
||||
if not detailed:
|
||||
return HealthCheckResponse(
|
||||
status="healthy" if last_cleanup else "unknown",
|
||||
orphan_vector_count=0,
|
||||
orphan_entity_count=0,
|
||||
stale_document_count=0,
|
||||
vectors_without_graph=0,
|
||||
docs_without_vectors=0,
|
||||
unindexed_pages=0,
|
||||
last_cleanup=last_cleanup,
|
||||
recommendations=[] if last_cleanup else ["No cleanup recorded. Run POST /maintenance/reconcile-index"]
|
||||
)
|
||||
|
||||
# Detailed mode - full analysis
|
||||
recommendations = []
|
||||
|
||||
# Count orphan vector chunks
|
||||
chunk_refs = await vector_service.get_all_chunk_references(user)
|
||||
wiki_pages = await wiki_client.list_all_pages()
|
||||
valid_page_ids = {p.get("id") for p in wiki_pages if p.get("id")}
|
||||
|
||||
orphan_vector_count = sum(
|
||||
1 for ref in chunk_refs
|
||||
if ref.get("doc_type") == "wiki"
|
||||
and ref.get("page_id") not in valid_page_ids
|
||||
)
|
||||
|
||||
if orphan_vector_count > 10:
|
||||
recommendations.append(
|
||||
f"Found {orphan_vector_count} orphan vector chunks. "
|
||||
"Consider running POST /maintenance/cleanup/vectors"
|
||||
)
|
||||
|
||||
# Count orphan entities
|
||||
orphan_entities = await graph_service.find_orphan_entities(user)
|
||||
orphan_entity_count = len(orphan_entities)
|
||||
|
||||
if orphan_entity_count > 5:
|
||||
recommendations.append(
|
||||
f"Found {orphan_entity_count} orphan entities. "
|
||||
"Consider running POST /maintenance/cleanup/graph"
|
||||
)
|
||||
|
||||
# Count stale documents
|
||||
graph_docs = await graph_service.get_all_document_references(user)
|
||||
wiki_docs = [d for d in graph_docs if d.get("doc_type") == "wiki" and d.get("page_id")]
|
||||
stale_document_count = sum(1 for d in wiki_docs if d["page_id"] not in valid_page_ids)
|
||||
|
||||
if stale_document_count > 0:
|
||||
recommendations.append(
|
||||
f"Found {stale_document_count} stale Document nodes. "
|
||||
"Consider running POST /maintenance/cleanup/graph"
|
||||
)
|
||||
|
||||
# Bidirectional: vectors without graph nodes
|
||||
vectors_without_graph = len(vector_service.find_chunks_without_graph_nodes(
|
||||
chunk_refs, graph_docs
|
||||
))
|
||||
|
||||
if vectors_without_graph > 5:
|
||||
recommendations.append(
|
||||
f"Found {vectors_without_graph} vectors without graph nodes. "
|
||||
"Consider running POST /maintenance/cleanup/vectors"
|
||||
)
|
||||
|
||||
# Bidirectional: graph docs without vectors
|
||||
docs_without_vectors_list = await graph_service.find_documents_without_vectors(
|
||||
user, chunk_refs
|
||||
)
|
||||
docs_without_vectors = len(docs_without_vectors_list)
|
||||
|
||||
if docs_without_vectors > 5:
|
||||
recommendations.append(
|
||||
f"Found {docs_without_vectors} graph docs without vectors. "
|
||||
"Consider running POST /maintenance/cleanup/graph"
|
||||
)
|
||||
|
||||
# Unindexed pages: wiki pages missing from vectors or graph
|
||||
pages_without_vectors, pages_without_graph = await _find_unindexed_pages(
|
||||
wiki_pages, chunk_refs, graph_docs
|
||||
)
|
||||
unindexed_pages = len(set(pages_without_vectors + pages_without_graph))
|
||||
|
||||
if unindexed_pages > 0:
|
||||
recommendations.append(
|
||||
f"Found {unindexed_pages} wiki pages not in indexes. "
|
||||
"Consider running POST /maintenance/reconcile-index"
|
||||
)
|
||||
|
||||
# Determine overall status
|
||||
total_issues = (orphan_vector_count + orphan_entity_count + stale_document_count +
|
||||
vectors_without_graph + docs_without_vectors + unindexed_pages)
|
||||
if total_issues == 0:
|
||||
status = "healthy"
|
||||
elif total_issues < 20:
|
||||
status = "degraded"
|
||||
else:
|
||||
status = "unhealthy"
|
||||
|
||||
# Get last cleanup timestamp from Redis
|
||||
last_cleanup = None
|
||||
if redis:
|
||||
last_cleanup = await _get_last_cleanup(redis, user)
|
||||
|
||||
return HealthCheckResponse(
|
||||
status=status,
|
||||
orphan_vector_count=orphan_vector_count,
|
||||
orphan_entity_count=orphan_entity_count,
|
||||
stale_document_count=stale_document_count,
|
||||
vectors_without_graph=vectors_without_graph,
|
||||
docs_without_vectors=docs_without_vectors,
|
||||
unindexed_pages=unindexed_pages,
|
||||
last_cleanup=last_cleanup,
|
||||
recommendations=recommendations
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Health check failed: {e}", exc_info=True)
|
||||
return HealthCheckResponse(
|
||||
status="unhealthy",
|
||||
orphan_vector_count=-1,
|
||||
orphan_entity_count=-1,
|
||||
stale_document_count=-1,
|
||||
vectors_without_graph=-1,
|
||||
docs_without_vectors=-1,
|
||||
unindexed_pages=-1,
|
||||
recommendations=[f"Health check failed: {str(e)}"]
|
||||
)
|
||||
|
||||
|
||||
@router.post("/reindex/{page_id}", response_model=ReindexResponse)
|
||||
async def reindex_page(
|
||||
page_id: int,
|
||||
user: str = Query(..., description="User identifier"),
|
||||
vector_service: VectorServiceDep = None,
|
||||
graph_service: GraphServiceDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Force re-index a wiki page.
|
||||
|
||||
Deletes existing vectors and graph data, then re-ingests.
|
||||
Useful for fixing corrupted or stale data for a specific page.
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
# Delete existing vectors
|
||||
vectors_deleted = await vector_service.delete_page_chunks(page_id, user)
|
||||
|
||||
# Delete and recreate graph node
|
||||
await graph_service.delete_page(page_id, user)
|
||||
|
||||
# Re-ingest
|
||||
vector_result = await vector_service.update_from_page(page_id, user, force_refresh=True)
|
||||
graph_result = await graph_service.update_from_page(page_id, user, force_refresh=True)
|
||||
|
||||
duration_ms = (time.time() - start_time) * 1000
|
||||
|
||||
return ReindexResponse(
|
||||
success=vector_result.success and graph_result.success,
|
||||
page_id=page_id,
|
||||
vectors_deleted=vectors_deleted,
|
||||
vectors_created=vector_result.chunks_created,
|
||||
graph_updated=graph_result.success,
|
||||
duration_ms=duration_ms,
|
||||
error=vector_result.error_message or graph_result.error_message
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
duration_ms = (time.time() - start_time) * 1000
|
||||
logger.error(f"Reindex failed for page {page_id}: {e}", exc_info=True)
|
||||
return ReindexResponse(
|
||||
success=False,
|
||||
page_id=page_id,
|
||||
vectors_deleted=0,
|
||||
vectors_created=0,
|
||||
graph_updated=False,
|
||||
duration_ms=duration_ms,
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
|
||||
@router.post("/reconcile-index", response_model=ReconcileIndexResponse)
|
||||
async def reconcile_index(
|
||||
user: str = Query(..., description="User identifier"),
|
||||
dry_run: bool = Query(False, description="If true, only detect issues without fixing"),
|
||||
vector_service: VectorServiceDep = None,
|
||||
graph_service: GraphServiceDep = None,
|
||||
wiki_client: WikiJSDep = None,
|
||||
redis: RedisDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Full index reconciliation: cleanup orphans + reindex missing pages.
|
||||
|
||||
This is the recommended daily maintenance endpoint. It:
|
||||
1. Cleans up orphan vectors and graph nodes (data without sources)
|
||||
2. Reindexes wiki pages that are missing from vectors or graph
|
||||
|
||||
**Scheduler Task** - Recommended to run daily at low-traffic time.
|
||||
|
||||
**Scheduler Integration:**
|
||||
```json
|
||||
{
|
||||
"task_name": "library_reconcile_index",
|
||||
"schedule": "0 4 * * *",
|
||||
"endpoint": "POST /maintenance/reconcile-index?user=jpmschweitzer",
|
||||
"description": "Daily index reconciliation - cleanup + reindex missing"
|
||||
}
|
||||
```
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
# Phase 1: Run full cleanup
|
||||
cleanup_result = await cleanup_all(
|
||||
user=user,
|
||||
dry_run=dry_run,
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
wiki_client=wiki_client,
|
||||
redis=redis,
|
||||
api_key=api_key
|
||||
)
|
||||
|
||||
# Phase 2: Find and reindex missing pages
|
||||
reindex_start = time.time()
|
||||
|
||||
# Get current state
|
||||
wiki_pages = await wiki_client.list_all_pages()
|
||||
chunk_refs = await vector_service.get_all_chunk_references(user)
|
||||
graph_docs = await graph_service.get_all_document_references(user)
|
||||
|
||||
# Find pages missing from indexes
|
||||
pages_without_vectors, pages_without_graph = await _find_unindexed_pages(
|
||||
wiki_pages, chunk_refs, graph_docs
|
||||
)
|
||||
|
||||
# Combine unique page IDs that need reindexing
|
||||
missing_page_ids = list(set(pages_without_vectors + pages_without_graph))
|
||||
|
||||
# Reindex missing pages (unless dry run)
|
||||
reindexed = 0
|
||||
failed = 0
|
||||
failed_ids = []
|
||||
|
||||
if not dry_run and missing_page_ids:
|
||||
reindexed, failed, failed_ids = await _reindex_missing_pages(
|
||||
missing_page_ids, user, vector_service, graph_service
|
||||
)
|
||||
|
||||
reindex_duration = (time.time() - reindex_start) * 1000
|
||||
total_duration = (time.time() - start_time) * 1000
|
||||
|
||||
# Record reconciliation timestamp
|
||||
if not dry_run and redis:
|
||||
await _set_last_cleanup(redis, user)
|
||||
|
||||
return ReconcileIndexResponse(
|
||||
success=cleanup_result.success and failed == 0,
|
||||
cleanup=cleanup_result,
|
||||
reindex_missing=ReindexMissingResult(
|
||||
pages_without_vectors=len(pages_without_vectors),
|
||||
pages_without_graph=len(pages_without_graph),
|
||||
pages_reindexed=reindexed,
|
||||
pages_failed=failed,
|
||||
failed_page_ids=failed_ids,
|
||||
duration_ms=reindex_duration
|
||||
),
|
||||
total_duration_ms=total_duration
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Reconcile-index failed: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
@@ -0,0 +1,273 @@
|
||||
"""
|
||||
Volatile cache router for Library Desk API.
|
||||
|
||||
Endpoints for ephemeral cached data with TTL - weather, news, financial, etc.
|
||||
Data is stored as vectors in Qdrant for semantic search retrieval.
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Depends, Query
|
||||
import logging
|
||||
|
||||
from src.models.volatile import (
|
||||
VolatileRecordCreate,
|
||||
VolatileRecordResponse,
|
||||
VolatileListResponse,
|
||||
VolatileScheduledResponse,
|
||||
VolatileStatsResponse,
|
||||
VolatileDeleteResponse,
|
||||
VolatileNamespace,
|
||||
NAMESPACE_DEFAULT_TTL,
|
||||
)
|
||||
from src.services.volatile_service import VolatileCacheService
|
||||
from src.core.dependencies import verify_api_key, QdrantDep, OllamaDep
|
||||
from src.core.multi_tenancy import DEFAULT_USER
|
||||
from src.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/volatile", tags=["Volatile Cache"])
|
||||
|
||||
|
||||
def get_volatile_service(qdrant: QdrantDep, ollama: OllamaDep) -> VolatileCacheService:
|
||||
"""Get volatile cache service instance."""
|
||||
settings = get_settings()
|
||||
return VolatileCacheService(
|
||||
qdrant_client=qdrant,
|
||||
ollama_client=ollama,
|
||||
settings=settings
|
||||
)
|
||||
|
||||
|
||||
@router.get("/stats", response_model=VolatileStatsResponse)
|
||||
async def get_stats(
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
qdrant: QdrantDep = None,
|
||||
ollama: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Get volatile cache statistics.
|
||||
|
||||
Returns counts of records by namespace and scheduled refresh info.
|
||||
"""
|
||||
service = get_volatile_service(qdrant, ollama)
|
||||
stats = await service.get_stats(user)
|
||||
|
||||
return VolatileStatsResponse(
|
||||
total_records=stats["total_records"],
|
||||
by_namespace=stats["by_namespace"],
|
||||
scheduled_count=stats["scheduled_count"],
|
||||
total_memory_bytes=None,
|
||||
user=user,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/scheduled", response_model=VolatileScheduledResponse)
|
||||
async def get_scheduled(
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
qdrant: QdrantDep = None,
|
||||
ollama: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Get records with refresh schedules.
|
||||
|
||||
Used by scheduler to determine what volatile data needs refreshing.
|
||||
Returns all records that have a refresh_schedule cron expression set.
|
||||
"""
|
||||
service = get_volatile_service(qdrant, ollama)
|
||||
records = await service.get_scheduled(user)
|
||||
|
||||
return VolatileScheduledResponse(
|
||||
records=records,
|
||||
count=len(records),
|
||||
user=user,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/namespaces")
|
||||
async def list_namespaces(
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
List available namespaces and their default TTLs.
|
||||
|
||||
Returns predefined namespaces with their default TTL values.
|
||||
"""
|
||||
return {
|
||||
"namespaces": [
|
||||
{
|
||||
"name": ns.value,
|
||||
"default_ttl": NAMESPACE_DEFAULT_TTL.get(ns, 3600),
|
||||
"description": _get_namespace_description(ns),
|
||||
}
|
||||
for ns in VolatileNamespace
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
def _get_namespace_description(ns: VolatileNamespace) -> str:
|
||||
"""Get human-readable description for namespace."""
|
||||
descriptions = {
|
||||
VolatileNamespace.WEATHER: "Weather conditions and forecasts",
|
||||
VolatileNamespace.NEWS: "Headlines and breaking news",
|
||||
VolatileNamespace.FINANCIAL: "Stock prices, exchange rates, crypto",
|
||||
VolatileNamespace.TRANSIT: "Train/bus schedules, delays",
|
||||
VolatileNamespace.TRAFFIC: "Commute times, road conditions",
|
||||
VolatileNamespace.AIR_QUALITY: "Pollution levels, pollen counts",
|
||||
VolatileNamespace.SPORTS: "Live scores, upcoming matches",
|
||||
VolatileNamespace.SOCIAL: "Social media mentions, notifications",
|
||||
VolatileNamespace.SYSTEM: "Service health, infrastructure status",
|
||||
VolatileNamespace.CONTEXT: "Conversation context, session state",
|
||||
VolatileNamespace.CUSTOM: "User-defined volatile data",
|
||||
}
|
||||
return descriptions.get(ns, "Custom namespace")
|
||||
|
||||
|
||||
@router.get("/search")
|
||||
async def search_volatile(
|
||||
q: str = Query(..., min_length=1, description="Search query"),
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
limit: int = Query(default=5, ge=1, le=20, description="Maximum results"),
|
||||
threshold: float = Query(default=0.75, ge=0.5, le=1.0, description="Minimum similarity score"),
|
||||
qdrant: QdrantDep = None,
|
||||
ollama: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Semantic search across volatile data.
|
||||
|
||||
Searches all volatile data for semantically similar content.
|
||||
Higher threshold = stricter matching.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
GET /volatile/search?q=weather%20rotterdam&user=jpmschweitzer
|
||||
```
|
||||
"""
|
||||
service = get_volatile_service(qdrant, ollama)
|
||||
results = await service.search(user, q, limit=limit, score_threshold=threshold)
|
||||
|
||||
return {
|
||||
"query": q,
|
||||
"results": results,
|
||||
"count": len(results),
|
||||
"user": user,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/store", response_model=VolatileRecordResponse)
|
||||
async def store_volatile(
|
||||
namespace: str = Query(..., description="Data namespace (weather, news, etc.)"),
|
||||
key: str = Query(..., description="Record key (e.g., 'rotterdam', 'nos-headlines')"),
|
||||
request: VolatileRecordCreate = None,
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
qdrant: QdrantDep = None,
|
||||
ollama: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Store volatile data.
|
||||
|
||||
Data is converted to natural language and embedded for semantic search.
|
||||
If the same namespace+key already exists, it will be updated.
|
||||
|
||||
**Example Request:**
|
||||
```json
|
||||
POST /volatile/store?namespace=weather&key=rotterdam
|
||||
{
|
||||
"data": {
|
||||
"temperature": 8,
|
||||
"conditions": "Cloudy",
|
||||
"humidity": 85
|
||||
},
|
||||
"source": "openweathermap",
|
||||
"ttl": 1800,
|
||||
"refresh_schedule": "0 * * * *"
|
||||
}
|
||||
```
|
||||
|
||||
**Refresh Schedule:**
|
||||
Optional cron expression for automatic refresh. The scheduler
|
||||
will query `/volatile/scheduled` and trigger refreshes.
|
||||
"""
|
||||
# Validate namespace if not custom
|
||||
if namespace != VolatileNamespace.CUSTOM:
|
||||
try:
|
||||
VolatileNamespace(namespace)
|
||||
except ValueError:
|
||||
valid = [ns.value for ns in VolatileNamespace]
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid namespace '{namespace}'. Valid: {valid}"
|
||||
)
|
||||
|
||||
service = get_volatile_service(qdrant, ollama)
|
||||
|
||||
try:
|
||||
record = await service.store(
|
||||
user=user,
|
||||
namespace=namespace,
|
||||
key=key,
|
||||
data=request.data,
|
||||
source=request.source,
|
||||
ttl=request.ttl,
|
||||
refresh_schedule=request.refresh_schedule,
|
||||
)
|
||||
return record
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to store volatile record: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Failed to store record: {str(e)}")
|
||||
|
||||
|
||||
@router.get("/{namespace}/{key}", response_model=VolatileRecordResponse)
|
||||
async def get_record(
|
||||
namespace: str,
|
||||
key: str,
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
qdrant: QdrantDep = None,
|
||||
ollama: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Get a specific volatile record by namespace and key.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
GET /volatile/weather/rotterdam?user=jpmschweitzer
|
||||
```
|
||||
"""
|
||||
service = get_volatile_service(qdrant, ollama)
|
||||
record = await service.get(user, namespace, key)
|
||||
|
||||
if not record:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Record '{key}' not found in namespace '{namespace}'"
|
||||
)
|
||||
|
||||
return record
|
||||
|
||||
|
||||
@router.delete("/{namespace}/{key}", response_model=VolatileDeleteResponse)
|
||||
async def delete_record(
|
||||
namespace: str,
|
||||
key: str,
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
qdrant: QdrantDep = None,
|
||||
ollama: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Delete a specific volatile record.
|
||||
"""
|
||||
service = get_volatile_service(qdrant, ollama)
|
||||
deleted = await service.delete(user, namespace, key)
|
||||
|
||||
return VolatileDeleteResponse(
|
||||
key=key,
|
||||
namespace=namespace,
|
||||
deleted=deleted,
|
||||
user=user,
|
||||
)
|
||||
+5
-4
@@ -5,8 +5,7 @@ Endpoints for wiki page and dossier management.
|
||||
All operations are scoped to user namespaces for multi-tenancy.
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Depends, Query, Security, BackgroundTasks
|
||||
from fastapi.security import HTTPAuthorizationCredentials
|
||||
from fastapi import APIRouter, HTTPException, Depends, Query, BackgroundTasks
|
||||
from typing import Optional
|
||||
import logging
|
||||
|
||||
@@ -24,7 +23,7 @@ from src.clients.neo4j_client import Neo4jClient
|
||||
from src.clients.qdrant_client import QdrantClientWrapper
|
||||
from src.clients.ollama_client import OllamaClient
|
||||
from src.core.dependencies import (
|
||||
WikiJSDep, Neo4jDep, QdrantDep, OllamaDep, SearXNGDep,
|
||||
WikiJSDep, Neo4jDep, QdrantDep, OllamaDep, SearXNGDep, ContentExtractorDep,
|
||||
verify_api_key, get_settings, get_hybrid_rag_service, get_ingestion_service
|
||||
)
|
||||
from src.core.multi_tenancy import DEFAULT_USER
|
||||
@@ -180,6 +179,7 @@ async def smart_create_page(
|
||||
qdrant_client: QdrantDep,
|
||||
ollama_client: OllamaDep,
|
||||
searxng_client: SearXNGDep,
|
||||
content_extractor: ContentExtractorDep,
|
||||
settings: Settings = Depends(get_settings),
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
@@ -225,9 +225,10 @@ async def smart_create_page(
|
||||
graph_service=graph_service,
|
||||
searxng_client=searxng_client,
|
||||
ollama_client=ollama_client,
|
||||
content_extractor=content_extractor,
|
||||
settings=settings
|
||||
)
|
||||
wiki_page_writer = WikiPageWriter(ollama_client=ollama_client)
|
||||
wiki_page_writer = WikiPageWriter(ollama_client=ollama_client, settings=settings)
|
||||
|
||||
# Step 1-5: Research + Generate + Create page
|
||||
page, research_data = await wiki_service.smart_create_page(
|
||||
|
||||
@@ -47,7 +47,7 @@ class ConsolidationService:
|
||||
self.ollama = ollama
|
||||
self.wiki = wiki
|
||||
self.settings = settings
|
||||
self.wiki_page_writer = WikiPageWriter(ollama_client=ollama)
|
||||
self.wiki_page_writer = WikiPageWriter(ollama_client=ollama, settings=settings)
|
||||
self.ingestion_service = ingestion_service # Optional to avoid circular dependency
|
||||
|
||||
async def consolidate_knowledge(
|
||||
@@ -410,13 +410,19 @@ This is a PERSONAL knowledge base using Schema.org-aligned taxonomy that capture
|
||||
- Projects: Work projects, personal projects (Schema.org: Project)
|
||||
- Reference: General knowledge, how-tos (Custom extension)
|
||||
|
||||
Identify information worth documenting:
|
||||
1. New topics/people/things that deserve their own wiki page
|
||||
2. Facts that could enhance existing pages
|
||||
3. Entities (people, places, things, concepts) for the knowledge graph
|
||||
ANALYSIS STEPS:
|
||||
1. Read each web result carefully for substantive, factual content
|
||||
2. Identify genuinely novel information not likely already known
|
||||
3. Match topics to appropriate taxonomy categories
|
||||
4. Generate valid paths following the exact format below
|
||||
|
||||
Be INCLUSIVE - if someone searched for it, it's likely worth documenting.
|
||||
Personal information is just as valuable as technical information.
|
||||
RULES:
|
||||
- Do NOT suggest pages for topics with insufficient information in results
|
||||
- Do NOT invent entities not explicitly mentioned in results
|
||||
- Do NOT suggest paths that don't match the taxonomy exactly
|
||||
- Do NOT suggest generic or vague page topics
|
||||
- Be CONSERVATIVE - fewer high-quality suggestions is better than many low-quality ones
|
||||
- ONLY suggest documentation for substantive, specific information
|
||||
|
||||
**CRITICAL: Use ONLY these Schema.org-aligned path prefixes (case-sensitive):**
|
||||
|
||||
@@ -462,11 +468,12 @@ Return ONLY valid JSON:
|
||||
JSON:"""
|
||||
|
||||
try:
|
||||
# Call Ollama for analysis
|
||||
# Call Ollama for analysis (temperature=0.0 for consistent classification)
|
||||
response = await self.ollama.generate_text(
|
||||
prompt=prompt,
|
||||
model=self.settings.reranker_model, # Use mistral-nemo
|
||||
stream=False
|
||||
model=self.settings.ollama_model,
|
||||
stream=False,
|
||||
temperature=0.0
|
||||
)
|
||||
|
||||
if not response:
|
||||
|
||||
@@ -1077,8 +1077,18 @@ Feel free to expand it with more details!
|
||||
search_query,
|
||||
{"terms": all_terms, "limit": limit}
|
||||
)
|
||||
logger.info(f"Graph search found {len(results)} documents")
|
||||
return results
|
||||
|
||||
# Deduplicate by page_id (safety net for any edge cases)
|
||||
seen_page_ids = set()
|
||||
unique_results = []
|
||||
for r in results:
|
||||
page_id = r.get("page_id")
|
||||
if page_id and page_id not in seen_page_ids:
|
||||
seen_page_ids.add(page_id)
|
||||
unique_results.append(r)
|
||||
|
||||
logger.info(f"Graph search found {len(unique_results)} unique documents (raw: {len(results)})")
|
||||
return unique_results
|
||||
except Exception as e:
|
||||
logger.error(f"Graph document search failed: {e}", exc_info=True)
|
||||
return []
|
||||
@@ -1251,3 +1261,353 @@ Feel free to expand it with more details!
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to create entity mentions: {e}", exc_info=True)
|
||||
return 0
|
||||
|
||||
# ========== Cleanup Methods ==========
|
||||
|
||||
async def delete_document_node(
|
||||
self,
|
||||
document_id: str,
|
||||
user: str
|
||||
) -> int:
|
||||
"""
|
||||
Delete a Document Store document node and all its relationships.
|
||||
|
||||
Args:
|
||||
document_id: Document UUID (Document Store)
|
||||
user: User identifier
|
||||
|
||||
Returns:
|
||||
Number of nodes deleted (1 if successful, 0 if not found)
|
||||
"""
|
||||
user_doc_label = get_neo4j_user_label(user)
|
||||
|
||||
delete_query = f"""
|
||||
MATCH (d:{user_doc_label}:Document {{document_id: $document_id}})
|
||||
DETACH DELETE d
|
||||
RETURN count(d) as deleted_count
|
||||
"""
|
||||
|
||||
try:
|
||||
result = await self.neo4j.execute_query(
|
||||
delete_query,
|
||||
{"document_id": document_id}
|
||||
)
|
||||
|
||||
deleted_count = result[0]["deleted_count"] if result else 0
|
||||
|
||||
if deleted_count > 0:
|
||||
logger.info(f"Deleted Document node for document {document_id}")
|
||||
else:
|
||||
logger.warning(f"No Document node found for document {document_id}")
|
||||
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to delete document {document_id} from graph: {e}", exc_info=True)
|
||||
return 0
|
||||
|
||||
async def delete_collection_node(
|
||||
self,
|
||||
collection_id: str,
|
||||
user: str
|
||||
) -> int:
|
||||
"""
|
||||
Delete a DocumentCollection node and all contained documents.
|
||||
|
||||
Args:
|
||||
collection_id: Collection UUID
|
||||
user: User identifier
|
||||
|
||||
Returns:
|
||||
Number of nodes deleted (collection + documents)
|
||||
"""
|
||||
user_doc_label = get_neo4j_user_label(user)
|
||||
|
||||
# Delete collection and all documents it contains
|
||||
delete_query = f"""
|
||||
MATCH (c:{user_doc_label}:DocumentCollection {{id: $collection_id}})
|
||||
OPTIONAL MATCH (c)-[:CONTAINS]->(d:Document)
|
||||
DETACH DELETE c, d
|
||||
RETURN count(c) + count(d) as deleted_count
|
||||
"""
|
||||
|
||||
try:
|
||||
result = await self.neo4j.execute_query(
|
||||
delete_query,
|
||||
{"collection_id": collection_id}
|
||||
)
|
||||
|
||||
deleted_count = result[0]["deleted_count"] if result else 0
|
||||
logger.info(f"Deleted collection {collection_id} with {deleted_count} total nodes")
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to delete collection {collection_id}: {e}", exc_info=True)
|
||||
return 0
|
||||
|
||||
async def find_orphan_entities(
|
||||
self,
|
||||
user: str
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Find entities with no MENTIONS relationships (orphaned).
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
|
||||
Returns:
|
||||
List of orphaned entities {id, name, type}
|
||||
"""
|
||||
from src.core.multi_tenancy import get_neo4j_user_base_label
|
||||
|
||||
user_base_label = get_neo4j_user_base_label(user)
|
||||
|
||||
query = f"""
|
||||
MATCH (e:{user_base_label})
|
||||
WHERE NOT e:Document
|
||||
AND NOT e:DocumentCollection
|
||||
AND NOT EXISTS {{ (d:Document)-[:MENTIONS]->(e) }}
|
||||
RETURN elementId(e) as id, e.name as name, labels(e) as labels
|
||||
"""
|
||||
|
||||
try:
|
||||
results = await self.neo4j.execute_query(query, {})
|
||||
|
||||
orphans = []
|
||||
for r in results:
|
||||
labels = r.get("labels", [])
|
||||
entity_type = next(
|
||||
(l for l in labels if l != user_base_label),
|
||||
"Unknown"
|
||||
)
|
||||
orphans.append({
|
||||
"id": r["id"],
|
||||
"name": r["name"],
|
||||
"type": entity_type
|
||||
})
|
||||
|
||||
logger.info(f"Found {len(orphans)} orphan entities for user {user}")
|
||||
return orphans
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to find orphan entities: {e}", exc_info=True)
|
||||
return []
|
||||
|
||||
async def purge_orphan_entities(
|
||||
self,
|
||||
user: str
|
||||
) -> int:
|
||||
"""
|
||||
Delete all orphaned entities (entities with no MENTIONS relationships).
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
|
||||
Returns:
|
||||
Number of entities purged
|
||||
"""
|
||||
from src.core.multi_tenancy import get_neo4j_user_base_label
|
||||
|
||||
user_base_label = get_neo4j_user_base_label(user)
|
||||
|
||||
query = f"""
|
||||
MATCH (e:{user_base_label})
|
||||
WHERE NOT e:Document
|
||||
AND NOT e:DocumentCollection
|
||||
AND NOT EXISTS {{ (d:Document)-[:MENTIONS]->(e) }}
|
||||
DETACH DELETE e
|
||||
RETURN count(e) as purged_count
|
||||
"""
|
||||
|
||||
try:
|
||||
results = await self.neo4j.execute_query(query, {})
|
||||
purged_count = results[0]["purged_count"] if results else 0
|
||||
|
||||
logger.info(f"Purged {purged_count} orphan entities for user {user}")
|
||||
return purged_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to purge orphan entities: {e}", exc_info=True)
|
||||
return 0
|
||||
|
||||
async def get_all_document_references(
|
||||
self,
|
||||
user: str
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Get all Document node references for orphan detection.
|
||||
|
||||
Returns page_id for wiki docs and document_id for Document Store docs.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
|
||||
Returns:
|
||||
List of document references {page_id, document_id, doc_type, title}
|
||||
"""
|
||||
user_doc_label = get_neo4j_user_label(user)
|
||||
|
||||
query = f"""
|
||||
MATCH (d:{user_doc_label}:Document)
|
||||
RETURN d.page_id as page_id,
|
||||
d.document_id as document_id,
|
||||
COALESCE(d.doc_type, 'wiki') as doc_type,
|
||||
d.title as title
|
||||
"""
|
||||
|
||||
try:
|
||||
results = await self.neo4j.execute_query(query, {})
|
||||
|
||||
references = []
|
||||
for r in results:
|
||||
references.append({
|
||||
"page_id": r.get("page_id"),
|
||||
"document_id": r.get("document_id"),
|
||||
"doc_type": r.get("doc_type", "wiki"),
|
||||
"title": r.get("title")
|
||||
})
|
||||
|
||||
logger.info(f"Found {len(references)} document references for user {user}")
|
||||
return references
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get document references: {e}", exc_info=True)
|
||||
return []
|
||||
|
||||
async def purge_stale_documents_by_ids(
|
||||
self,
|
||||
user: str,
|
||||
page_ids: List[int] = None,
|
||||
document_ids: List[str] = None
|
||||
) -> int:
|
||||
"""
|
||||
Delete specific stale Document nodes by their IDs.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
page_ids: List of wiki page IDs to delete
|
||||
document_ids: List of Document Store document IDs to delete
|
||||
|
||||
Returns:
|
||||
Number of documents purged
|
||||
"""
|
||||
user_doc_label = get_neo4j_user_label(user)
|
||||
total_purged = 0
|
||||
|
||||
try:
|
||||
# Purge by page_id (wiki docs)
|
||||
if page_ids:
|
||||
query = f"""
|
||||
MATCH (d:{user_doc_label}:Document)
|
||||
WHERE d.page_id IN $page_ids
|
||||
DETACH DELETE d
|
||||
RETURN count(d) as purged_count
|
||||
"""
|
||||
results = await self.neo4j.execute_query(query, {"page_ids": page_ids})
|
||||
count = results[0]["purged_count"] if results else 0
|
||||
total_purged += count
|
||||
logger.info(f"Purged {count} wiki Document nodes")
|
||||
|
||||
# Purge by document_id (Document Store docs)
|
||||
if document_ids:
|
||||
query = f"""
|
||||
MATCH (d:{user_doc_label}:Document)
|
||||
WHERE d.document_id IN $document_ids
|
||||
DETACH DELETE d
|
||||
RETURN count(d) as purged_count
|
||||
"""
|
||||
results = await self.neo4j.execute_query(query, {"document_ids": document_ids})
|
||||
count = results[0]["purged_count"] if results else 0
|
||||
total_purged += count
|
||||
logger.info(f"Purged {count} Document Store Document nodes")
|
||||
|
||||
return total_purged
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to purge stale documents: {e}", exc_info=True)
|
||||
return 0
|
||||
|
||||
async def cleanup_broken_relationships(
|
||||
self,
|
||||
user: str
|
||||
) -> int:
|
||||
"""
|
||||
Clean up broken FOUND relationships from SearchQuery nodes.
|
||||
|
||||
Removes relationships pointing to deleted documents.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
|
||||
Returns:
|
||||
Number of relationships cleaned
|
||||
"""
|
||||
query = """
|
||||
MATCH (sq:SearchQuery)-[r:FOUND]->(d)
|
||||
WHERE NOT EXISTS { (d) }
|
||||
DELETE r
|
||||
RETURN count(r) as cleaned_count
|
||||
"""
|
||||
|
||||
try:
|
||||
results = await self.neo4j.execute_query(query, {})
|
||||
cleaned_count = results[0]["cleaned_count"] if results else 0
|
||||
|
||||
if cleaned_count > 0:
|
||||
logger.info(f"Cleaned {cleaned_count} broken FOUND relationships")
|
||||
|
||||
return cleaned_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to cleanup broken relationships: {e}", exc_info=True)
|
||||
return 0
|
||||
|
||||
async def find_documents_without_vectors(
|
||||
self,
|
||||
user: str,
|
||||
vector_references: List[Dict[str, Any]]
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Find Document nodes that have no corresponding vectors.
|
||||
|
||||
Used for bidirectional orphan detection - graph nodes without vector data.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
vector_references: List of vector refs from VectorService.get_all_chunk_references()
|
||||
|
||||
Returns:
|
||||
List of orphan documents {page_id, document_id, doc_type, title}
|
||||
"""
|
||||
# Get all graph document references
|
||||
graph_docs = await self.get_all_document_references(user)
|
||||
|
||||
if not graph_docs:
|
||||
return []
|
||||
|
||||
# Build sets of IDs that have vectors
|
||||
vector_page_ids = {
|
||||
ref.get("page_id") for ref in vector_references
|
||||
if ref.get("doc_type") == "wiki" and ref.get("page_id")
|
||||
}
|
||||
vector_doc_ids = {
|
||||
ref.get("document_id") for ref in vector_references
|
||||
if ref.get("doc_type") != "wiki" and ref.get("document_id")
|
||||
}
|
||||
|
||||
# Find graph docs with no vectors
|
||||
orphans = []
|
||||
for doc in graph_docs:
|
||||
doc_type = doc.get("doc_type", "wiki")
|
||||
|
||||
if doc_type == "wiki":
|
||||
page_id = doc.get("page_id")
|
||||
if page_id and page_id not in vector_page_ids:
|
||||
orphans.append(doc)
|
||||
else:
|
||||
document_id = doc.get("document_id")
|
||||
if document_id and document_id not in vector_doc_ids:
|
||||
orphans.append(doc)
|
||||
|
||||
logger.info(f"Found {len(orphans)} graph documents without vectors for user {user}")
|
||||
return orphans
|
||||
|
||||
@@ -6,7 +6,7 @@ HybridRAG service combining vector, graph, and web search.
|
||||
1. Parallel Retrieval - Vector + Graph + Web search
|
||||
2. RRF Fusion - Merge results with Reciprocal Rank Fusion
|
||||
3. Enrichment - Add related dossiers via graph
|
||||
4. LLM Re-ranking - Re-rank with mistral-nemo
|
||||
4. LLM Re-ranking - Re-rank with configured Ollama model
|
||||
5. Context Formatting - Format for LLM consumption
|
||||
6. Persistence - Store for Librarian processing
|
||||
"""
|
||||
@@ -20,6 +20,7 @@ import logging
|
||||
|
||||
from src.services.vector_service import VectorService
|
||||
from src.services.graph_service import GraphService
|
||||
from src.services.volatile_service import VolatileCacheService
|
||||
from src.clients.searxng_client import SearXNGClient
|
||||
from src.clients.ollama_client import OllamaClient
|
||||
from src.clients.content_extractor import ContentExtractor
|
||||
@@ -46,7 +47,8 @@ class HybridRAGService:
|
||||
searxng_client: SearXNGClient,
|
||||
ollama_client: OllamaClient,
|
||||
content_extractor: ContentExtractor,
|
||||
settings: Settings
|
||||
settings: Settings,
|
||||
volatile_service: Optional[VolatileCacheService] = None
|
||||
):
|
||||
"""
|
||||
Initialize HybridRAG service.
|
||||
@@ -58,6 +60,7 @@ class HybridRAGService:
|
||||
ollama_client: Client for LLM (keyword extraction, re-ranking)
|
||||
content_extractor: Client for extracting full content from URLs
|
||||
settings: Application settings
|
||||
volatile_service: Service for volatile cache search (optional)
|
||||
"""
|
||||
self.vector = vector_service
|
||||
self.graph = graph_service
|
||||
@@ -65,7 +68,8 @@ class HybridRAGService:
|
||||
self.ollama = ollama_client
|
||||
self.content_extractor = content_extractor
|
||||
self.settings = settings
|
||||
self.reranker_model = settings.reranker_model
|
||||
self.volatile = volatile_service
|
||||
self.reranker_model = settings.ollama_model
|
||||
|
||||
async def search(
|
||||
self,
|
||||
@@ -104,15 +108,24 @@ class HybridRAGService:
|
||||
timing["vector_ms"] = raw_results.get("timing", {}).get("vector_ms", 0)
|
||||
timing["graph_ms"] = raw_results.get("timing", {}).get("graph_ms", 0)
|
||||
timing["web_ms"] = raw_results.get("timing", {}).get("web_ms", 0)
|
||||
timing["volatile_ms"] = raw_results.get("timing", {}).get("volatile_ms", 0)
|
||||
|
||||
# Phase 2: RRF Fusion
|
||||
# Phase 2: Three-Source RRF Fusion
|
||||
phase2_start = time.time()
|
||||
|
||||
# Stage 1: Merge wiki sources (vector + graph) into single ranking
|
||||
wiki_merged = self._merge_wiki_sources(
|
||||
vector_results=raw_results.get("vector", []),
|
||||
graph_results=raw_results.get("graph", []),
|
||||
k=config.rrf_k
|
||||
)
|
||||
|
||||
# Stage 2: Final RRF between wiki, volatile, and web
|
||||
# Volatile gets priority boost (smaller k = higher contribution per rank)
|
||||
fused_results = self._reciprocal_rank_fusion(
|
||||
results_by_source={
|
||||
"vector": raw_results.get("vector", []),
|
||||
"graph": raw_results.get("graph", []),
|
||||
"web": raw_results.get("web", [])
|
||||
},
|
||||
wiki_results=wiki_merged,
|
||||
web_results=raw_results.get("web", []),
|
||||
volatile_results=raw_results.get("volatile", []),
|
||||
k=config.rrf_k
|
||||
)
|
||||
timing["fusion_ms"] = (time.time() - phase2_start) * 1000
|
||||
@@ -189,25 +202,21 @@ class HybridRAGService:
|
||||
Returns:
|
||||
Dictionary with keywords, entities, synonyms, expansions
|
||||
"""
|
||||
prompt = f"""Extract search terms from this query. For each important word, provide synonyms and expansions.
|
||||
prompt = f"""Extract search terms from this query.
|
||||
|
||||
Query: "{query}"
|
||||
|
||||
Return ONLY valid JSON:
|
||||
{{
|
||||
"core_keywords": ["key", "words", "from", "query"],
|
||||
"synonyms": {{
|
||||
"word": ["alternative", "terms"]
|
||||
}}
|
||||
}}
|
||||
RULES:
|
||||
- Extract ONLY keywords explicitly present or directly implied in the query
|
||||
- Do NOT invent terms, concepts, or synonyms not clearly related
|
||||
- Do NOT add general knowledge or associations
|
||||
- Provide synonyms ONLY for technical terms with well-known alternatives
|
||||
- Return valid JSON only, no commentary
|
||||
|
||||
Example for "Docker container hosting":
|
||||
Return format:
|
||||
{{
|
||||
"core_keywords": ["docker", "container", "hosting"],
|
||||
"synonyms": {{
|
||||
"docker": ["containerization", "container runtime"],
|
||||
"hosting": ["server", "infrastructure"]
|
||||
}}
|
||||
"core_keywords": ["words", "from", "query"],
|
||||
"synonyms": {{"term": ["direct", "alternatives"]}}
|
||||
}}
|
||||
|
||||
JSON:"""
|
||||
@@ -215,7 +224,8 @@ JSON:"""
|
||||
try:
|
||||
response = await self.ollama.generate_text(
|
||||
prompt=prompt,
|
||||
model=self.reranker_model
|
||||
model=self.reranker_model,
|
||||
temperature=0.0 # Deterministic for consistent extraction
|
||||
)
|
||||
|
||||
# Parse JSON response (handle potential extra text)
|
||||
@@ -288,7 +298,8 @@ JSON:"""
|
||||
response = await self.vector.search(
|
||||
query=query,
|
||||
user=user,
|
||||
limit=config.vector_limit
|
||||
limit=config.vector_limit,
|
||||
score_threshold=self.settings.vector_similarity_threshold
|
||||
)
|
||||
results = [
|
||||
{
|
||||
@@ -313,11 +324,19 @@ JSON:"""
|
||||
async def graph_search():
|
||||
start = time.time()
|
||||
try:
|
||||
# Skip synonyms for graph search - only use core keywords
|
||||
# Synonyms like "author" can match unrelated entities like "author2000"
|
||||
graph_keywords = {
|
||||
"core_keywords": keywords_data.get("core_keywords", []),
|
||||
"entities": keywords_data.get("entities", []),
|
||||
"synonyms": {}, # No synonyms for exact entity matching
|
||||
"expansions": {}
|
||||
}
|
||||
results = await self.graph.search_documents(
|
||||
query=query,
|
||||
user=user,
|
||||
limit=config.graph_limit,
|
||||
keywords_data=keywords_data
|
||||
keywords_data=graph_keywords
|
||||
)
|
||||
formatted = [
|
||||
{
|
||||
@@ -377,6 +396,37 @@ JSON:"""
|
||||
|
||||
tasks["web"] = web_search()
|
||||
|
||||
# Volatile cache search
|
||||
if config.enable_volatile and self.volatile:
|
||||
async def volatile_search():
|
||||
start = time.time()
|
||||
try:
|
||||
results = await self.volatile.search(
|
||||
user=user,
|
||||
query=query,
|
||||
limit=config.volatile_limit,
|
||||
score_threshold=config.volatile_threshold
|
||||
)
|
||||
formatted = [
|
||||
{
|
||||
"key": r.key,
|
||||
"namespace": r.namespace,
|
||||
"title": f"{r.namespace}: {r.key}",
|
||||
"content": r.data.get("text", "") if isinstance(r.data, dict) else str(r.data),
|
||||
"raw_data": r.data,
|
||||
"source_api": r.source,
|
||||
"ttl_remaining": r.ttl_remaining,
|
||||
"source": "volatile"
|
||||
}
|
||||
for r in results
|
||||
]
|
||||
return formatted, (time.time() - start) * 1000
|
||||
except Exception as e:
|
||||
logger.error(f"Volatile search failed: {e}", exc_info=True)
|
||||
return [], (time.time() - start) * 1000
|
||||
|
||||
tasks["volatile"] = volatile_search()
|
||||
|
||||
# Execute all searches in parallel
|
||||
results_dict = await asyncio.gather(*tasks.values())
|
||||
|
||||
@@ -389,57 +439,159 @@ JSON:"""
|
||||
|
||||
logger.info(
|
||||
f"Parallel retrieval: vector={len(output.get('vector', []))}, "
|
||||
f"graph={len(output.get('graph', []))}, web={len(output.get('web', []))}"
|
||||
f"graph={len(output.get('graph', []))}, web={len(output.get('web', []))}, "
|
||||
f"volatile={len(output.get('volatile', []))}"
|
||||
)
|
||||
|
||||
return output
|
||||
|
||||
def _reciprocal_rank_fusion(
|
||||
def _merge_wiki_sources(
|
||||
self,
|
||||
results_by_source: Dict[str, List],
|
||||
vector_results: List[Dict],
|
||||
graph_results: List[Dict],
|
||||
k: int = 60
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Phase 2: Merge results using Reciprocal Rank Fusion.
|
||||
Stage 1: Merge vector and graph into single wiki ranking using RRF.
|
||||
|
||||
RRF formula: score = sum(1 / (k + rank)) for each source
|
||||
Both sources search the same wiki pool, so we combine them before
|
||||
final RRF with web to avoid double-counting wiki pages.
|
||||
|
||||
Args:
|
||||
results_by_source: Results from each source
|
||||
vector_results: Results from vector search
|
||||
graph_results: Results from graph search
|
||||
k: RRF constant (default 60)
|
||||
|
||||
Returns:
|
||||
Merged and sorted results
|
||||
Merged wiki results sorted by wiki RRF score
|
||||
"""
|
||||
wiki_scores = {}
|
||||
|
||||
# Process vector results
|
||||
for rank, result in enumerate(vector_results, start=1):
|
||||
page_id = result.get("page_id")
|
||||
if not page_id:
|
||||
continue
|
||||
result_id = f"page_{page_id}"
|
||||
|
||||
if result_id not in wiki_scores:
|
||||
wiki_scores[result_id] = {
|
||||
"result": dict(result), # Copy to avoid mutation
|
||||
"wiki_rrf_score": 0.0,
|
||||
"found_by": []
|
||||
}
|
||||
|
||||
wiki_scores[result_id]["wiki_rrf_score"] += 1 / (k + rank)
|
||||
wiki_scores[result_id]["found_by"].append("vector")
|
||||
|
||||
# Process graph results
|
||||
for rank, result in enumerate(graph_results, start=1):
|
||||
page_id = result.get("page_id")
|
||||
if not page_id:
|
||||
continue
|
||||
result_id = f"page_{page_id}"
|
||||
|
||||
if result_id not in wiki_scores:
|
||||
wiki_scores[result_id] = {
|
||||
"result": dict(result),
|
||||
"wiki_rrf_score": 0.0,
|
||||
"found_by": []
|
||||
}
|
||||
|
||||
wiki_scores[result_id]["wiki_rrf_score"] += 1 / (k + rank)
|
||||
wiki_scores[result_id]["found_by"].append("graph")
|
||||
|
||||
# Add graph metadata to existing result
|
||||
wiki_scores[result_id]["result"]["entity_matches"] = result.get("entity_matches")
|
||||
wiki_scores[result_id]["result"]["matched_entities"] = result.get("matched_entities")
|
||||
|
||||
# Sort by wiki RRF score
|
||||
sorted_wiki = sorted(
|
||||
wiki_scores.values(),
|
||||
key=lambda x: x["wiki_rrf_score"],
|
||||
reverse=True
|
||||
)
|
||||
|
||||
# Return merged results with wiki ranking
|
||||
merged = []
|
||||
for wiki_rank, item in enumerate(sorted_wiki, start=1):
|
||||
merged.append({
|
||||
**item["result"],
|
||||
"wiki_rank": wiki_rank,
|
||||
"wiki_rrf_score": item["wiki_rrf_score"],
|
||||
"found_by": item["found_by"],
|
||||
"source": "wiki"
|
||||
})
|
||||
|
||||
logger.info(f"Wiki merge: {len(merged)} unique pages from vector+graph")
|
||||
return merged
|
||||
|
||||
def _reciprocal_rank_fusion(
|
||||
self,
|
||||
wiki_results: List[Dict],
|
||||
web_results: List[Dict],
|
||||
volatile_results: Optional[List[Dict]] = None,
|
||||
k: int = 60
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Stage 2: Final RRF between wiki, volatile, and web.
|
||||
|
||||
Wiki results are pre-merged from vector+graph. Volatile results
|
||||
get a priority boost (smaller effective k) since they represent
|
||||
current, time-sensitive information.
|
||||
|
||||
Args:
|
||||
wiki_results: Pre-merged wiki results from _merge_wiki_sources()
|
||||
web_results: Results from web search
|
||||
volatile_results: Results from volatile cache (fresh data)
|
||||
k: RRF constant (default 60)
|
||||
|
||||
Returns:
|
||||
Final merged and sorted results
|
||||
"""
|
||||
rrf_scores = {}
|
||||
volatile_results = volatile_results or []
|
||||
|
||||
for source, results in results_by_source.items():
|
||||
for rank, result in enumerate(results, start=1):
|
||||
# Use page_id for wiki results, url hash for web results
|
||||
if result.get("page_id"):
|
||||
result_id = f"page_{result['page_id']}"
|
||||
elif result.get("url"):
|
||||
result_id = f"url_{hash(result['url'])}"
|
||||
else:
|
||||
continue # Skip results without ID
|
||||
# Volatile results get priority boost (k/2 = stronger score per rank)
|
||||
volatile_k = k // 2
|
||||
for rank, result in enumerate(volatile_results, start=1):
|
||||
key = result.get("key")
|
||||
namespace = result.get("namespace", "unknown")
|
||||
if not key:
|
||||
continue
|
||||
result_id = f"volatile_{namespace}_{key}"
|
||||
rrf_scores[result_id] = {
|
||||
"result": result,
|
||||
"rrf_score": 1 / (volatile_k + rank), # Priority boost
|
||||
"sources": ["volatile"],
|
||||
"source_type": "volatile"
|
||||
}
|
||||
|
||||
if result_id not in rrf_scores:
|
||||
rrf_scores[result_id] = {
|
||||
"result": result,
|
||||
"rrf_score": 0.0,
|
||||
"sources": [],
|
||||
"source_type": source
|
||||
}
|
||||
# Wiki results (single source, already merged)
|
||||
for rank, result in enumerate(wiki_results, start=1):
|
||||
page_id = result.get("page_id")
|
||||
if not page_id:
|
||||
continue
|
||||
result_id = f"page_{page_id}"
|
||||
rrf_scores[result_id] = {
|
||||
"result": result,
|
||||
"rrf_score": 1 / (k + rank),
|
||||
"sources": result.get("found_by", ["wiki"]),
|
||||
"source_type": "wiki"
|
||||
}
|
||||
|
||||
# RRF formula: sum of 1/(k + rank) across sources
|
||||
rrf_scores[result_id]["rrf_score"] += 1 / (k + rank)
|
||||
rrf_scores[result_id]["sources"].append(source)
|
||||
|
||||
# If result appears in multiple sources, update source_type
|
||||
if len(rrf_scores[result_id]["sources"]) > 1:
|
||||
rrf_scores[result_id]["source_type"] = "+".join(
|
||||
sorted(set(rrf_scores[result_id]["sources"]))
|
||||
)
|
||||
# Web results (single source)
|
||||
for rank, result in enumerate(web_results, start=1):
|
||||
url = result.get("url")
|
||||
if not url:
|
||||
continue
|
||||
result_id = f"url_{hash(url)}"
|
||||
rrf_scores[result_id] = {
|
||||
"result": result,
|
||||
"rrf_score": 1 / (k + rank),
|
||||
"sources": ["web"],
|
||||
"source_type": "web"
|
||||
}
|
||||
|
||||
# Sort by RRF score descending
|
||||
sorted_results = sorted(
|
||||
@@ -448,7 +600,8 @@ JSON:"""
|
||||
reverse=True
|
||||
)
|
||||
|
||||
logger.info(f"RRF fusion: {len(sorted_results)} unique results from {len(results_by_source)} sources")
|
||||
volatile_count = len([r for r in sorted_results if r["source_type"] == "volatile"])
|
||||
logger.info(f"Final RRF: {len(sorted_results)} results (wiki + volatile[{volatile_count}] + web)")
|
||||
|
||||
return sorted_results
|
||||
|
||||
@@ -526,21 +679,27 @@ JSON:"""
|
||||
for i, r in enumerate(results)
|
||||
])
|
||||
|
||||
prompt = f"""Given this search query and documents, rank them by relevance.
|
||||
prompt = f"""Rank these documents by relevance to the query.
|
||||
|
||||
Query: {query}
|
||||
|
||||
Documents:
|
||||
{docs_text}
|
||||
|
||||
Return only the numbers in order of relevance (most relevant first).
|
||||
Example: 3,1,5,2,4
|
||||
RULES:
|
||||
- Rank ONLY by how well content answers the query
|
||||
- Do NOT consider document length, formatting, or style
|
||||
- Do NOT add explanation or commentary
|
||||
- Return ONLY comma-separated numbers, most relevant first
|
||||
|
||||
Example output: 3,1,5,2,4
|
||||
|
||||
Ranking:"""
|
||||
|
||||
response = await self.ollama.generate_text(
|
||||
prompt=prompt,
|
||||
model=self.reranker_model
|
||||
model=self.reranker_model,
|
||||
temperature=0.0 # Deterministic for consistent rankings
|
||||
)
|
||||
|
||||
# Parse response: "3,1,5,2,4" → [2, 0, 4, 1, 3] (0-indexed)
|
||||
|
||||
@@ -356,3 +356,189 @@ class VectorService:
|
||||
collections=[],
|
||||
total=0
|
||||
)
|
||||
|
||||
# ========== Cleanup Methods ==========
|
||||
|
||||
async def delete_document_chunks(
|
||||
self,
|
||||
document_id: str,
|
||||
user: str
|
||||
) -> int:
|
||||
"""
|
||||
Delete all chunks for a document (Document Store).
|
||||
|
||||
Args:
|
||||
document_id: Document UUID
|
||||
user: User identifier
|
||||
|
||||
Returns:
|
||||
Number of chunks deleted
|
||||
"""
|
||||
collection_name = get_qdrant_collection_name(user)
|
||||
|
||||
try:
|
||||
deleted_count = await self.qdrant.delete_by_filter(
|
||||
collection_name=collection_name,
|
||||
filter_conditions={"document_id": document_id}
|
||||
)
|
||||
|
||||
logger.info(f"Deleted chunks for document {document_id}")
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to delete chunks for document {document_id}: {e}", exc_info=True)
|
||||
return 0
|
||||
|
||||
async def delete_collection_chunks(
|
||||
self,
|
||||
collection_id: str,
|
||||
user: str
|
||||
) -> int:
|
||||
"""
|
||||
Delete all chunks for a document collection.
|
||||
|
||||
Args:
|
||||
collection_id: Collection UUID
|
||||
user: User identifier
|
||||
|
||||
Returns:
|
||||
Number of chunks deleted
|
||||
"""
|
||||
collection_name = get_qdrant_collection_name(user)
|
||||
|
||||
try:
|
||||
deleted_count = await self.qdrant.delete_by_filter(
|
||||
collection_name=collection_name,
|
||||
filter_conditions={"collection_id": collection_id}
|
||||
)
|
||||
|
||||
logger.info(f"Deleted chunks for collection {collection_id}")
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to delete chunks for collection {collection_id}: {e}", exc_info=True)
|
||||
return 0
|
||||
|
||||
async def get_all_chunk_references(
|
||||
self,
|
||||
user: str
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Get all chunk references for orphan detection.
|
||||
|
||||
Returns list of {id, page_id, document_id} for all chunks.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
|
||||
Returns:
|
||||
List of chunk references
|
||||
"""
|
||||
collection_name = get_qdrant_collection_name(user)
|
||||
|
||||
try:
|
||||
# Check if collection exists
|
||||
exists = await self.qdrant.collection_exists(collection_name)
|
||||
if not exists:
|
||||
return []
|
||||
|
||||
all_points = await self.qdrant.scroll_all_points(
|
||||
collection_name=collection_name,
|
||||
batch_size=100,
|
||||
with_payload=True
|
||||
)
|
||||
|
||||
references = []
|
||||
for point in all_points:
|
||||
payload = point.get("payload", {})
|
||||
references.append({
|
||||
"chunk_id": point["id"],
|
||||
"page_id": payload.get("page_id"),
|
||||
"document_id": payload.get("document_id"),
|
||||
"collection_id": payload.get("collection_id"),
|
||||
"doc_type": payload.get("doc_type", "wiki")
|
||||
})
|
||||
|
||||
logger.info(f"Found {len(references)} chunks for user {user}")
|
||||
return references
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get chunk references: {e}", exc_info=True)
|
||||
return []
|
||||
|
||||
async def purge_chunks_by_ids(
|
||||
self,
|
||||
user: str,
|
||||
chunk_ids: List[str]
|
||||
) -> int:
|
||||
"""
|
||||
Delete specific chunks by their IDs.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
chunk_ids: List of chunk IDs to delete
|
||||
|
||||
Returns:
|
||||
Number of chunks deleted
|
||||
"""
|
||||
if not chunk_ids:
|
||||
return 0
|
||||
|
||||
collection_name = get_qdrant_collection_name(user)
|
||||
|
||||
try:
|
||||
deleted_count = await self.qdrant.delete_by_ids(
|
||||
collection_name=collection_name,
|
||||
point_ids=chunk_ids
|
||||
)
|
||||
|
||||
logger.info(f"Purged {deleted_count} orphan chunks for user {user}")
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to purge chunks: {e}", exc_info=True)
|
||||
return 0
|
||||
|
||||
def find_chunks_without_graph_nodes(
|
||||
self,
|
||||
chunk_references: List[Dict[str, Any]],
|
||||
graph_references: List[Dict[str, Any]]
|
||||
) -> List[str]:
|
||||
"""
|
||||
Find vector chunks that have no corresponding graph Document node.
|
||||
|
||||
Used for bidirectional orphan detection - vectors without graph representation.
|
||||
|
||||
Args:
|
||||
chunk_references: List from get_all_chunk_references()
|
||||
graph_references: List from GraphService.get_all_document_references()
|
||||
|
||||
Returns:
|
||||
List of orphan chunk IDs
|
||||
"""
|
||||
# Build sets of IDs that have graph nodes
|
||||
graph_page_ids = {
|
||||
ref.get("page_id") for ref in graph_references
|
||||
if ref.get("doc_type") == "wiki" and ref.get("page_id")
|
||||
}
|
||||
graph_doc_ids = {
|
||||
ref.get("document_id") for ref in graph_references
|
||||
if ref.get("doc_type") != "wiki" and ref.get("document_id")
|
||||
}
|
||||
|
||||
# Find chunks with no graph node
|
||||
orphan_ids = []
|
||||
for chunk in chunk_references:
|
||||
doc_type = chunk.get("doc_type", "wiki")
|
||||
|
||||
if doc_type == "wiki":
|
||||
page_id = chunk.get("page_id")
|
||||
if page_id and page_id not in graph_page_ids:
|
||||
orphan_ids.append(chunk["chunk_id"])
|
||||
else:
|
||||
document_id = chunk.get("document_id")
|
||||
if document_id and document_id not in graph_doc_ids:
|
||||
orphan_ids.append(chunk["chunk_id"])
|
||||
|
||||
logger.info(f"Found {len(orphan_ids)} vector chunks without graph nodes")
|
||||
return orphan_ids
|
||||
|
||||
@@ -0,0 +1,564 @@
|
||||
"""
|
||||
Volatile Cache service for Library Desk.
|
||||
|
||||
Provides ephemeral data storage with TTL using Qdrant vectors:
|
||||
- Weather, news, financial data
|
||||
- Transit schedules, traffic conditions
|
||||
- System status, social notifications
|
||||
|
||||
Data is stored as embedded vectors for semantic search retrieval.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import time
|
||||
from datetime import datetime
|
||||
from typing import List, Optional, Dict, Any
|
||||
|
||||
from src.clients.qdrant_client import QdrantClientWrapper
|
||||
from src.clients.ollama_client import OllamaClient
|
||||
from src.config import Settings
|
||||
from src.models.volatile import (
|
||||
VolatileRecordResponse,
|
||||
VolatileNamespace,
|
||||
NAMESPACE_DEFAULT_TTL,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class VolatileCacheService:
|
||||
"""
|
||||
Service for volatile data with TTL stored in Qdrant.
|
||||
|
||||
Stores ephemeral data as vectors for semantic search retrieval.
|
||||
Each user has an isolated volatile collection.
|
||||
"""
|
||||
|
||||
COLLECTION_PREFIX = "volatile_"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
qdrant_client: QdrantClientWrapper,
|
||||
ollama_client: OllamaClient,
|
||||
settings: Settings
|
||||
):
|
||||
"""
|
||||
Initialize volatile cache service.
|
||||
|
||||
Args:
|
||||
qdrant_client: Qdrant client for vector storage
|
||||
ollama_client: Ollama client for embeddings
|
||||
settings: Application settings
|
||||
"""
|
||||
self.qdrant = qdrant_client
|
||||
self.ollama = ollama_client
|
||||
self.settings = settings
|
||||
|
||||
logger.info("Initialized VolatileCacheService (Qdrant backend)")
|
||||
|
||||
def _collection_name(self, user: str) -> str:
|
||||
"""Get volatile collection name for user."""
|
||||
return f"{self.COLLECTION_PREFIX}{user}"
|
||||
|
||||
def _make_vector_id(self, namespace: str, key: str) -> str:
|
||||
"""
|
||||
Generate deterministic vector ID for namespace/key.
|
||||
|
||||
Same namespace+key always produces same ID for upsert behavior.
|
||||
"""
|
||||
combined = f"{namespace}:{key}"
|
||||
return hashlib.md5(combined.encode()).hexdigest()
|
||||
|
||||
def _get_default_ttl(self, namespace: str) -> int:
|
||||
"""Get default TTL for a namespace."""
|
||||
try:
|
||||
ns = VolatileNamespace(namespace)
|
||||
return NAMESPACE_DEFAULT_TTL.get(ns, self.settings.volatile_default_ttl)
|
||||
except ValueError:
|
||||
return self.settings.volatile_default_ttl
|
||||
|
||||
def _current_timestamp_ms(self) -> int:
|
||||
"""Get current timestamp in milliseconds."""
|
||||
return int(time.time() * 1000)
|
||||
|
||||
def _to_natural_language(
|
||||
self,
|
||||
namespace: str,
|
||||
key: str,
|
||||
data: Dict[str, Any]
|
||||
) -> str:
|
||||
"""
|
||||
Convert structured data to natural language for embedding.
|
||||
|
||||
This creates a text representation that embeds well semantically.
|
||||
"""
|
||||
# Template-based conversion for known namespaces
|
||||
if namespace == VolatileNamespace.WEATHER:
|
||||
temp = data.get("temperature", data.get("temp", "unknown"))
|
||||
conditions = data.get("conditions", data.get("weather", ""))
|
||||
humidity = data.get("humidity", "")
|
||||
text = f"Current weather in {key}: {temp}°C"
|
||||
if conditions:
|
||||
text += f", {conditions}"
|
||||
if humidity:
|
||||
text += f", humidity {humidity}%"
|
||||
return text
|
||||
|
||||
elif namespace == VolatileNamespace.NEWS:
|
||||
title = data.get("title", data.get("headline", ""))
|
||||
summary = data.get("summary", data.get("description", ""))
|
||||
source = data.get("source", "")
|
||||
text = f"News: {title}"
|
||||
if summary:
|
||||
text += f". {summary}"
|
||||
if source:
|
||||
text += f" (Source: {source})"
|
||||
return text
|
||||
|
||||
elif namespace == VolatileNamespace.FINANCIAL:
|
||||
symbol = data.get("symbol", key)
|
||||
price = data.get("price", "")
|
||||
change = data.get("change", data.get("change_percent", ""))
|
||||
text = f"Financial data for {symbol}"
|
||||
if price:
|
||||
text += f": price {price}"
|
||||
if change:
|
||||
text += f", change {change}%"
|
||||
return text
|
||||
|
||||
elif namespace == VolatileNamespace.TRANSIT:
|
||||
route = data.get("route", data.get("line", key))
|
||||
status = data.get("status", "")
|
||||
delay = data.get("delay", data.get("delay_minutes", ""))
|
||||
text = f"Transit {route}"
|
||||
if status:
|
||||
text += f": {status}"
|
||||
if delay:
|
||||
text += f", delay {delay} minutes"
|
||||
return text
|
||||
|
||||
elif namespace == VolatileNamespace.TRAFFIC:
|
||||
location = data.get("location", key)
|
||||
duration = data.get("duration", data.get("travel_time", ""))
|
||||
congestion = data.get("congestion", "")
|
||||
text = f"Traffic for {location}"
|
||||
if duration:
|
||||
text += f": {duration} minutes"
|
||||
if congestion:
|
||||
text += f", congestion level {congestion}"
|
||||
return text
|
||||
|
||||
elif namespace == VolatileNamespace.AIR_QUALITY:
|
||||
location = data.get("location", key)
|
||||
aqi = data.get("aqi", data.get("index", ""))
|
||||
quality = data.get("quality", "")
|
||||
text = f"Air quality in {location}"
|
||||
if aqi:
|
||||
text += f": AQI {aqi}"
|
||||
if quality:
|
||||
text += f" ({quality})"
|
||||
return text
|
||||
|
||||
elif namespace == VolatileNamespace.SPORTS:
|
||||
event = data.get("event", data.get("match", key))
|
||||
score = data.get("score", "")
|
||||
status = data.get("status", "")
|
||||
text = f"Sports: {event}"
|
||||
if score:
|
||||
text += f" - Score: {score}"
|
||||
if status:
|
||||
text += f" ({status})"
|
||||
return text
|
||||
|
||||
elif namespace == VolatileNamespace.SYSTEM:
|
||||
service = data.get("service", key)
|
||||
status = data.get("status", "unknown")
|
||||
message = data.get("message", "")
|
||||
text = f"System status for {service}: {status}"
|
||||
if message:
|
||||
text += f". {message}"
|
||||
return text
|
||||
|
||||
# Fallback: serialize key fields
|
||||
text_parts = [f"{namespace} data for {key}:"]
|
||||
for k, v in data.items():
|
||||
if isinstance(v, (str, int, float, bool)):
|
||||
text_parts.append(f"{k}: {v}")
|
||||
return " ".join(text_parts)
|
||||
|
||||
async def store(
|
||||
self,
|
||||
user: str,
|
||||
namespace: str,
|
||||
key: str,
|
||||
data: Dict[str, Any],
|
||||
source: Optional[str] = None,
|
||||
ttl: Optional[int] = None,
|
||||
refresh_schedule: Optional[str] = None
|
||||
) -> VolatileRecordResponse:
|
||||
"""
|
||||
Store volatile data as an embedded vector.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
namespace: Data namespace (from controlled list)
|
||||
key: Record key (normalized slug)
|
||||
data: Structured data to store
|
||||
source: Origin API/service
|
||||
ttl: TTL in seconds (uses namespace default if not set)
|
||||
refresh_schedule: Optional cron expression for refresh
|
||||
|
||||
Returns:
|
||||
The stored record
|
||||
"""
|
||||
collection = self._collection_name(user)
|
||||
|
||||
# Ensure collection exists
|
||||
await self.qdrant.ensure_collection(collection)
|
||||
|
||||
# Calculate TTL and expiry
|
||||
effective_ttl = ttl if ttl is not None else self._get_default_ttl(namespace)
|
||||
now_ms = self._current_timestamp_ms()
|
||||
expiry_ms = now_ms + (effective_ttl * 1000)
|
||||
|
||||
# Convert to natural language for embedding
|
||||
text = self._to_natural_language(namespace, key, data)
|
||||
|
||||
# Generate embedding
|
||||
embedding = await self.ollama.embed(text)
|
||||
if not embedding:
|
||||
raise ValueError("Failed to generate embedding for volatile data")
|
||||
|
||||
# Build payload
|
||||
now = datetime.utcnow()
|
||||
payload = {
|
||||
"doc_type": "volatile",
|
||||
"namespace": namespace,
|
||||
"key": key,
|
||||
"text": text,
|
||||
"raw_data": data,
|
||||
"source": source,
|
||||
"created_at": now.isoformat(),
|
||||
"updated_at": now.isoformat(),
|
||||
"ttl": effective_ttl,
|
||||
"ttl_expiry": expiry_ms,
|
||||
"refresh_schedule": refresh_schedule,
|
||||
"user": user,
|
||||
}
|
||||
|
||||
# Upsert vector (same namespace+key = same ID = update)
|
||||
vector_id = self._make_vector_id(namespace, key)
|
||||
success = await self.qdrant.upsert_vector(
|
||||
collection_name=collection,
|
||||
vector_id=vector_id,
|
||||
vector=embedding,
|
||||
payload=payload
|
||||
)
|
||||
|
||||
if not success:
|
||||
raise ValueError("Failed to store volatile vector")
|
||||
|
||||
logger.debug(f"Stored volatile {namespace}:{key} with TTL {effective_ttl}s")
|
||||
|
||||
return VolatileRecordResponse(
|
||||
key=key,
|
||||
namespace=namespace,
|
||||
data=data,
|
||||
source=source,
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
ttl=effective_ttl,
|
||||
ttl_remaining=effective_ttl,
|
||||
refresh_schedule=refresh_schedule,
|
||||
user=user,
|
||||
)
|
||||
|
||||
async def search(
|
||||
self,
|
||||
user: str,
|
||||
query: str,
|
||||
limit: int = 5,
|
||||
score_threshold: float = 0.75
|
||||
) -> List[VolatileRecordResponse]:
|
||||
"""
|
||||
Semantic search across volatile data.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
query: Search query
|
||||
limit: Maximum results
|
||||
score_threshold: Minimum similarity score (higher = stricter)
|
||||
|
||||
Returns:
|
||||
List of matching volatile records
|
||||
"""
|
||||
collection = self._collection_name(user)
|
||||
|
||||
# Check if collection exists
|
||||
if not await self.qdrant.collection_exists(collection):
|
||||
return []
|
||||
|
||||
# Generate query embedding
|
||||
query_embedding = await self.ollama.embed(query)
|
||||
if not query_embedding:
|
||||
logger.error("Failed to embed query for volatile search")
|
||||
return []
|
||||
|
||||
# Search with expiry filter
|
||||
now_ms = self._current_timestamp_ms()
|
||||
results = await self.qdrant.search_with_expiry_filter(
|
||||
collection_name=collection,
|
||||
query_vector=query_embedding,
|
||||
current_timestamp=now_ms,
|
||||
limit=limit,
|
||||
score_threshold=score_threshold
|
||||
)
|
||||
|
||||
# Convert to response models
|
||||
responses = []
|
||||
for result in results:
|
||||
payload = result["payload"]
|
||||
ttl_expiry = payload.get("ttl_expiry", 0)
|
||||
ttl_remaining = max(0, (ttl_expiry - now_ms) // 1000)
|
||||
|
||||
responses.append(VolatileRecordResponse(
|
||||
key=payload["key"],
|
||||
namespace=payload["namespace"],
|
||||
data=payload.get("raw_data", {}),
|
||||
source=payload.get("source"),
|
||||
created_at=datetime.fromisoformat(payload["created_at"]),
|
||||
updated_at=datetime.fromisoformat(payload["updated_at"]),
|
||||
ttl=payload.get("ttl", 0),
|
||||
ttl_remaining=ttl_remaining,
|
||||
refresh_schedule=payload.get("refresh_schedule"),
|
||||
user=payload["user"],
|
||||
))
|
||||
|
||||
return responses
|
||||
|
||||
async def get(
|
||||
self,
|
||||
user: str,
|
||||
namespace: str,
|
||||
key: str
|
||||
) -> Optional[VolatileRecordResponse]:
|
||||
"""
|
||||
Get a specific volatile record by namespace and key.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
namespace: Data namespace
|
||||
key: Record key
|
||||
|
||||
Returns:
|
||||
Record if found and not expired, None otherwise
|
||||
"""
|
||||
# Use search with high threshold to find exact match
|
||||
query = self._to_natural_language(namespace, key, {"key": key})
|
||||
results = await self.search(user, query, limit=10, score_threshold=0.5)
|
||||
|
||||
# Find exact namespace+key match
|
||||
for result in results:
|
||||
if result.namespace == namespace and result.key == key:
|
||||
return result
|
||||
|
||||
return None
|
||||
|
||||
async def delete(
|
||||
self,
|
||||
user: str,
|
||||
namespace: str,
|
||||
key: str
|
||||
) -> bool:
|
||||
"""
|
||||
Delete a specific volatile record.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
namespace: Data namespace
|
||||
key: Record key
|
||||
|
||||
Returns:
|
||||
True if deleted, False if not found
|
||||
"""
|
||||
collection = self._collection_name(user)
|
||||
|
||||
if not await self.qdrant.collection_exists(collection):
|
||||
return False
|
||||
|
||||
vector_id = self._make_vector_id(namespace, key)
|
||||
|
||||
try:
|
||||
deleted = await self.qdrant.delete_by_ids(
|
||||
collection_name=collection,
|
||||
point_ids=[vector_id]
|
||||
)
|
||||
return deleted > 0
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to delete volatile {namespace}:{key}: {e}")
|
||||
return False
|
||||
|
||||
async def get_scheduled(
|
||||
self,
|
||||
user: str
|
||||
) -> List[VolatileRecordResponse]:
|
||||
"""
|
||||
Get all records with refresh schedules.
|
||||
|
||||
Used by scheduler to determine what needs refreshing.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
|
||||
Returns:
|
||||
List of records with refresh_schedule set
|
||||
"""
|
||||
collection = self._collection_name(user)
|
||||
|
||||
if not await self.qdrant.collection_exists(collection):
|
||||
return []
|
||||
|
||||
now_ms = self._current_timestamp_ms()
|
||||
scheduled = []
|
||||
|
||||
# Scroll through all non-expired records
|
||||
try:
|
||||
all_points = await self.qdrant.scroll_all_points(
|
||||
collection_name=collection,
|
||||
with_payload=True
|
||||
)
|
||||
|
||||
for point in all_points:
|
||||
payload = point.get("payload", {})
|
||||
ttl_expiry = payload.get("ttl_expiry", 0)
|
||||
|
||||
# Skip expired
|
||||
if ttl_expiry <= now_ms:
|
||||
continue
|
||||
|
||||
# Only include if has refresh schedule
|
||||
if payload.get("refresh_schedule"):
|
||||
ttl_remaining = max(0, (ttl_expiry - now_ms) // 1000)
|
||||
scheduled.append(VolatileRecordResponse(
|
||||
key=payload["key"],
|
||||
namespace=payload["namespace"],
|
||||
data=payload.get("raw_data", {}),
|
||||
source=payload.get("source"),
|
||||
created_at=datetime.fromisoformat(payload["created_at"]),
|
||||
updated_at=datetime.fromisoformat(payload["updated_at"]),
|
||||
ttl=payload.get("ttl", 0),
|
||||
ttl_remaining=ttl_remaining,
|
||||
refresh_schedule=payload["refresh_schedule"],
|
||||
user=payload["user"],
|
||||
))
|
||||
|
||||
return scheduled
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get scheduled volatile records: {e}")
|
||||
return []
|
||||
|
||||
async def get_stats(
|
||||
self,
|
||||
user: str
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Get cache statistics for user.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
|
||||
Returns:
|
||||
Statistics dict
|
||||
"""
|
||||
collection = self._collection_name(user)
|
||||
|
||||
if not await self.qdrant.collection_exists(collection):
|
||||
return {
|
||||
"total_records": 0,
|
||||
"by_namespace": {},
|
||||
"scheduled_count": 0,
|
||||
"expired_count": 0,
|
||||
}
|
||||
|
||||
now_ms = self._current_timestamp_ms()
|
||||
by_namespace: Dict[str, int] = {}
|
||||
total = 0
|
||||
scheduled = 0
|
||||
expired = 0
|
||||
|
||||
try:
|
||||
all_points = await self.qdrant.scroll_all_points(
|
||||
collection_name=collection,
|
||||
with_payload=True
|
||||
)
|
||||
|
||||
for point in all_points:
|
||||
payload = point.get("payload", {})
|
||||
namespace = payload.get("namespace", "unknown")
|
||||
ttl_expiry = payload.get("ttl_expiry", 0)
|
||||
|
||||
if ttl_expiry <= now_ms:
|
||||
expired += 1
|
||||
else:
|
||||
total += 1
|
||||
by_namespace[namespace] = by_namespace.get(namespace, 0) + 1
|
||||
if payload.get("refresh_schedule"):
|
||||
scheduled += 1
|
||||
|
||||
return {
|
||||
"total_records": total,
|
||||
"by_namespace": by_namespace,
|
||||
"scheduled_count": scheduled,
|
||||
"expired_count": expired,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get volatile stats: {e}")
|
||||
return {
|
||||
"total_records": 0,
|
||||
"by_namespace": {},
|
||||
"scheduled_count": 0,
|
||||
"expired_count": 0,
|
||||
}
|
||||
|
||||
async def purge_expired(
|
||||
self,
|
||||
user: str
|
||||
) -> int:
|
||||
"""
|
||||
Purge all expired volatile records for user.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
|
||||
Returns:
|
||||
Number of records purged
|
||||
"""
|
||||
collection = self._collection_name(user)
|
||||
|
||||
if not await self.qdrant.collection_exists(collection):
|
||||
return 0
|
||||
|
||||
now_ms = self._current_timestamp_ms()
|
||||
return await self.qdrant.delete_expired_vectors(collection, now_ms)
|
||||
|
||||
async def purge_all_expired(self) -> Dict[str, int]:
|
||||
"""
|
||||
Purge expired records from all volatile collections.
|
||||
|
||||
Returns:
|
||||
Dict of collection -> purged count
|
||||
"""
|
||||
collections = await self.qdrant.get_volatile_collections()
|
||||
results = {}
|
||||
now_ms = self._current_timestamp_ms()
|
||||
|
||||
for collection in collections:
|
||||
purged = await self.qdrant.delete_expired_vectors(collection, now_ms)
|
||||
if purged > 0:
|
||||
results[collection] = purged
|
||||
logger.info(f"Purged {purged} expired from {collection}")
|
||||
|
||||
return results
|
||||
@@ -1,7 +1,7 @@
|
||||
"""
|
||||
Intelligent Wiki Page Writer Service
|
||||
|
||||
Uses LLM (mistral-nemo) to create and reconstruct wiki pages with:
|
||||
Uses LLM to create and reconstruct wiki pages with:
|
||||
- Holistic content restructuring
|
||||
- Zero fact loss (unless superseded)
|
||||
- Conflict detection and flagging
|
||||
@@ -25,15 +25,16 @@ class WikiPageWriter:
|
||||
Intelligent wiki page writer using LLM for content generation and restructuring.
|
||||
"""
|
||||
|
||||
def __init__(self, ollama_client):
|
||||
def __init__(self, ollama_client, settings):
|
||||
"""
|
||||
Initialize wiki page writer.
|
||||
|
||||
Args:
|
||||
ollama_client: OllamaClient for LLM operations
|
||||
settings: Application settings
|
||||
"""
|
||||
self.ollama = ollama_client
|
||||
self.model = "mistral-nemo" # Default model for writing
|
||||
self.model = settings.ollama_model
|
||||
|
||||
async def create_page(
|
||||
self,
|
||||
@@ -130,8 +131,8 @@ class WikiPageWriter:
|
||||
conflicts=conflicts
|
||||
)
|
||||
|
||||
# Reconstruct with LLM
|
||||
reconstructed = await self._call_llm(prompt)
|
||||
# Reconstruct with LLM (lower temperature for precise merging)
|
||||
reconstructed = await self._call_llm(prompt, temperature=0.2)
|
||||
|
||||
# Ensure standard sections are present
|
||||
reconstructed = self._ensure_standard_sections(
|
||||
@@ -154,7 +155,7 @@ class WikiPageWriter:
|
||||
Returns:
|
||||
List of conflicts with: {fact_a, fact_b, confidence, context}
|
||||
"""
|
||||
prompt = f"""Analyze these two pieces of content for factual conflicts.
|
||||
prompt = f"""Analyze these contents for direct factual conflicts.
|
||||
|
||||
EXISTING CONTENT:
|
||||
{existing_content[:2000]}
|
||||
@@ -162,25 +163,26 @@ EXISTING CONTENT:
|
||||
NEW INFORMATION:
|
||||
{new_information[:2000]}
|
||||
|
||||
Identify any facts that contradict each other. For each conflict, provide:
|
||||
1. The fact from existing content
|
||||
2. The contradicting fact from new information
|
||||
3. Confidence level (low/medium/high)
|
||||
4. Context/explanation
|
||||
ANALYSIS STEPS:
|
||||
1. Identify specific factual claims in existing content (dates, numbers, names, states)
|
||||
2. Identify specific factual claims in new content
|
||||
3. Compare ONLY for direct contradictions (X says A, Y says not-A)
|
||||
|
||||
Return ONLY valid JSON:
|
||||
RULES:
|
||||
- Do NOT flag differences in wording or phrasing as conflicts
|
||||
- Do NOT flag new/additional information as conflicts
|
||||
- Do NOT flag opinion differences as conflicts
|
||||
- ONLY flag direct factual contradictions
|
||||
- Return valid JSON only, no commentary
|
||||
|
||||
Return format:
|
||||
{{
|
||||
"conflicts": [
|
||||
{{
|
||||
"existing_fact": "fact from old content",
|
||||
"new_fact": "contradicting fact",
|
||||
"confidence": "medium",
|
||||
"context": "explanation of why these conflict"
|
||||
}}
|
||||
{{"existing_fact": "...", "new_fact": "...", "confidence": "low/medium/high", "context": "..."}}
|
||||
]
|
||||
}}
|
||||
|
||||
If no conflicts, return: {{"conflicts": []}}
|
||||
If no conflicts: {{"conflicts": []}}
|
||||
|
||||
JSON:"""
|
||||
|
||||
@@ -188,7 +190,8 @@ JSON:"""
|
||||
response = await self.ollama.generate_text(
|
||||
prompt=prompt,
|
||||
model=self.model,
|
||||
stream=False
|
||||
stream=False,
|
||||
temperature=0.0 # Deterministic for consistent conflict detection
|
||||
)
|
||||
|
||||
# Extract JSON
|
||||
@@ -347,6 +350,13 @@ FORMATTING RULES:
|
||||
- Keep sections focused and scannable
|
||||
- Adapt structure to content - not all sections apply to all topics
|
||||
|
||||
CRITICAL CONSTRAINTS:
|
||||
- Do NOT invent facts not present in the source information above
|
||||
- Do NOT add speculative information or assumptions
|
||||
- Do NOT fill sections with placeholder text or generic statements
|
||||
- If information for a section is not available, OMIT the section entirely
|
||||
- Base ALL content strictly on provided source information
|
||||
|
||||
Generate ONLY the markdown content (do not include Sources, Knowledge Graph, or Mind Map sections - those are added automatically).
|
||||
|
||||
MARKDOWN:"""
|
||||
@@ -402,6 +412,13 @@ FORMATTING RULES:
|
||||
- Bold important terms
|
||||
- Add subsections (###) where it improves clarity
|
||||
|
||||
CRITICAL CONSTRAINTS:
|
||||
- Do NOT rephrase facts in ways that change their meaning
|
||||
- Do NOT remove ANY information unless explicitly superseded by newer facts
|
||||
- Do NOT add information not present in existing content or new information
|
||||
- Preserve exact quotes, dates, numbers, and names verbatim
|
||||
- Do NOT fill gaps with assumptions or general knowledge
|
||||
|
||||
OUTPUT INSTRUCTIONS:
|
||||
- Return complete page content (do not include Sources, Knowledge Graph, Mind Map - those are added automatically)
|
||||
- Include updated "Changes & Updates" section noting what was changed today
|
||||
@@ -409,13 +426,21 @@ OUTPUT INSTRUCTIONS:
|
||||
|
||||
RECONSTRUCTED MARKDOWN:"""
|
||||
|
||||
async def _call_llm(self, prompt: str) -> str:
|
||||
"""Call LLM with prompt and return response."""
|
||||
async def _call_llm(self, prompt: str, temperature: float = 0.3) -> str:
|
||||
"""
|
||||
Call LLM with prompt and return response.
|
||||
|
||||
Args:
|
||||
prompt: The prompt text
|
||||
temperature: Sampling temperature (0.0=deterministic, higher=creative)
|
||||
Default 0.3 for controlled but natural content generation
|
||||
"""
|
||||
try:
|
||||
response = await self.ollama.generate_text(
|
||||
prompt=prompt,
|
||||
model=self.model,
|
||||
stream=False
|
||||
stream=False,
|
||||
temperature=temperature
|
||||
)
|
||||
|
||||
if not response:
|
||||
|
||||
+20
-9
@@ -1,5 +1,6 @@
|
||||
"""Pytest configuration and shared fixtures for Library Desk tests."""
|
||||
|
||||
import os
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from typing import AsyncGenerator
|
||||
@@ -7,6 +8,9 @@ from typing import AsyncGenerator
|
||||
# Test configuration
|
||||
pytest_plugins = ("pytest_asyncio",)
|
||||
|
||||
# Use real host for tests (services available at this IP)
|
||||
TEST_HOST = os.environ.get("TEST_HOST", "192.168.86.149")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def test_user() -> str:
|
||||
@@ -17,49 +21,56 @@ def test_user() -> str:
|
||||
@pytest.fixture
|
||||
def neo4j_test_uri() -> str:
|
||||
"""Test Neo4j URI."""
|
||||
return "bolt://neo4j:7687"
|
||||
return f"bolt://{TEST_HOST}:7687"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def neo4j_test_auth() -> tuple:
|
||||
"""Test Neo4j authentication."""
|
||||
return ("neo4j", "test_password")
|
||||
from src.config import get_settings
|
||||
settings = get_settings()
|
||||
return ("neo4j", settings.neo4j_password)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def qdrant_test_url() -> str:
|
||||
"""Test Qdrant URL."""
|
||||
return "http://qdrant:6333"
|
||||
return f"http://{TEST_HOST}:6333"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def wikijs_test_config() -> dict:
|
||||
"""Test Wiki.js configuration."""
|
||||
from src.config import get_settings
|
||||
settings = get_settings()
|
||||
return {
|
||||
"base_url": "http://wiki:3000",
|
||||
"api_key": "test_api_key"
|
||||
"base_url": f"http://{TEST_HOST}:3000",
|
||||
"username": settings.wikijs_username,
|
||||
"password": settings.wikijs_password
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def searxng_test_url() -> str:
|
||||
"""Test SearXNG URL."""
|
||||
return "http://searxng:8080"
|
||||
return f"http://{TEST_HOST}:8080"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def ollama_test_config() -> dict:
|
||||
"""Test Ollama configuration."""
|
||||
from src.config import get_settings
|
||||
settings = get_settings()
|
||||
return {
|
||||
"base_url": "http://ollama:11434",
|
||||
"model": "nomic-embed-text"
|
||||
"base_url": f"http://{TEST_HOST}:11434",
|
||||
"model": settings.ollama_embedding_model
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def redis_test_url() -> str:
|
||||
"""Test Redis URL."""
|
||||
return "redis://redis-shared:6379/4"
|
||||
return f"redis://{TEST_HOST}:6379/4"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
|
||||
@@ -38,10 +38,10 @@ def settings():
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def neo4j_client(settings) -> AsyncGenerator[Neo4jClient, None]:
|
||||
async def neo4j_client(settings, neo4j_test_uri) -> AsyncGenerator[Neo4jClient, None]:
|
||||
"""Get connected Neo4j client."""
|
||||
client = Neo4jClient(
|
||||
uri=settings.neo4j_uri,
|
||||
uri=neo4j_test_uri,
|
||||
user=settings.neo4j_user,
|
||||
password=settings.neo4j_password
|
||||
)
|
||||
@@ -51,12 +51,12 @@ async def neo4j_client(settings) -> AsyncGenerator[Neo4jClient, None]:
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def wiki_client(settings) -> AsyncGenerator[WikiJSClient, None]:
|
||||
async def wiki_client(wikijs_test_config) -> AsyncGenerator[WikiJSClient, None]:
|
||||
"""Get Wiki.js client."""
|
||||
client = WikiJSClient(
|
||||
base_url=settings.wikijs_url,
|
||||
username=settings.wikijs_username,
|
||||
password=settings.wikijs_password
|
||||
base_url=wikijs_test_config["base_url"],
|
||||
username=wikijs_test_config["username"],
|
||||
password=wikijs_test_config["password"]
|
||||
)
|
||||
yield client
|
||||
|
||||
|
||||
+53
-49
@@ -43,10 +43,10 @@ def settings():
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def neo4j_client(settings) -> AsyncGenerator[Neo4jClient, None]:
|
||||
async def neo4j_client(settings, neo4j_test_uri) -> AsyncGenerator[Neo4jClient, None]:
|
||||
"""Get connected Neo4j client."""
|
||||
client = Neo4jClient(
|
||||
uri=settings.neo4j_uri,
|
||||
uri=neo4j_test_uri,
|
||||
user=settings.neo4j_user,
|
||||
password=settings.neo4j_password
|
||||
)
|
||||
@@ -56,32 +56,35 @@ async def neo4j_client(settings) -> AsyncGenerator[Neo4jClient, None]:
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def qdrant_client(settings) -> QdrantClientWrapper:
|
||||
def qdrant_client(qdrant_test_url) -> QdrantClientWrapper:
|
||||
"""Get Qdrant client."""
|
||||
return QdrantClientWrapper(url=settings.qdrant_url)
|
||||
return QdrantClientWrapper(url=qdrant_test_url)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def wiki_client(settings) -> AsyncGenerator[WikiJSClient, None]:
|
||||
async def wiki_client(wikijs_test_config) -> AsyncGenerator[WikiJSClient, None]:
|
||||
"""Get Wiki.js client."""
|
||||
client = WikiJSClient(
|
||||
base_url=settings.wikijs_url,
|
||||
username=settings.wikijs_username,
|
||||
password=settings.wikijs_password
|
||||
base_url=wikijs_test_config["base_url"],
|
||||
username=wikijs_test_config["username"],
|
||||
password=wikijs_test_config["password"]
|
||||
)
|
||||
yield client
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def searxng_client(settings) -> SearXNGClient:
|
||||
def searxng_client(searxng_test_url) -> SearXNGClient:
|
||||
"""Get SearXNG client."""
|
||||
return SearXNGClient(base_url=settings.searxng_url)
|
||||
return SearXNGClient(base_url=searxng_test_url)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def ollama_client(settings) -> OllamaClient:
|
||||
def ollama_client(ollama_test_config) -> OllamaClient:
|
||||
"""Get Ollama client."""
|
||||
return OllamaClient(base_url=settings.ollama_url)
|
||||
return OllamaClient(
|
||||
base_url=ollama_test_config["base_url"],
|
||||
model=ollama_test_config["model"]
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
@@ -219,53 +222,54 @@ async def test_vector_data(vector_service, test_wiki_page):
|
||||
# ============================================================================
|
||||
|
||||
class TestRRFFusion:
|
||||
"""Test Reciprocal Rank Fusion algorithm."""
|
||||
"""Test two-stage Reciprocal Rank Fusion algorithm."""
|
||||
|
||||
def test_rrf_single_source(self, hybrid_rag_service):
|
||||
"""Test RRF with single source."""
|
||||
results_by_source = {
|
||||
"vector": [
|
||||
{"page_id": 1, "title": "Doc 1", "content": "test"},
|
||||
{"page_id": 2, "title": "Doc 2", "content": "test"}
|
||||
]
|
||||
}
|
||||
def test_wiki_merge_single_source(self, hybrid_rag_service):
|
||||
"""Test wiki merge with single source (vector only)."""
|
||||
vector_results = [
|
||||
{"page_id": 1, "title": "Doc 1", "content": "test"},
|
||||
{"page_id": 2, "title": "Doc 2", "content": "test"}
|
||||
]
|
||||
|
||||
fused = hybrid_rag_service._reciprocal_rank_fusion(results_by_source, k=60)
|
||||
merged = hybrid_rag_service._merge_wiki_sources(vector_results, [], k=60)
|
||||
|
||||
assert len(fused) == 2
|
||||
assert fused[0]["rrf_score"] > fused[1]["rrf_score"] # Rank 1 > Rank 2
|
||||
assert fused[0]["sources"] == ["vector"]
|
||||
assert len(merged) == 2
|
||||
assert merged[0]["wiki_rrf_score"] > merged[1]["wiki_rrf_score"] # Rank 1 > Rank 2
|
||||
assert merged[0]["found_by"] == ["vector"]
|
||||
|
||||
def test_rrf_multiple_sources_same_doc(self, hybrid_rag_service):
|
||||
"""Test RRF with same document from multiple sources."""
|
||||
results_by_source = {
|
||||
"vector": [{"page_id": 1, "title": "Doc 1", "content": "test"}],
|
||||
"graph": [{"page_id": 1, "title": "Doc 1", "content": ""}],
|
||||
}
|
||||
def test_wiki_merge_multiple_sources_same_doc(self, hybrid_rag_service):
|
||||
"""Test wiki merge with same document from vector and graph."""
|
||||
vector_results = [{"page_id": 1, "title": "Doc 1", "content": "test"}]
|
||||
graph_results = [{"page_id": 1, "title": "Doc 1", "content": ""}]
|
||||
|
||||
fused = hybrid_rag_service._reciprocal_rank_fusion(results_by_source, k=60)
|
||||
merged = hybrid_rag_service._merge_wiki_sources(vector_results, graph_results, k=60)
|
||||
|
||||
assert len(fused) == 1 # Deduplicated
|
||||
assert len(fused[0]["sources"]) == 2 # Both sources
|
||||
assert "vector" in fused[0]["sources"]
|
||||
assert "graph" in fused[0]["sources"]
|
||||
# RRF score should be sum: 1/(60+1) + 1/(60+1)
|
||||
assert len(merged) == 1 # Deduplicated
|
||||
assert len(merged[0]["found_by"]) == 2 # Both sources
|
||||
assert "vector" in merged[0]["found_by"]
|
||||
assert "graph" in merged[0]["found_by"]
|
||||
# Wiki RRF score should be sum: 1/(60+1) + 1/(60+1)
|
||||
expected_score = 1/61 + 1/61
|
||||
assert abs(fused[0]["rrf_score"] - expected_score) < 0.001
|
||||
assert abs(merged[0]["wiki_rrf_score"] - expected_score) < 0.001
|
||||
|
||||
def test_rrf_web_results(self, hybrid_rag_service):
|
||||
"""Test RRF with web results (URL-based)."""
|
||||
results_by_source = {
|
||||
"web": [
|
||||
{"url": "https://example.com/1", "title": "Web 1", "content": "test"},
|
||||
{"url": "https://example.com/2", "title": "Web 2", "content": "test"}
|
||||
]
|
||||
}
|
||||
def test_final_rrf_wiki_and_web(self, hybrid_rag_service):
|
||||
"""Test final RRF between wiki and web results."""
|
||||
# Pre-merged wiki results
|
||||
wiki_results = [
|
||||
{"page_id": 1, "title": "Wiki 1", "content": "test", "found_by": ["vector"]}
|
||||
]
|
||||
web_results = [
|
||||
{"url": "https://example.com/1", "title": "Web 1", "content": "test"},
|
||||
{"url": "https://example.com/2", "title": "Web 2", "content": "test"}
|
||||
]
|
||||
|
||||
fused = hybrid_rag_service._reciprocal_rank_fusion(results_by_source, k=60)
|
||||
fused = hybrid_rag_service._reciprocal_rank_fusion(wiki_results, web_results, k=60)
|
||||
|
||||
assert len(fused) == 2
|
||||
assert fused[0]["result"]["url"] == "https://example.com/1"
|
||||
assert len(fused) == 3
|
||||
# Wiki rank 1 and web rank 1 should have same RRF score
|
||||
wiki_score = next(r["rrf_score"] for r in fused if r["source_type"] == "wiki")
|
||||
web_score = next(r["rrf_score"] for r in fused if r["source_type"] == "web")
|
||||
assert abs(wiki_score - web_score) < 0.001 # Equal footing
|
||||
|
||||
|
||||
class TestContextFormatting:
|
||||
|
||||
+16
-15
@@ -30,10 +30,10 @@ def settings():
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def neo4j_client(settings) -> AsyncGenerator[Neo4jClient, None]:
|
||||
async def neo4j_client(settings, neo4j_test_uri) -> AsyncGenerator[Neo4jClient, None]:
|
||||
"""Get connected Neo4j client."""
|
||||
client = Neo4jClient(
|
||||
uri=settings.neo4j_uri,
|
||||
uri=neo4j_test_uri,
|
||||
user=settings.neo4j_user,
|
||||
password=settings.neo4j_password
|
||||
)
|
||||
@@ -43,45 +43,46 @@ async def neo4j_client(settings) -> AsyncGenerator[Neo4jClient, None]:
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def qdrant_client(settings) -> QdrantClientWrapper:
|
||||
def qdrant_client(settings, qdrant_test_url) -> QdrantClientWrapper:
|
||||
"""Get Qdrant client."""
|
||||
return QdrantClientWrapper(url=settings.qdrant_url)
|
||||
return QdrantClientWrapper(url=qdrant_test_url)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def wikijs_client(settings) -> AsyncGenerator[WikiJSClient, None]:
|
||||
async def wikijs_client(wikijs_test_config) -> AsyncGenerator[WikiJSClient, None]:
|
||||
"""Get Wiki.js client."""
|
||||
client = WikiJSClient(
|
||||
base_url=settings.wikijs_url,
|
||||
api_key=settings.wikijs_api_key
|
||||
base_url=wikijs_test_config["base_url"],
|
||||
username=wikijs_test_config["username"],
|
||||
password=wikijs_test_config["password"]
|
||||
)
|
||||
yield client
|
||||
await client.close()
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def searxng_client(settings) -> AsyncGenerator[SearXNGClient, None]:
|
||||
async def searxng_client(searxng_test_url) -> AsyncGenerator[SearXNGClient, None]:
|
||||
"""Get SearXNG client."""
|
||||
client = SearXNGClient(base_url=settings.searxng_url)
|
||||
client = SearXNGClient(base_url=searxng_test_url)
|
||||
yield client
|
||||
await client.close()
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def ollama_client(settings) -> AsyncGenerator[OllamaClient, None]:
|
||||
"""Get Ollama client."""
|
||||
async def ollama_client(ollama_test_config) -> AsyncGenerator[OllamaClient, None]:
|
||||
"""Get Ollama client for embeddings."""
|
||||
client = OllamaClient(
|
||||
base_url=settings.ollama_url,
|
||||
model=settings.ollama_model
|
||||
base_url=ollama_test_config["base_url"],
|
||||
model=ollama_test_config["model"]
|
||||
)
|
||||
yield client
|
||||
await client.close()
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def job_manager(settings) -> AsyncGenerator[JobManager, None]:
|
||||
async def job_manager(redis_test_url) -> AsyncGenerator[JobManager, None]:
|
||||
"""Get job manager."""
|
||||
manager = JobManager(redis_url=settings.redis_url)
|
||||
manager = JobManager(redis_url=redis_test_url)
|
||||
await manager.connect()
|
||||
yield manager
|
||||
await manager.close()
|
||||
|
||||
@@ -0,0 +1,488 @@
|
||||
"""
|
||||
Tests for maintenance router and cleanup functionality.
|
||||
|
||||
Tests cleanup of:
|
||||
- Orphan vector chunks
|
||||
- Orphan entities in graph
|
||||
- Stale document nodes
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
from src.routers.maintenance import (
|
||||
cleanup_vectors,
|
||||
cleanup_graph,
|
||||
cleanup_all,
|
||||
maintenance_health,
|
||||
reindex_page,
|
||||
CleanupResult,
|
||||
VectorCleanupResponse,
|
||||
GraphCleanupResponse,
|
||||
FullCleanupResponse,
|
||||
HealthCheckResponse,
|
||||
ReindexResponse
|
||||
)
|
||||
|
||||
|
||||
class TestCleanupResult:
|
||||
"""Test CleanupResult model."""
|
||||
|
||||
def test_cleanup_result_defaults(self):
|
||||
"""Test CleanupResult with default values."""
|
||||
result = CleanupResult(duration_ms=100.0)
|
||||
assert result.orphans_found == 0
|
||||
assert result.orphans_purged == 0
|
||||
assert result.duration_ms == 100.0
|
||||
|
||||
def test_cleanup_result_with_values(self):
|
||||
"""Test CleanupResult with actual values."""
|
||||
result = CleanupResult(
|
||||
orphans_found=10,
|
||||
orphans_purged=8,
|
||||
duration_ms=250.5
|
||||
)
|
||||
assert result.orphans_found == 10
|
||||
assert result.orphans_purged == 8
|
||||
assert result.duration_ms == 250.5
|
||||
|
||||
|
||||
class TestVectorCleanupResponse:
|
||||
"""Test VectorCleanupResponse model."""
|
||||
|
||||
def test_vector_cleanup_response(self):
|
||||
"""Test VectorCleanupResponse structure."""
|
||||
response = VectorCleanupResponse(
|
||||
success=True,
|
||||
wiki_chunks=CleanupResult(orphans_found=5, orphans_purged=5, duration_ms=50),
|
||||
document_chunks=CleanupResult(orphans_found=3, orphans_purged=3, duration_ms=50),
|
||||
chunks_without_graph=CleanupResult(orphans_found=2, orphans_purged=2, duration_ms=50),
|
||||
total_chunks_scanned=100,
|
||||
total_orphans_purged=10,
|
||||
duration_ms=100
|
||||
)
|
||||
assert response.success is True
|
||||
assert response.wiki_chunks.orphans_found == 5
|
||||
assert response.document_chunks.orphans_found == 3
|
||||
assert response.chunks_without_graph.orphans_found == 2
|
||||
assert response.total_orphans_purged == 10
|
||||
|
||||
|
||||
class TestGraphCleanupResponse:
|
||||
"""Test GraphCleanupResponse model."""
|
||||
|
||||
def test_graph_cleanup_response(self):
|
||||
"""Test GraphCleanupResponse structure."""
|
||||
response = GraphCleanupResponse(
|
||||
success=True,
|
||||
orphan_entities=CleanupResult(orphans_found=10, orphans_purged=10, duration_ms=25),
|
||||
stale_wiki_documents=CleanupResult(orphans_found=2, orphans_purged=2, duration_ms=25),
|
||||
stale_store_documents=CleanupResult(orphans_found=0, orphans_purged=0, duration_ms=25),
|
||||
docs_without_vectors=CleanupResult(orphans_found=1, orphans_purged=1, duration_ms=25),
|
||||
broken_relationships_cleaned=5,
|
||||
duration_ms=100
|
||||
)
|
||||
assert response.success is True
|
||||
assert response.orphan_entities.orphans_found == 10
|
||||
assert response.docs_without_vectors.orphans_found == 1
|
||||
assert response.broken_relationships_cleaned == 5
|
||||
|
||||
|
||||
class TestHealthCheckResponse:
|
||||
"""Test HealthCheckResponse model."""
|
||||
|
||||
def test_health_check_healthy(self):
|
||||
"""Test healthy status."""
|
||||
response = HealthCheckResponse(
|
||||
status="healthy",
|
||||
orphan_vector_count=0,
|
||||
orphan_entity_count=0,
|
||||
stale_document_count=0
|
||||
)
|
||||
assert response.status == "healthy"
|
||||
assert response.recommendations == []
|
||||
|
||||
def test_health_check_degraded(self):
|
||||
"""Test degraded status with recommendations."""
|
||||
response = HealthCheckResponse(
|
||||
status="degraded",
|
||||
orphan_vector_count=15,
|
||||
orphan_entity_count=3,
|
||||
stale_document_count=0,
|
||||
recommendations=[
|
||||
"Found 15 orphan vector chunks. Consider running POST /maintenance/cleanup/vectors"
|
||||
]
|
||||
)
|
||||
assert response.status == "degraded"
|
||||
assert len(response.recommendations) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
class TestVectorCleanup:
|
||||
"""Test vector cleanup endpoint."""
|
||||
|
||||
async def test_cleanup_vectors_no_orphans(self):
|
||||
"""Test cleanup when no orphans exist."""
|
||||
# Mock services
|
||||
vector_service = AsyncMock()
|
||||
vector_service.get_all_chunk_references.return_value = [
|
||||
{"chunk_id": "c1", "page_id": 1, "doc_type": "wiki"}
|
||||
]
|
||||
# find_chunks_without_graph_nodes is not async
|
||||
vector_service.find_chunks_without_graph_nodes = MagicMock(return_value=[])
|
||||
|
||||
graph_service = AsyncMock()
|
||||
graph_service.get_all_document_references.return_value = [
|
||||
{"page_id": 1, "doc_type": "wiki", "title": "Test"}
|
||||
]
|
||||
|
||||
wiki_client = AsyncMock()
|
||||
wiki_client.list_all_pages.return_value = [{"id": 1, "path": "test"}]
|
||||
|
||||
# Call cleanup
|
||||
result = await cleanup_vectors(
|
||||
user="testuser",
|
||||
dry_run=False,
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
wiki_client=wiki_client,
|
||||
api_key="test"
|
||||
)
|
||||
|
||||
assert result.success is True
|
||||
assert result.wiki_chunks.orphans_found == 0
|
||||
assert result.chunks_without_graph.orphans_found == 0
|
||||
assert result.total_orphans_purged == 0
|
||||
|
||||
async def test_cleanup_vectors_with_orphans(self):
|
||||
"""Test cleanup when orphans exist."""
|
||||
# Mock services
|
||||
vector_service = AsyncMock()
|
||||
vector_service.get_all_chunk_references.return_value = [
|
||||
{"chunk_id": "c1", "page_id": 1, "doc_type": "wiki"},
|
||||
{"chunk_id": "c2", "page_id": 999, "doc_type": "wiki"}, # Orphan
|
||||
{"chunk_id": "c3", "page_id": 999, "doc_type": "wiki"}, # Orphan
|
||||
]
|
||||
vector_service.purge_chunks_by_ids.return_value = 2
|
||||
# find_chunks_without_graph_nodes is not async
|
||||
vector_service.find_chunks_without_graph_nodes = MagicMock(return_value=[])
|
||||
|
||||
graph_service = AsyncMock()
|
||||
graph_service.get_all_document_references.return_value = [
|
||||
{"page_id": 1, "doc_type": "wiki", "title": "Test"}
|
||||
]
|
||||
|
||||
wiki_client = AsyncMock()
|
||||
wiki_client.list_all_pages.return_value = [{"id": 1, "path": "test"}]
|
||||
|
||||
# Call cleanup
|
||||
result = await cleanup_vectors(
|
||||
user="testuser",
|
||||
dry_run=False,
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
wiki_client=wiki_client,
|
||||
api_key="test"
|
||||
)
|
||||
|
||||
assert result.success is True
|
||||
assert result.wiki_chunks.orphans_found == 2
|
||||
assert result.wiki_chunks.orphans_purged == 2
|
||||
assert result.total_orphans_purged == 2
|
||||
|
||||
async def test_cleanup_vectors_dry_run(self):
|
||||
"""Test cleanup dry run doesn't purge."""
|
||||
# Mock services
|
||||
vector_service = AsyncMock()
|
||||
vector_service.get_all_chunk_references.return_value = [
|
||||
{"chunk_id": "c1", "page_id": 999, "doc_type": "wiki"}, # Orphan
|
||||
]
|
||||
# find_chunks_without_graph_nodes is not async
|
||||
vector_service.find_chunks_without_graph_nodes = MagicMock(return_value=[])
|
||||
|
||||
graph_service = AsyncMock()
|
||||
graph_service.get_all_document_references.return_value = []
|
||||
|
||||
wiki_client = AsyncMock()
|
||||
wiki_client.list_all_pages.return_value = []
|
||||
|
||||
# Call cleanup in dry run mode
|
||||
result = await cleanup_vectors(
|
||||
user="testuser",
|
||||
dry_run=True,
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
wiki_client=wiki_client,
|
||||
api_key="test"
|
||||
)
|
||||
|
||||
assert result.success is True
|
||||
assert result.wiki_chunks.orphans_found == 1
|
||||
assert result.wiki_chunks.orphans_purged == 0 # Not purged due to dry run
|
||||
vector_service.purge_chunks_by_ids.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
class TestGraphCleanup:
|
||||
"""Test graph cleanup endpoint."""
|
||||
|
||||
async def test_cleanup_graph_no_orphans(self):
|
||||
"""Test cleanup when no orphans exist."""
|
||||
vector_service = AsyncMock()
|
||||
vector_service.get_all_chunk_references.return_value = [
|
||||
{"chunk_id": "c1", "page_id": 1, "doc_type": "wiki"}
|
||||
]
|
||||
|
||||
graph_service = AsyncMock()
|
||||
graph_service.find_orphan_entities.return_value = []
|
||||
graph_service.get_all_document_references.return_value = [
|
||||
{"page_id": 1, "doc_type": "wiki", "title": "Test"}
|
||||
]
|
||||
graph_service.find_documents_without_vectors.return_value = []
|
||||
graph_service.cleanup_broken_relationships.return_value = 0
|
||||
|
||||
wiki_client = AsyncMock()
|
||||
wiki_client.list_all_pages.return_value = [{"id": 1, "path": "test"}]
|
||||
|
||||
result = await cleanup_graph(
|
||||
user="testuser",
|
||||
dry_run=False,
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
wiki_client=wiki_client,
|
||||
api_key="test"
|
||||
)
|
||||
|
||||
assert result.success is True
|
||||
assert result.orphan_entities.orphans_found == 0
|
||||
assert result.stale_wiki_documents.orphans_found == 0
|
||||
assert result.docs_without_vectors.orphans_found == 0
|
||||
|
||||
async def test_cleanup_graph_with_orphan_entities(self):
|
||||
"""Test cleanup of orphan entities."""
|
||||
vector_service = AsyncMock()
|
||||
vector_service.get_all_chunk_references.return_value = []
|
||||
|
||||
graph_service = AsyncMock()
|
||||
graph_service.find_orphan_entities.return_value = [
|
||||
{"id": "e1", "name": "Orphan1", "type": "Person"},
|
||||
{"id": "e2", "name": "Orphan2", "type": "Technology"},
|
||||
]
|
||||
graph_service.purge_orphan_entities.return_value = 2
|
||||
graph_service.get_all_document_references.return_value = []
|
||||
graph_service.find_documents_without_vectors.return_value = []
|
||||
graph_service.cleanup_broken_relationships.return_value = 0
|
||||
|
||||
wiki_client = AsyncMock()
|
||||
wiki_client.list_all_pages.return_value = []
|
||||
|
||||
result = await cleanup_graph(
|
||||
user="testuser",
|
||||
dry_run=False,
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
wiki_client=wiki_client,
|
||||
api_key="test"
|
||||
)
|
||||
|
||||
assert result.success is True
|
||||
assert result.orphan_entities.orphans_found == 2
|
||||
assert result.orphan_entities.orphans_purged == 2
|
||||
|
||||
async def test_cleanup_graph_with_stale_documents(self):
|
||||
"""Test cleanup of stale document nodes."""
|
||||
vector_service = AsyncMock()
|
||||
vector_service.get_all_chunk_references.return_value = [
|
||||
{"chunk_id": "c1", "page_id": 1, "doc_type": "wiki"}
|
||||
]
|
||||
|
||||
graph_service = AsyncMock()
|
||||
graph_service.find_orphan_entities.return_value = []
|
||||
graph_service.get_all_document_references.return_value = [
|
||||
{"page_id": 1, "doc_type": "wiki", "title": "Exists"},
|
||||
{"page_id": 999, "doc_type": "wiki", "title": "Deleted"}, # Stale
|
||||
]
|
||||
graph_service.find_documents_without_vectors.return_value = []
|
||||
graph_service.purge_stale_documents_by_ids.return_value = 1
|
||||
graph_service.cleanup_broken_relationships.return_value = 0
|
||||
|
||||
wiki_client = AsyncMock()
|
||||
wiki_client.list_all_pages.return_value = [{"id": 1, "path": "test"}]
|
||||
|
||||
result = await cleanup_graph(
|
||||
user="testuser",
|
||||
dry_run=False,
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
wiki_client=wiki_client,
|
||||
api_key="test"
|
||||
)
|
||||
|
||||
assert result.success is True
|
||||
assert result.stale_wiki_documents.orphans_found == 1
|
||||
assert result.stale_wiki_documents.orphans_purged == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
class TestFullCleanup:
|
||||
"""Test full cleanup endpoint."""
|
||||
|
||||
async def test_full_cleanup(self):
|
||||
"""Test full cleanup runs both vector and graph cleanup."""
|
||||
vector_service = AsyncMock()
|
||||
vector_service.get_all_chunk_references.return_value = []
|
||||
# find_chunks_without_graph_nodes is not async
|
||||
vector_service.find_chunks_without_graph_nodes = MagicMock(return_value=[])
|
||||
|
||||
graph_service = AsyncMock()
|
||||
graph_service.find_orphan_entities.return_value = []
|
||||
graph_service.get_all_document_references.return_value = []
|
||||
graph_service.find_documents_without_vectors.return_value = []
|
||||
graph_service.cleanup_broken_relationships.return_value = 0
|
||||
|
||||
wiki_client = AsyncMock()
|
||||
wiki_client.list_all_pages.return_value = []
|
||||
|
||||
result = await cleanup_all(
|
||||
user="testuser",
|
||||
dry_run=False,
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
wiki_client=wiki_client,
|
||||
api_key="test"
|
||||
)
|
||||
|
||||
assert result.success is True
|
||||
assert result.vector_cleanup.success is True
|
||||
assert result.graph_cleanup.success is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
class TestMaintenanceHealth:
|
||||
"""Test maintenance health endpoint."""
|
||||
|
||||
async def test_health_healthy(self):
|
||||
"""Test healthy status when no orphans."""
|
||||
vector_service = AsyncMock()
|
||||
vector_service.get_all_chunk_references.return_value = []
|
||||
# find_chunks_without_graph_nodes is not async
|
||||
vector_service.find_chunks_without_graph_nodes = MagicMock(return_value=[])
|
||||
|
||||
graph_service = AsyncMock()
|
||||
graph_service.find_orphan_entities.return_value = []
|
||||
graph_service.get_all_document_references.return_value = []
|
||||
graph_service.find_documents_without_vectors.return_value = []
|
||||
|
||||
wiki_client = AsyncMock()
|
||||
wiki_client.list_all_pages.return_value = []
|
||||
|
||||
result = await maintenance_health(
|
||||
user="testuser",
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
wiki_client=wiki_client,
|
||||
api_key="test"
|
||||
)
|
||||
|
||||
assert result.status == "healthy"
|
||||
assert result.orphan_vector_count == 0
|
||||
assert result.orphan_entity_count == 0
|
||||
assert result.vectors_without_graph == 0
|
||||
assert result.docs_without_vectors == 0
|
||||
|
||||
async def test_health_degraded(self):
|
||||
"""Test degraded status with orphans."""
|
||||
vector_service = AsyncMock()
|
||||
vector_service.get_all_chunk_references.return_value = [
|
||||
{"chunk_id": f"c{i}", "page_id": 999, "doc_type": "wiki"}
|
||||
for i in range(15)
|
||||
]
|
||||
# find_chunks_without_graph_nodes is not async
|
||||
vector_service.find_chunks_without_graph_nodes = MagicMock(return_value=[])
|
||||
|
||||
graph_service = AsyncMock()
|
||||
graph_service.find_orphan_entities.return_value = [
|
||||
{"id": f"e{i}", "name": f"Entity{i}", "type": "Entity"}
|
||||
for i in range(3)
|
||||
]
|
||||
graph_service.get_all_document_references.return_value = []
|
||||
graph_service.find_documents_without_vectors.return_value = []
|
||||
|
||||
wiki_client = AsyncMock()
|
||||
wiki_client.list_all_pages.return_value = []
|
||||
|
||||
result = await maintenance_health(
|
||||
user="testuser",
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
wiki_client=wiki_client,
|
||||
api_key="test"
|
||||
)
|
||||
|
||||
assert result.status == "degraded"
|
||||
assert result.orphan_vector_count == 15
|
||||
assert result.orphan_entity_count == 3
|
||||
assert len(result.recommendations) >= 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
class TestReindexPage:
|
||||
"""Test reindex page endpoint."""
|
||||
|
||||
async def test_reindex_success(self):
|
||||
"""Test successful page reindex."""
|
||||
vector_service = AsyncMock()
|
||||
vector_service.delete_page_chunks.return_value = 5
|
||||
vector_service.update_from_page.return_value = MagicMock(
|
||||
success=True,
|
||||
chunks_created=6,
|
||||
error_message=None
|
||||
)
|
||||
|
||||
graph_service = AsyncMock()
|
||||
graph_service.delete_page.return_value = 1
|
||||
graph_service.update_from_page.return_value = MagicMock(
|
||||
success=True,
|
||||
error_message=None
|
||||
)
|
||||
|
||||
result = await reindex_page(
|
||||
page_id=123,
|
||||
user="testuser",
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
api_key="test"
|
||||
)
|
||||
|
||||
assert result.success is True
|
||||
assert result.page_id == 123
|
||||
assert result.vectors_deleted == 5
|
||||
assert result.vectors_created == 6
|
||||
assert result.graph_updated is True
|
||||
|
||||
async def test_reindex_failure(self):
|
||||
"""Test reindex with failure."""
|
||||
vector_service = AsyncMock()
|
||||
vector_service.delete_page_chunks.return_value = 0
|
||||
vector_service.update_from_page.return_value = MagicMock(
|
||||
success=False,
|
||||
chunks_created=0,
|
||||
error_message="Page not found"
|
||||
)
|
||||
|
||||
graph_service = AsyncMock()
|
||||
graph_service.delete_page.return_value = 0
|
||||
graph_service.update_from_page.return_value = MagicMock(
|
||||
success=False,
|
||||
error_message="Page not found"
|
||||
)
|
||||
|
||||
result = await reindex_page(
|
||||
page_id=999,
|
||||
user="testuser",
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
api_key="test"
|
||||
)
|
||||
|
||||
assert result.success is False
|
||||
assert result.error == "Page not found"
|
||||
@@ -0,0 +1,567 @@
|
||||
"""
|
||||
Tests for volatile cache router and service (Qdrant backend).
|
||||
|
||||
Tests:
|
||||
- Volatile record CRUD operations
|
||||
- Namespace listing and management
|
||||
- Scheduled record retrieval
|
||||
- TTL behavior and expiry filtering
|
||||
- Semantic search
|
||||
- Natural language conversion
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from datetime import datetime
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
from src.models.volatile import (
|
||||
VolatileRecord,
|
||||
VolatileRecordCreate,
|
||||
VolatileRecordResponse,
|
||||
VolatileListResponse,
|
||||
VolatileScheduledResponse,
|
||||
VolatileStatsResponse,
|
||||
VolatileDeleteResponse,
|
||||
VolatileBulkDeleteResponse,
|
||||
VolatileNamespace,
|
||||
NAMESPACE_DEFAULT_TTL,
|
||||
)
|
||||
|
||||
|
||||
class TestVolatileModels:
|
||||
"""Test volatile data models."""
|
||||
|
||||
def test_volatile_record_creation(self):
|
||||
"""Test VolatileRecord model creation."""
|
||||
record = VolatileRecord(
|
||||
key="rotterdam",
|
||||
namespace="weather",
|
||||
data={"temperature": 18, "conditions": "Cloudy"},
|
||||
source="openweathermap",
|
||||
ttl=1800,
|
||||
user="jpmschweitzer",
|
||||
)
|
||||
assert record.key == "rotterdam"
|
||||
assert record.namespace == "weather"
|
||||
assert record.data["temperature"] == 18
|
||||
assert record.ttl == 1800
|
||||
assert record.refresh_schedule is None
|
||||
|
||||
def test_volatile_record_with_schedule(self):
|
||||
"""Test VolatileRecord with refresh schedule."""
|
||||
record = VolatileRecord(
|
||||
key="nos-headlines",
|
||||
namespace="news",
|
||||
data={"headlines": ["Test headline"]},
|
||||
source="nos.nl",
|
||||
ttl=3600,
|
||||
refresh_schedule="0 * * * *",
|
||||
user="jpmschweitzer",
|
||||
)
|
||||
assert record.refresh_schedule == "0 * * * *"
|
||||
|
||||
def test_volatile_record_create(self):
|
||||
"""Test VolatileRecordCreate model."""
|
||||
create = VolatileRecordCreate(
|
||||
data={"price": 150.50, "change": 2.3},
|
||||
source="alpha_vantage",
|
||||
ttl=300,
|
||||
)
|
||||
assert create.data["price"] == 150.50
|
||||
assert create.ttl == 300
|
||||
|
||||
def test_volatile_record_response(self):
|
||||
"""Test VolatileRecordResponse model."""
|
||||
response = VolatileRecordResponse(
|
||||
key="rotterdam",
|
||||
namespace="weather",
|
||||
data={"temperature": 18},
|
||||
source="openweathermap",
|
||||
created_at=datetime.utcnow(),
|
||||
updated_at=datetime.utcnow(),
|
||||
ttl=1800,
|
||||
ttl_remaining=1500,
|
||||
user="jpmschweitzer",
|
||||
)
|
||||
assert response.ttl_remaining == 1500
|
||||
assert response.ttl == 1800
|
||||
|
||||
|
||||
class TestVolatileNamespaces:
|
||||
"""Test volatile namespaces and defaults."""
|
||||
|
||||
def test_all_namespaces_have_default_ttl(self):
|
||||
"""Verify all namespaces have default TTLs defined."""
|
||||
for ns in VolatileNamespace:
|
||||
assert ns in NAMESPACE_DEFAULT_TTL, f"Missing TTL for {ns}"
|
||||
assert NAMESPACE_DEFAULT_TTL[ns] > 0
|
||||
|
||||
def test_weather_default_ttl(self):
|
||||
"""Test weather namespace default TTL."""
|
||||
assert NAMESPACE_DEFAULT_TTL[VolatileNamespace.WEATHER] == 1800 # 30 min
|
||||
|
||||
def test_financial_default_ttl(self):
|
||||
"""Test financial namespace default TTL."""
|
||||
assert NAMESPACE_DEFAULT_TTL[VolatileNamespace.FINANCIAL] == 300 # 5 min
|
||||
|
||||
def test_sports_default_ttl(self):
|
||||
"""Test sports namespace default TTL (fast updates)."""
|
||||
assert NAMESPACE_DEFAULT_TTL[VolatileNamespace.SPORTS] == 60 # 1 min
|
||||
|
||||
def test_namespace_count(self):
|
||||
"""Test we have the expected number of namespaces."""
|
||||
assert len(VolatileNamespace) == 11
|
||||
|
||||
|
||||
class TestVolatileListResponse:
|
||||
"""Test list response models."""
|
||||
|
||||
def test_list_response(self):
|
||||
"""Test VolatileListResponse model."""
|
||||
response = VolatileListResponse(
|
||||
namespace="weather",
|
||||
keys=["rotterdam", "amsterdam", "utrecht"],
|
||||
count=3,
|
||||
user="jpmschweitzer",
|
||||
)
|
||||
assert response.count == 3
|
||||
assert "rotterdam" in response.keys
|
||||
|
||||
|
||||
class TestVolatileScheduledResponse:
|
||||
"""Test scheduled records response."""
|
||||
|
||||
def test_scheduled_response_empty(self):
|
||||
"""Test empty scheduled response."""
|
||||
response = VolatileScheduledResponse(
|
||||
records=[],
|
||||
count=0,
|
||||
user="jpmschweitzer",
|
||||
)
|
||||
assert response.count == 0
|
||||
assert response.records == []
|
||||
|
||||
def test_scheduled_response_with_records(self):
|
||||
"""Test scheduled response with records."""
|
||||
record = VolatileRecordResponse(
|
||||
key="nos-headlines",
|
||||
namespace="news",
|
||||
data={"headlines": []},
|
||||
source="nos.nl",
|
||||
created_at=datetime.utcnow(),
|
||||
updated_at=datetime.utcnow(),
|
||||
ttl=3600,
|
||||
ttl_remaining=3000,
|
||||
refresh_schedule="0 */6 * * *",
|
||||
user="jpmschweitzer",
|
||||
)
|
||||
response = VolatileScheduledResponse(
|
||||
records=[record],
|
||||
count=1,
|
||||
user="jpmschweitzer",
|
||||
)
|
||||
assert response.count == 1
|
||||
assert response.records[0].refresh_schedule == "0 */6 * * *"
|
||||
|
||||
|
||||
class TestVolatileStatsResponse:
|
||||
"""Test stats response model."""
|
||||
|
||||
def test_stats_response(self):
|
||||
"""Test VolatileStatsResponse model."""
|
||||
response = VolatileStatsResponse(
|
||||
total_records=15,
|
||||
by_namespace={"weather": 3, "news": 5, "financial": 7},
|
||||
scheduled_count=2,
|
||||
total_memory_bytes=None,
|
||||
user="jpmschweitzer",
|
||||
)
|
||||
assert response.total_records == 15
|
||||
assert response.by_namespace["weather"] == 3
|
||||
assert response.scheduled_count == 2
|
||||
|
||||
|
||||
class TestVolatileDeleteResponses:
|
||||
"""Test delete response models."""
|
||||
|
||||
def test_delete_response(self):
|
||||
"""Test VolatileDeleteResponse model."""
|
||||
response = VolatileDeleteResponse(
|
||||
key="rotterdam",
|
||||
namespace="weather",
|
||||
deleted=True,
|
||||
user="jpmschweitzer",
|
||||
)
|
||||
assert response.deleted is True
|
||||
|
||||
def test_delete_not_found(self):
|
||||
"""Test delete response when record not found."""
|
||||
response = VolatileDeleteResponse(
|
||||
key="nonexistent",
|
||||
namespace="weather",
|
||||
deleted=False,
|
||||
user="jpmschweitzer",
|
||||
)
|
||||
assert response.deleted is False
|
||||
|
||||
def test_bulk_delete_response(self):
|
||||
"""Test VolatileBulkDeleteResponse model."""
|
||||
response = VolatileBulkDeleteResponse(
|
||||
namespace="weather",
|
||||
deleted_count=5,
|
||||
user="jpmschweitzer",
|
||||
)
|
||||
assert response.deleted_count == 5
|
||||
assert response.namespace == "weather"
|
||||
|
||||
|
||||
class TestVolatileService:
|
||||
"""Test VolatileCacheService functionality (Qdrant backend)."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_qdrant(self):
|
||||
"""Create mock Qdrant client."""
|
||||
qdrant = AsyncMock()
|
||||
qdrant.ensure_collection = AsyncMock()
|
||||
qdrant.collection_exists = AsyncMock(return_value=True)
|
||||
qdrant.upsert_vector = AsyncMock(return_value=True)
|
||||
qdrant.delete_by_ids = AsyncMock(return_value=1)
|
||||
qdrant.search_with_expiry_filter = AsyncMock(return_value=[])
|
||||
qdrant.scroll_all_points = AsyncMock(return_value=[])
|
||||
qdrant.delete_expired_vectors = AsyncMock(return_value=0)
|
||||
qdrant.get_volatile_collections = AsyncMock(return_value=[])
|
||||
return qdrant
|
||||
|
||||
@pytest.fixture
|
||||
def mock_ollama(self):
|
||||
"""Create mock Ollama client."""
|
||||
ollama = AsyncMock()
|
||||
ollama.embed = AsyncMock(return_value=[0.1] * 768) # Return 768-dim embedding
|
||||
return ollama
|
||||
|
||||
@pytest.fixture
|
||||
def mock_settings(self):
|
||||
"""Create mock settings."""
|
||||
settings = MagicMock()
|
||||
settings.volatile_default_ttl = 3600
|
||||
return settings
|
||||
|
||||
@pytest.fixture
|
||||
def volatile_service(self, mock_qdrant, mock_ollama, mock_settings):
|
||||
"""Create VolatileCacheService with mocks."""
|
||||
from src.services.volatile_service import VolatileCacheService
|
||||
return VolatileCacheService(
|
||||
qdrant_client=mock_qdrant,
|
||||
ollama_client=mock_ollama,
|
||||
settings=mock_settings
|
||||
)
|
||||
|
||||
def test_collection_name(self, volatile_service):
|
||||
"""Test collection naming pattern."""
|
||||
name = volatile_service._collection_name("jpmschweitzer")
|
||||
assert name == "volatile_jpmschweitzer"
|
||||
|
||||
def test_make_vector_id(self, volatile_service):
|
||||
"""Test deterministic vector ID generation."""
|
||||
id1 = volatile_service._make_vector_id("weather", "rotterdam")
|
||||
id2 = volatile_service._make_vector_id("weather", "rotterdam")
|
||||
id3 = volatile_service._make_vector_id("weather", "amsterdam")
|
||||
|
||||
assert id1 == id2 # Same namespace+key = same ID
|
||||
assert id1 != id3 # Different key = different ID
|
||||
assert len(id1) == 32 # MD5 hex length
|
||||
|
||||
def test_get_default_ttl_known_namespace(self, volatile_service):
|
||||
"""Test default TTL for known namespace."""
|
||||
ttl = volatile_service._get_default_ttl("weather")
|
||||
assert ttl == 1800 # Weather namespace default
|
||||
|
||||
def test_get_default_ttl_unknown_namespace(self, volatile_service):
|
||||
"""Test default TTL for unknown namespace."""
|
||||
ttl = volatile_service._get_default_ttl("unknown_namespace")
|
||||
assert ttl == 3600 # Falls back to settings default
|
||||
|
||||
def test_to_natural_language_weather(self, volatile_service):
|
||||
"""Test natural language conversion for weather data."""
|
||||
text = volatile_service._to_natural_language(
|
||||
namespace="weather",
|
||||
key="rotterdam",
|
||||
data={"temperature": 18, "conditions": "Cloudy", "humidity": 75}
|
||||
)
|
||||
assert "rotterdam" in text.lower()
|
||||
assert "18" in text
|
||||
assert "Cloudy" in text
|
||||
assert "75" in text
|
||||
|
||||
def test_to_natural_language_news(self, volatile_service):
|
||||
"""Test natural language conversion for news data."""
|
||||
text = volatile_service._to_natural_language(
|
||||
namespace="news",
|
||||
key="nos-headlines",
|
||||
data={"title": "Breaking News", "summary": "Something happened", "source": "NOS"}
|
||||
)
|
||||
assert "Breaking News" in text
|
||||
assert "Something happened" in text
|
||||
assert "NOS" in text
|
||||
|
||||
def test_to_natural_language_financial(self, volatile_service):
|
||||
"""Test natural language conversion for financial data."""
|
||||
text = volatile_service._to_natural_language(
|
||||
namespace="financial",
|
||||
key="AAPL",
|
||||
data={"symbol": "AAPL", "price": 150.50, "change": 2.3}
|
||||
)
|
||||
assert "AAPL" in text
|
||||
assert "price" in text.lower()
|
||||
assert "change" in text.lower()
|
||||
|
||||
def test_to_natural_language_transit(self, volatile_service):
|
||||
"""Test natural language conversion for transit data."""
|
||||
text = volatile_service._to_natural_language(
|
||||
namespace="transit",
|
||||
key="ns-intercity",
|
||||
data={"route": "Amsterdam-Rotterdam", "status": "On time", "delay": 0}
|
||||
)
|
||||
assert "Amsterdam-Rotterdam" in text or "ns-intercity" in text.lower()
|
||||
assert "On time" in text
|
||||
|
||||
def test_to_natural_language_fallback(self, volatile_service):
|
||||
"""Test natural language fallback for unknown namespace."""
|
||||
text = volatile_service._to_natural_language(
|
||||
namespace="custom",
|
||||
key="test-key",
|
||||
data={"foo": "bar", "count": 42}
|
||||
)
|
||||
assert "custom" in text.lower()
|
||||
assert "foo" in text or "bar" in text
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_store_success(self, volatile_service, mock_qdrant, mock_ollama):
|
||||
"""Test successful store operation."""
|
||||
result = await volatile_service.store(
|
||||
user="jpmschweitzer",
|
||||
namespace="weather",
|
||||
key="rotterdam",
|
||||
data={"temperature": 18, "conditions": "Sunny"},
|
||||
source="openweathermap",
|
||||
ttl=1800
|
||||
)
|
||||
|
||||
assert result.key == "rotterdam"
|
||||
assert result.namespace == "weather"
|
||||
assert result.ttl == 1800
|
||||
mock_qdrant.ensure_collection.assert_called_once()
|
||||
mock_ollama.embed.assert_called_once()
|
||||
mock_qdrant.upsert_vector.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_store_uses_namespace_default_ttl(self, volatile_service, mock_qdrant, mock_ollama):
|
||||
"""Test store uses namespace default TTL when not specified."""
|
||||
result = await volatile_service.store(
|
||||
user="jpmschweitzer",
|
||||
namespace="weather",
|
||||
key="amsterdam",
|
||||
data={"temperature": 16},
|
||||
source="openweathermap",
|
||||
ttl=None # Not specified
|
||||
)
|
||||
|
||||
assert result.ttl == 1800 # Weather default
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_empty_collection(self, volatile_service, mock_qdrant, mock_ollama):
|
||||
"""Test search when collection doesn't exist."""
|
||||
mock_qdrant.collection_exists.return_value = False
|
||||
|
||||
results = await volatile_service.search(
|
||||
user="jpmschweitzer",
|
||||
query="weather rotterdam"
|
||||
)
|
||||
|
||||
assert results == []
|
||||
mock_ollama.embed.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_with_results(self, volatile_service, mock_qdrant, mock_ollama):
|
||||
"""Test search returns results."""
|
||||
import time
|
||||
now_ms = int(time.time() * 1000)
|
||||
|
||||
mock_qdrant.search_with_expiry_filter.return_value = [
|
||||
{
|
||||
"score": 0.95,
|
||||
"payload": {
|
||||
"key": "rotterdam",
|
||||
"namespace": "weather",
|
||||
"raw_data": {"temperature": 18},
|
||||
"source": "openweathermap",
|
||||
"created_at": datetime.utcnow().isoformat(),
|
||||
"updated_at": datetime.utcnow().isoformat(),
|
||||
"ttl": 1800,
|
||||
"ttl_expiry": now_ms + 900000, # 15 min remaining
|
||||
"refresh_schedule": None,
|
||||
"user": "jpmschweitzer"
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
results = await volatile_service.search(
|
||||
user="jpmschweitzer",
|
||||
query="weather rotterdam"
|
||||
)
|
||||
|
||||
assert len(results) == 1
|
||||
assert results[0].key == "rotterdam"
|
||||
assert results[0].namespace == "weather"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_success(self, volatile_service, mock_qdrant):
|
||||
"""Test successful delete."""
|
||||
mock_qdrant.delete_by_ids.return_value = 1
|
||||
|
||||
result = await volatile_service.delete("jpmschweitzer", "weather", "rotterdam")
|
||||
|
||||
assert result is True
|
||||
mock_qdrant.delete_by_ids.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_not_found(self, volatile_service, mock_qdrant):
|
||||
"""Test delete when record not found."""
|
||||
mock_qdrant.delete_by_ids.return_value = 0
|
||||
|
||||
result = await volatile_service.delete("jpmschweitzer", "weather", "nonexistent")
|
||||
|
||||
assert result is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_stats_empty(self, volatile_service, mock_qdrant):
|
||||
"""Test stats with no records."""
|
||||
mock_qdrant.collection_exists.return_value = False
|
||||
|
||||
stats = await volatile_service.get_stats("jpmschweitzer")
|
||||
|
||||
assert stats["total_records"] == 0
|
||||
assert stats["by_namespace"] == {}
|
||||
assert stats["scheduled_count"] == 0
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_stats_with_records(self, volatile_service, mock_qdrant):
|
||||
"""Test stats with records."""
|
||||
import time
|
||||
now_ms = int(time.time() * 1000)
|
||||
|
||||
mock_qdrant.scroll_all_points.return_value = [
|
||||
{"payload": {"namespace": "weather", "ttl_expiry": now_ms + 100000}},
|
||||
{"payload": {"namespace": "weather", "ttl_expiry": now_ms + 100000, "refresh_schedule": "0 * * * *"}},
|
||||
{"payload": {"namespace": "news", "ttl_expiry": now_ms + 100000}},
|
||||
{"payload": {"namespace": "weather", "ttl_expiry": now_ms - 100000}}, # Expired
|
||||
]
|
||||
|
||||
stats = await volatile_service.get_stats("jpmschweitzer")
|
||||
|
||||
assert stats["total_records"] == 3 # Excludes expired
|
||||
assert stats["by_namespace"]["weather"] == 2
|
||||
assert stats["by_namespace"]["news"] == 1
|
||||
assert stats["scheduled_count"] == 1
|
||||
assert stats["expired_count"] == 1
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_purge_expired(self, volatile_service, mock_qdrant):
|
||||
"""Test purging expired records."""
|
||||
mock_qdrant.delete_expired_vectors.return_value = 5
|
||||
|
||||
result = await volatile_service.purge_expired("jpmschweitzer")
|
||||
|
||||
assert result == 5
|
||||
mock_qdrant.delete_expired_vectors.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_purge_all_expired(self, volatile_service, mock_qdrant):
|
||||
"""Test purging expired from all collections."""
|
||||
mock_qdrant.get_volatile_collections.return_value = [
|
||||
"volatile_user1",
|
||||
"volatile_user2"
|
||||
]
|
||||
mock_qdrant.delete_expired_vectors.side_effect = [3, 2]
|
||||
|
||||
results = await volatile_service.purge_all_expired()
|
||||
|
||||
assert results["volatile_user1"] == 3
|
||||
assert results["volatile_user2"] == 2
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_scheduled(self, volatile_service, mock_qdrant):
|
||||
"""Test getting scheduled records."""
|
||||
import time
|
||||
now_ms = int(time.time() * 1000)
|
||||
|
||||
mock_qdrant.scroll_all_points.return_value = [
|
||||
{
|
||||
"payload": {
|
||||
"key": "nos-headlines",
|
||||
"namespace": "news",
|
||||
"raw_data": {"headlines": []},
|
||||
"source": "nos.nl",
|
||||
"created_at": datetime.utcnow().isoformat(),
|
||||
"updated_at": datetime.utcnow().isoformat(),
|
||||
"ttl": 3600,
|
||||
"ttl_expiry": now_ms + 1800000,
|
||||
"refresh_schedule": "0 */6 * * *",
|
||||
"user": "jpmschweitzer"
|
||||
}
|
||||
},
|
||||
{
|
||||
"payload": {
|
||||
"key": "rotterdam",
|
||||
"namespace": "weather",
|
||||
"raw_data": {"temperature": 18},
|
||||
"source": "openweathermap",
|
||||
"created_at": datetime.utcnow().isoformat(),
|
||||
"updated_at": datetime.utcnow().isoformat(),
|
||||
"ttl": 1800,
|
||||
"ttl_expiry": now_ms + 900000,
|
||||
"refresh_schedule": None, # Not scheduled
|
||||
"user": "jpmschweitzer"
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
scheduled = await volatile_service.get_scheduled("jpmschweitzer")
|
||||
|
||||
assert len(scheduled) == 1
|
||||
assert scheduled[0].key == "nos-headlines"
|
||||
assert scheduled[0].refresh_schedule == "0 */6 * * *"
|
||||
|
||||
|
||||
class TestVolatileCleanupEndpoint:
|
||||
"""Test volatile cleanup in maintenance router."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cleanup_volatile(self):
|
||||
"""Test volatile cleanup endpoint."""
|
||||
from src.routers.maintenance import cleanup_volatile, VolatileCleanupResponse
|
||||
|
||||
mock_qdrant = AsyncMock()
|
||||
mock_qdrant.get_volatile_collections = AsyncMock(return_value=[
|
||||
"volatile_user1",
|
||||
"volatile_user2"
|
||||
])
|
||||
mock_qdrant.delete_expired_vectors = AsyncMock(side_effect=[3, 2])
|
||||
|
||||
mock_ollama = AsyncMock()
|
||||
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.volatile_default_ttl = 3600
|
||||
|
||||
with patch('src.routers.maintenance.get_settings', return_value=mock_settings):
|
||||
result = await cleanup_volatile(
|
||||
qdrant=mock_qdrant,
|
||||
ollama=mock_ollama,
|
||||
api_key="test"
|
||||
)
|
||||
|
||||
assert result.success is True
|
||||
assert result.collections_processed == 2
|
||||
assert result.total_expired_purged == 5
|
||||
assert result.by_collection["volatile_user1"] == 3
|
||||
assert result.by_collection["volatile_user2"] == 2
|
||||
@@ -0,0 +1,50 @@
|
||||
|
||||
|
||||
#!/bin/bash
|
||||
# Library-Desk Server Startup Script
|
||||
|
||||
set -e
|
||||
|
||||
# Colors for output
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
RED='\033[0;31m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
echo -e "${GREEN}Starting Library-Desk server...${NC}"
|
||||
|
||||
# Check if port 8778 is already in use
|
||||
if lsof -Pi :8778 -sTCP:LISTEN -t >/dev/null 2>&1 ; then
|
||||
echo -e "${RED}Error: Port 8778 is already in use${NC}"
|
||||
echo "Run: lsof -i :8778 to see what's using it"
|
||||
echo "Or run: kill \$(lsof -t -i:8778) to stop it"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Activate virtual environment if not already activated
|
||||
if [ -z "$VIRTUAL_ENV" ]; then
|
||||
if [ -d ".venv" ]; then
|
||||
echo -e "${YELLOW}Activating virtual environment...${NC}"
|
||||
source .venv/bin/activate
|
||||
else
|
||||
echo -e "${RED}Error: Virtual environment not found${NC}"
|
||||
echo "Run: python -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# Create logs directory if it doesn't exist
|
||||
LOGS_DIR="logs"
|
||||
mkdir -p "$LOGS_DIR"
|
||||
|
||||
# Clear/create log file
|
||||
LOG_FILE="$LOGS_DIR/server.log"
|
||||
> "$LOG_FILE"
|
||||
echo -e "${YELLOW}Logs will be written to: ${LOG_FILE}${NC}"
|
||||
|
||||
# Start the server
|
||||
echo -e "${GREEN}Starting uvicorn server on http://tower-of-joy:8778${NC}"
|
||||
echo -e "${YELLOW}Press Ctrl+C to stop the server${NC}"
|
||||
echo ""
|
||||
|
||||
uvicorn src.main:app --reload --host 0.0.0.0 --port 8778 2>&1 | tee "$LOG_FILE"
|
||||
Reference in New Issue
Block a user