Compare commits

...
2 Commits
Author SHA1 Message Date
jpmschweitzerandClaude Opus 4.5 376284f90e fix: add content_extractor to smart-create endpoint
Build and Push / build (release) Successful in 30s
The POST /wiki/pages/smart-create endpoint was failing with 500
Internal Server Error because HybridRAGService.__init__() was
missing the required content_extractor parameter.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-16 09:31:23 +01:00
jpmschweitzerandClaude Opus 4.5 f095de1162 docs: add HybridRAG architecture documentation
Documents two-stage RRF, configuration options, and notes
potential vector search noise improvements for future reference.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-15 17:54:38 +01:00
4 changed files with 68 additions and 2 deletions
+6
View File
@@ -5,6 +5,12 @@ All notable changes to Library Desk will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [1.3.1] - 2025-12-16
### Fixed
- Smart create endpoint missing `content_extractor` dependency causing 500 errors on `POST /wiki/pages/smart-create`
## [1.3.0] - 2025-12-15
### Changed
+58
View File
@@ -0,0 +1,58 @@
# HybridRAG Architecture
## Overview
HybridRAG combines three search sources to provide comprehensive results:
- **Vector search** (Qdrant) - Semantic similarity via embeddings
- **Graph search** (Neo4j) - Entity relationships in knowledge graph
- **Web search** (SearXNG) - External web results via Trafilatura extraction
## Two-Stage RRF Fusion (v1.3.0+)
To ensure fair ranking between wiki and web results, we use a two-stage Reciprocal Rank Fusion:
```
Stage 1: Wiki Merge
vector results ─┬─→ Mini-RRF ─→ Unified wiki ranking
graph results ─┘
Stage 2: Final RRF
wiki (merged) ─┬─→ Final RRF ─→ Combined results
web results ─┘
```
**Why two stages?**
Previously, wiki pages found by BOTH vector and graph received double RRF contribution, giving them an unfair 2x advantage over web results. The two-stage approach:
1. Merges vector+graph into a single "wiki" source
2. Wiki's internal ranking still benefits from multi-source confirmation
3. Wiki and web compete as equals in final ranking
## Configuration
| Setting | Default | Description |
|---------|---------|-------------|
| `VECTOR_SIMILARITY_THRESHOLD` | 0.7 | Minimum similarity score for vector results |
| `HYBRID_RAG_VECTOR_LIMIT` | 10 | Max vector results |
| `HYBRID_RAG_GRAPH_LIMIT` | 10 | Max graph results |
| `HYBRID_RAG_WEB_LIMIT` | 5 | Max web results |
## Known Limitations & Future Improvements
### Vector Search Noise
**Status:** Open for improvement if needed after observation period.
Vector search may return generic category/index pages (e.g., "Reference", "Projects", "Places") with high similarity scores (~0.86). These pages often have similar boilerplate content leading to uniform scores.
**Potential solutions if this becomes problematic:**
1. **Raise threshold** - Increase `VECTOR_SIMILARITY_THRESHOLD` to 0.85+
2. **Page-type filtering** - Exclude pages tagged as category/index/stub
3. **Content length signal** - Penalize pages with minimal content
4. **Duplicate score detection** - Flag results with suspiciously identical scores
The LLM re-ranking phase typically demotes these low-quality results, so this may not require immediate action.
### Graph Search
Graph search uses only core keywords (no LLM-generated synonyms) to avoid false matches like "author" → "author2000". This is intentional - vector search handles semantic similarity via embeddings.
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "library-desk"
version = "1.3.0"
version = "1.3.1"
description = "Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and knowledge consolidation"
readme = "README.md"
requires-python = ">=3.12"
+3 -1
View File
@@ -24,7 +24,7 @@ from src.clients.neo4j_client import Neo4jClient
from src.clients.qdrant_client import QdrantClientWrapper
from src.clients.ollama_client import OllamaClient
from src.core.dependencies import (
WikiJSDep, Neo4jDep, QdrantDep, OllamaDep, SearXNGDep,
WikiJSDep, Neo4jDep, QdrantDep, OllamaDep, SearXNGDep, ContentExtractorDep,
verify_api_key, get_settings, get_hybrid_rag_service, get_ingestion_service
)
from src.core.multi_tenancy import DEFAULT_USER
@@ -180,6 +180,7 @@ async def smart_create_page(
qdrant_client: QdrantDep,
ollama_client: OllamaDep,
searxng_client: SearXNGDep,
content_extractor: ContentExtractorDep,
settings: Settings = Depends(get_settings),
api_key: str = Depends(verify_api_key)
):
@@ -225,6 +226,7 @@ async def smart_create_page(
graph_service=graph_service,
searxng_client=searxng_client,
ollama_client=ollama_client,
content_extractor=content_extractor,
settings=settings
)
wiki_page_writer = WikiPageWriter(ollama_client=ollama_client)