Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
376284f90e | ||
|
|
f095de1162 |
@@ -5,6 +5,12 @@ All notable changes to Library Desk will be documented in this file.
|
|||||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||||
|
|
||||||
|
## [1.3.1] - 2025-12-16
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- Smart create endpoint missing `content_extractor` dependency causing 500 errors on `POST /wiki/pages/smart-create`
|
||||||
|
|
||||||
## [1.3.0] - 2025-12-15
|
## [1.3.0] - 2025-12-15
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
|
|||||||
@@ -0,0 +1,58 @@
|
|||||||
|
# HybridRAG Architecture
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
HybridRAG combines three search sources to provide comprehensive results:
|
||||||
|
- **Vector search** (Qdrant) - Semantic similarity via embeddings
|
||||||
|
- **Graph search** (Neo4j) - Entity relationships in knowledge graph
|
||||||
|
- **Web search** (SearXNG) - External web results via Trafilatura extraction
|
||||||
|
|
||||||
|
## Two-Stage RRF Fusion (v1.3.0+)
|
||||||
|
|
||||||
|
To ensure fair ranking between wiki and web results, we use a two-stage Reciprocal Rank Fusion:
|
||||||
|
|
||||||
|
```
|
||||||
|
Stage 1: Wiki Merge
|
||||||
|
vector results ─┬─→ Mini-RRF ─→ Unified wiki ranking
|
||||||
|
graph results ─┘
|
||||||
|
|
||||||
|
Stage 2: Final RRF
|
||||||
|
wiki (merged) ─┬─→ Final RRF ─→ Combined results
|
||||||
|
web results ─┘
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why two stages?**
|
||||||
|
|
||||||
|
Previously, wiki pages found by BOTH vector and graph received double RRF contribution, giving them an unfair 2x advantage over web results. The two-stage approach:
|
||||||
|
1. Merges vector+graph into a single "wiki" source
|
||||||
|
2. Wiki's internal ranking still benefits from multi-source confirmation
|
||||||
|
3. Wiki and web compete as equals in final ranking
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
| Setting | Default | Description |
|
||||||
|
|---------|---------|-------------|
|
||||||
|
| `VECTOR_SIMILARITY_THRESHOLD` | 0.7 | Minimum similarity score for vector results |
|
||||||
|
| `HYBRID_RAG_VECTOR_LIMIT` | 10 | Max vector results |
|
||||||
|
| `HYBRID_RAG_GRAPH_LIMIT` | 10 | Max graph results |
|
||||||
|
| `HYBRID_RAG_WEB_LIMIT` | 5 | Max web results |
|
||||||
|
|
||||||
|
## Known Limitations & Future Improvements
|
||||||
|
|
||||||
|
### Vector Search Noise
|
||||||
|
|
||||||
|
**Status:** Open for improvement if needed after observation period.
|
||||||
|
|
||||||
|
Vector search may return generic category/index pages (e.g., "Reference", "Projects", "Places") with high similarity scores (~0.86). These pages often have similar boilerplate content leading to uniform scores.
|
||||||
|
|
||||||
|
**Potential solutions if this becomes problematic:**
|
||||||
|
1. **Raise threshold** - Increase `VECTOR_SIMILARITY_THRESHOLD` to 0.85+
|
||||||
|
2. **Page-type filtering** - Exclude pages tagged as category/index/stub
|
||||||
|
3. **Content length signal** - Penalize pages with minimal content
|
||||||
|
4. **Duplicate score detection** - Flag results with suspiciously identical scores
|
||||||
|
|
||||||
|
The LLM re-ranking phase typically demotes these low-quality results, so this may not require immediate action.
|
||||||
|
|
||||||
|
### Graph Search
|
||||||
|
|
||||||
|
Graph search uses only core keywords (no LLM-generated synonyms) to avoid false matches like "author" → "author2000". This is intentional - vector search handles semantic similarity via embeddings.
|
||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
[project]
|
[project]
|
||||||
name = "library-desk"
|
name = "library-desk"
|
||||||
version = "1.3.0"
|
version = "1.3.1"
|
||||||
description = "Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and knowledge consolidation"
|
description = "Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and knowledge consolidation"
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.12"
|
requires-python = ">=3.12"
|
||||||
|
|||||||
+3
-1
@@ -24,7 +24,7 @@ from src.clients.neo4j_client import Neo4jClient
|
|||||||
from src.clients.qdrant_client import QdrantClientWrapper
|
from src.clients.qdrant_client import QdrantClientWrapper
|
||||||
from src.clients.ollama_client import OllamaClient
|
from src.clients.ollama_client import OllamaClient
|
||||||
from src.core.dependencies import (
|
from src.core.dependencies import (
|
||||||
WikiJSDep, Neo4jDep, QdrantDep, OllamaDep, SearXNGDep,
|
WikiJSDep, Neo4jDep, QdrantDep, OllamaDep, SearXNGDep, ContentExtractorDep,
|
||||||
verify_api_key, get_settings, get_hybrid_rag_service, get_ingestion_service
|
verify_api_key, get_settings, get_hybrid_rag_service, get_ingestion_service
|
||||||
)
|
)
|
||||||
from src.core.multi_tenancy import DEFAULT_USER
|
from src.core.multi_tenancy import DEFAULT_USER
|
||||||
@@ -180,6 +180,7 @@ async def smart_create_page(
|
|||||||
qdrant_client: QdrantDep,
|
qdrant_client: QdrantDep,
|
||||||
ollama_client: OllamaDep,
|
ollama_client: OllamaDep,
|
||||||
searxng_client: SearXNGDep,
|
searxng_client: SearXNGDep,
|
||||||
|
content_extractor: ContentExtractorDep,
|
||||||
settings: Settings = Depends(get_settings),
|
settings: Settings = Depends(get_settings),
|
||||||
api_key: str = Depends(verify_api_key)
|
api_key: str = Depends(verify_api_key)
|
||||||
):
|
):
|
||||||
@@ -225,6 +226,7 @@ async def smart_create_page(
|
|||||||
graph_service=graph_service,
|
graph_service=graph_service,
|
||||||
searxng_client=searxng_client,
|
searxng_client=searxng_client,
|
||||||
ollama_client=ollama_client,
|
ollama_client=ollama_client,
|
||||||
|
content_extractor=content_extractor,
|
||||||
settings=settings
|
settings=settings
|
||||||
)
|
)
|
||||||
wiki_page_writer = WikiPageWriter(ollama_client=ollama_client)
|
wiki_page_writer = WikiPageWriter(ollama_client=ollama_client)
|
||||||
|
|||||||
Reference in New Issue
Block a user