Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
376284f90e | ||
|
|
f095de1162 |
@@ -5,6 +5,12 @@ All notable changes to Library Desk will be documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [1.3.1] - 2025-12-16
|
||||
|
||||
### Fixed
|
||||
|
||||
- Smart create endpoint missing `content_extractor` dependency causing 500 errors on `POST /wiki/pages/smart-create`
|
||||
|
||||
## [1.3.0] - 2025-12-15
|
||||
|
||||
### Changed
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
# HybridRAG Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
HybridRAG combines three search sources to provide comprehensive results:
|
||||
- **Vector search** (Qdrant) - Semantic similarity via embeddings
|
||||
- **Graph search** (Neo4j) - Entity relationships in knowledge graph
|
||||
- **Web search** (SearXNG) - External web results via Trafilatura extraction
|
||||
|
||||
## Two-Stage RRF Fusion (v1.3.0+)
|
||||
|
||||
To ensure fair ranking between wiki and web results, we use a two-stage Reciprocal Rank Fusion:
|
||||
|
||||
```
|
||||
Stage 1: Wiki Merge
|
||||
vector results ─┬─→ Mini-RRF ─→ Unified wiki ranking
|
||||
graph results ─┘
|
||||
|
||||
Stage 2: Final RRF
|
||||
wiki (merged) ─┬─→ Final RRF ─→ Combined results
|
||||
web results ─┘
|
||||
```
|
||||
|
||||
**Why two stages?**
|
||||
|
||||
Previously, wiki pages found by BOTH vector and graph received double RRF contribution, giving them an unfair 2x advantage over web results. The two-stage approach:
|
||||
1. Merges vector+graph into a single "wiki" source
|
||||
2. Wiki's internal ranking still benefits from multi-source confirmation
|
||||
3. Wiki and web compete as equals in final ranking
|
||||
|
||||
## Configuration
|
||||
|
||||
| Setting | Default | Description |
|
||||
|---------|---------|-------------|
|
||||
| `VECTOR_SIMILARITY_THRESHOLD` | 0.7 | Minimum similarity score for vector results |
|
||||
| `HYBRID_RAG_VECTOR_LIMIT` | 10 | Max vector results |
|
||||
| `HYBRID_RAG_GRAPH_LIMIT` | 10 | Max graph results |
|
||||
| `HYBRID_RAG_WEB_LIMIT` | 5 | Max web results |
|
||||
|
||||
## Known Limitations & Future Improvements
|
||||
|
||||
### Vector Search Noise
|
||||
|
||||
**Status:** Open for improvement if needed after observation period.
|
||||
|
||||
Vector search may return generic category/index pages (e.g., "Reference", "Projects", "Places") with high similarity scores (~0.86). These pages often have similar boilerplate content leading to uniform scores.
|
||||
|
||||
**Potential solutions if this becomes problematic:**
|
||||
1. **Raise threshold** - Increase `VECTOR_SIMILARITY_THRESHOLD` to 0.85+
|
||||
2. **Page-type filtering** - Exclude pages tagged as category/index/stub
|
||||
3. **Content length signal** - Penalize pages with minimal content
|
||||
4. **Duplicate score detection** - Flag results with suspiciously identical scores
|
||||
|
||||
The LLM re-ranking phase typically demotes these low-quality results, so this may not require immediate action.
|
||||
|
||||
### Graph Search
|
||||
|
||||
Graph search uses only core keywords (no LLM-generated synonyms) to avoid false matches like "author" → "author2000". This is intentional - vector search handles semantic similarity via embeddings.
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "library-desk"
|
||||
version = "1.3.0"
|
||||
version = "1.3.1"
|
||||
description = "Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and knowledge consolidation"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
|
||||
+3
-1
@@ -24,7 +24,7 @@ from src.clients.neo4j_client import Neo4jClient
|
||||
from src.clients.qdrant_client import QdrantClientWrapper
|
||||
from src.clients.ollama_client import OllamaClient
|
||||
from src.core.dependencies import (
|
||||
WikiJSDep, Neo4jDep, QdrantDep, OllamaDep, SearXNGDep,
|
||||
WikiJSDep, Neo4jDep, QdrantDep, OllamaDep, SearXNGDep, ContentExtractorDep,
|
||||
verify_api_key, get_settings, get_hybrid_rag_service, get_ingestion_service
|
||||
)
|
||||
from src.core.multi_tenancy import DEFAULT_USER
|
||||
@@ -180,6 +180,7 @@ async def smart_create_page(
|
||||
qdrant_client: QdrantDep,
|
||||
ollama_client: OllamaDep,
|
||||
searxng_client: SearXNGDep,
|
||||
content_extractor: ContentExtractorDep,
|
||||
settings: Settings = Depends(get_settings),
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
@@ -225,6 +226,7 @@ async def smart_create_page(
|
||||
graph_service=graph_service,
|
||||
searxng_client=searxng_client,
|
||||
ollama_client=ollama_client,
|
||||
content_extractor=content_extractor,
|
||||
settings=settings
|
||||
)
|
||||
wiki_page_writer = WikiPageWriter(ollama_client=ollama_client)
|
||||
|
||||
Reference in New Issue
Block a user