Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a1832e3245 | ||
|
|
5be31a5a00 | ||
|
|
376284f90e | ||
|
|
f095de1162 | ||
|
|
c359fcbcd8 | ||
|
|
464ec5380c |
@@ -23,6 +23,32 @@
|
|||||||
* **Update `CHANGELOG.md`** with every user-facing change.
|
* **Update `CHANGELOG.md`** with every user-facing change.
|
||||||
* Format: `## [Unreleased] - YYYY-MM-DD` followed by `### Added`, `### Changed`, or `### Fixed`.
|
* Format: `## [Unreleased] - YYYY-MM-DD` followed by `### Added`, `### Changed`, or `### Fixed`.
|
||||||
|
|
||||||
|
### 🚀 Release Flow
|
||||||
|
When changes are ready for deployment:
|
||||||
|
|
||||||
|
1. **Ask user if deploy cycle is desired **
|
||||||
|
|
||||||
|
2. **Update version** in `pyproject.toml`:
|
||||||
|
- Bug fixes: bump patch version (1.8.3 → 1.8.4)
|
||||||
|
- New features: bump minor version (1.8.4 → 1.9.0)
|
||||||
|
|
||||||
|
3. **Update CHANGELOG.md**:
|
||||||
|
- Move items from `[Unreleased]` to new version section
|
||||||
|
- Add release date: `## [1.8.4] - 2025-12-16`
|
||||||
|
|
||||||
|
4. **Commit and tag**:
|
||||||
|
```bash
|
||||||
|
git add -A
|
||||||
|
git commit -m "fix: description of changes"
|
||||||
|
git tag v1.8.4
|
||||||
|
git push origin main --tags
|
||||||
|
```
|
||||||
|
|
||||||
|
5. **CI/CD triggers automatically**:
|
||||||
|
- Gitea CI builds Docker image on new tag
|
||||||
|
- Watchtower pulls and deploys to production
|
||||||
|
- Verify deployment: `curl http://192.168.86.149:8000/health`
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 2. FastAPI Architecture & Best Practices
|
## 2. FastAPI Architecture & Best Practices
|
||||||
|
|||||||
@@ -5,6 +5,52 @@ All notable changes to Library Desk will be documented in this file.
|
|||||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||||
|
|
||||||
|
## [1.3.2] - 2025-12-22
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- **Consolidated Ollama model configuration** - All LLM operations now use single `OLLAMA_MODEL` environment variable
|
||||||
|
- Removed separate `reranker_model` setting
|
||||||
|
- HybridRAG re-ranking, consolidation analysis, and wiki page writing all use the same model
|
||||||
|
- Improves VRAM efficiency by keeping one model hot
|
||||||
|
- Added `OLLAMA_EMBEDDING_MODEL` environment variable for embedding model (previously overloaded `OLLAMA_MODEL`)
|
||||||
|
- Updated WikiPageWriter to accept settings instead of hardcoded model name
|
||||||
|
|
||||||
|
## [1.3.1] - 2025-12-16
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- Smart create endpoint missing `content_extractor` dependency causing 500 errors on `POST /wiki/pages/smart-create`
|
||||||
|
|
||||||
|
## [1.3.0] - 2025-12-15
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- **Two-Stage RRF Architecture** - Major refactor to level the playing field between wiki and web results
|
||||||
|
- Stage 1: Vector and graph results merged into single "wiki" ranking using mini-RRF
|
||||||
|
- Stage 2: Final RRF between wiki (single source) and web (single source)
|
||||||
|
- Wiki pages no longer get 2x advantage from appearing in both vector and graph searches
|
||||||
|
- Multi-source confirmation still determines wiki internal ranking
|
||||||
|
|
||||||
|
- **Skip synonyms in graph search** - LLM-generated synonyms (e.g., "author") no longer match unrelated graph entities (e.g., "author2000")
|
||||||
|
- Vector search still uses synonyms for semantic similarity
|
||||||
|
- Graph search uses only core keywords for exact entity matching
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- `VECTOR_SIMILARITY_THRESHOLD` config setting (default: 0.7) to filter weak vector matches
|
||||||
|
- Deduplication in graph search to prevent same document appearing multiple times
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- Graph search duplicate entity bug where same document could appear twice if entity linked multiple times
|
||||||
|
|
||||||
|
## [1.2.1] - 2025-12-15
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- HybridRAG router missing `content_extractor` dependency causing 500 errors on `/query/hybrid` endpoint
|
||||||
|
|
||||||
## [1.2.0] - 2025-12-15
|
## [1.2.0] - 2025-12-15
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|||||||
@@ -49,7 +49,8 @@ QDRANT_PORT=6333
|
|||||||
WIKIJS_URL=http://wiki:3000
|
WIKIJS_URL=http://wiki:3000
|
||||||
SEARXNG_URL=http://searxng:8080
|
SEARXNG_URL=http://searxng:8080
|
||||||
OLLAMA_URL=http://ollama:11434
|
OLLAMA_URL=http://ollama:11434
|
||||||
OLLAMA_MODEL=nomic-embed-text
|
OLLAMA_MODEL=mistral-nemo-large:latest
|
||||||
|
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
|
||||||
REDIS_HOST=redis-shared
|
REDIS_HOST=redis-shared
|
||||||
REDIS_PORT=6379
|
REDIS_PORT=6379
|
||||||
REDIS_DB=2
|
REDIS_DB=2
|
||||||
|
|||||||
@@ -0,0 +1,58 @@
|
|||||||
|
# HybridRAG Architecture
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
HybridRAG combines three search sources to provide comprehensive results:
|
||||||
|
- **Vector search** (Qdrant) - Semantic similarity via embeddings
|
||||||
|
- **Graph search** (Neo4j) - Entity relationships in knowledge graph
|
||||||
|
- **Web search** (SearXNG) - External web results via Trafilatura extraction
|
||||||
|
|
||||||
|
## Two-Stage RRF Fusion (v1.3.0+)
|
||||||
|
|
||||||
|
To ensure fair ranking between wiki and web results, we use a two-stage Reciprocal Rank Fusion:
|
||||||
|
|
||||||
|
```
|
||||||
|
Stage 1: Wiki Merge
|
||||||
|
vector results ─┬─→ Mini-RRF ─→ Unified wiki ranking
|
||||||
|
graph results ─┘
|
||||||
|
|
||||||
|
Stage 2: Final RRF
|
||||||
|
wiki (merged) ─┬─→ Final RRF ─→ Combined results
|
||||||
|
web results ─┘
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why two stages?**
|
||||||
|
|
||||||
|
Previously, wiki pages found by BOTH vector and graph received double RRF contribution, giving them an unfair 2x advantage over web results. The two-stage approach:
|
||||||
|
1. Merges vector+graph into a single "wiki" source
|
||||||
|
2. Wiki's internal ranking still benefits from multi-source confirmation
|
||||||
|
3. Wiki and web compete as equals in final ranking
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
| Setting | Default | Description |
|
||||||
|
|---------|---------|-------------|
|
||||||
|
| `VECTOR_SIMILARITY_THRESHOLD` | 0.7 | Minimum similarity score for vector results |
|
||||||
|
| `HYBRID_RAG_VECTOR_LIMIT` | 10 | Max vector results |
|
||||||
|
| `HYBRID_RAG_GRAPH_LIMIT` | 10 | Max graph results |
|
||||||
|
| `HYBRID_RAG_WEB_LIMIT` | 5 | Max web results |
|
||||||
|
|
||||||
|
## Known Limitations & Future Improvements
|
||||||
|
|
||||||
|
### Vector Search Noise
|
||||||
|
|
||||||
|
**Status:** Open for improvement if needed after observation period.
|
||||||
|
|
||||||
|
Vector search may return generic category/index pages (e.g., "Reference", "Projects", "Places") with high similarity scores (~0.86). These pages often have similar boilerplate content leading to uniform scores.
|
||||||
|
|
||||||
|
**Potential solutions if this becomes problematic:**
|
||||||
|
1. **Raise threshold** - Increase `VECTOR_SIMILARITY_THRESHOLD` to 0.85+
|
||||||
|
2. **Page-type filtering** - Exclude pages tagged as category/index/stub
|
||||||
|
3. **Content length signal** - Penalize pages with minimal content
|
||||||
|
4. **Duplicate score detection** - Flag results with suspiciously identical scores
|
||||||
|
|
||||||
|
The LLM re-ranking phase typically demotes these low-quality results, so this may not require immediate action.
|
||||||
|
|
||||||
|
### Graph Search
|
||||||
|
|
||||||
|
Graph search uses only core keywords (no LLM-generated synonyms) to avoid false matches like "author" → "author2000". This is intentional - vector search handles semantic similarity via embeddings.
|
||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
[project]
|
[project]
|
||||||
name = "library-desk"
|
name = "library-desk"
|
||||||
version = "1.2.0"
|
version = "1.3.2"
|
||||||
description = "Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and knowledge consolidation"
|
description = "Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and knowledge consolidation"
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.12"
|
requires-python = ">=3.12"
|
||||||
|
|||||||
+4
-3
@@ -62,16 +62,17 @@ class Settings(BaseSettings):
|
|||||||
# SearXNG Configuration
|
# SearXNG Configuration
|
||||||
searxng_url: str = Field(default="http://searxng:8080", description="SearXNG URL")
|
searxng_url: str = Field(default="http://searxng:8080", description="SearXNG URL")
|
||||||
|
|
||||||
# Ollama Configuration (for embeddings)
|
# Ollama Configuration
|
||||||
ollama_url: str = Field(default="http://ollama:11434", description="Ollama URL")
|
ollama_url: str = Field(default="http://ollama:11434", description="Ollama URL")
|
||||||
ollama_model: str = Field(default="nomic-embed-text", description="Ollama embedding model")
|
ollama_model: str = Field(default="mistral-nemo-large:latest", description="Ollama LLM model")
|
||||||
|
ollama_embedding_model: str = Field(default="nomic-embed-text", description="Ollama embedding model")
|
||||||
|
|
||||||
# HybridRAG Configuration
|
# HybridRAG Configuration
|
||||||
reranker_model: str = Field(default="mistral-nemo", description="Model for LLM re-ranking")
|
|
||||||
reranker_enabled: bool = Field(default=True, description="Enable LLM re-ranking")
|
reranker_enabled: bool = Field(default=True, description="Enable LLM re-ranking")
|
||||||
hybrid_rag_vector_limit: int = Field(default=10, ge=1, le=50, description="Vector search limit")
|
hybrid_rag_vector_limit: int = Field(default=10, ge=1, le=50, description="Vector search limit")
|
||||||
hybrid_rag_graph_limit: int = Field(default=10, ge=1, le=50, description="Graph search limit")
|
hybrid_rag_graph_limit: int = Field(default=10, ge=1, le=50, description="Graph search limit")
|
||||||
hybrid_rag_web_limit: int = Field(default=5, ge=1, le=20, description="Web search limit")
|
hybrid_rag_web_limit: int = Field(default=5, ge=1, le=20, description="Web search limit")
|
||||||
|
vector_similarity_threshold: float = Field(default=0.7, ge=0.0, le=1.0, description="Minimum similarity score for vector results")
|
||||||
|
|
||||||
# Entity Linking Fuzzy Matching Configuration
|
# Entity Linking Fuzzy Matching Configuration
|
||||||
entity_linking_min_confidence: float = Field(default=0.70, ge=0.0, le=1.0, description="Minimum confidence for entity-document matching")
|
entity_linking_min_confidence: float = Field(default=0.70, ge=0.0, le=1.0, description="Minimum confidence for entity-document matching")
|
||||||
|
|||||||
@@ -113,7 +113,7 @@ def get_ollama_client() -> OllamaClient:
|
|||||||
settings = get_settings()
|
settings = get_settings()
|
||||||
client = OllamaClient(
|
client = OllamaClient(
|
||||||
base_url=settings.ollama_url,
|
base_url=settings.ollama_url,
|
||||||
model=settings.ollama_model
|
model=settings.ollama_embedding_model
|
||||||
)
|
)
|
||||||
logger.debug("Created Ollama client instance")
|
logger.debug("Created Ollama client instance")
|
||||||
return client
|
return client
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ from src.clients.searxng_client import SearXNGClient
|
|||||||
from src.clients.ollama_client import OllamaClient
|
from src.clients.ollama_client import OllamaClient
|
||||||
from src.core.dependencies import (
|
from src.core.dependencies import (
|
||||||
Neo4jDep, WikiJSDep, QdrantDep, OllamaDep,
|
Neo4jDep, WikiJSDep, QdrantDep, OllamaDep,
|
||||||
SearXNGDep, verify_api_key, get_settings
|
SearXNGDep, ContentExtractorDep, verify_api_key, get_settings
|
||||||
)
|
)
|
||||||
from src.config import Settings
|
from src.config import Settings
|
||||||
|
|
||||||
@@ -32,6 +32,7 @@ def get_hybrid_rag_service(
|
|||||||
qdrant_client: QdrantDep,
|
qdrant_client: QdrantDep,
|
||||||
ollama_client: OllamaDep,
|
ollama_client: OllamaDep,
|
||||||
searxng_client: SearXNGDep,
|
searxng_client: SearXNGDep,
|
||||||
|
content_extractor: ContentExtractorDep,
|
||||||
settings: Settings = Depends(get_settings)
|
settings: Settings = Depends(get_settings)
|
||||||
) -> HybridRAGService:
|
) -> HybridRAGService:
|
||||||
"""Get HybridRAG service instance with all dependencies."""
|
"""Get HybridRAG service instance with all dependencies."""
|
||||||
@@ -48,6 +49,7 @@ def get_hybrid_rag_service(
|
|||||||
graph_service=graph_service,
|
graph_service=graph_service,
|
||||||
searxng_client=searxng_client,
|
searxng_client=searxng_client,
|
||||||
ollama_client=ollama_client,
|
ollama_client=ollama_client,
|
||||||
|
content_extractor=content_extractor,
|
||||||
settings=settings
|
settings=settings
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
+4
-2
@@ -24,7 +24,7 @@ from src.clients.neo4j_client import Neo4jClient
|
|||||||
from src.clients.qdrant_client import QdrantClientWrapper
|
from src.clients.qdrant_client import QdrantClientWrapper
|
||||||
from src.clients.ollama_client import OllamaClient
|
from src.clients.ollama_client import OllamaClient
|
||||||
from src.core.dependencies import (
|
from src.core.dependencies import (
|
||||||
WikiJSDep, Neo4jDep, QdrantDep, OllamaDep, SearXNGDep,
|
WikiJSDep, Neo4jDep, QdrantDep, OllamaDep, SearXNGDep, ContentExtractorDep,
|
||||||
verify_api_key, get_settings, get_hybrid_rag_service, get_ingestion_service
|
verify_api_key, get_settings, get_hybrid_rag_service, get_ingestion_service
|
||||||
)
|
)
|
||||||
from src.core.multi_tenancy import DEFAULT_USER
|
from src.core.multi_tenancy import DEFAULT_USER
|
||||||
@@ -180,6 +180,7 @@ async def smart_create_page(
|
|||||||
qdrant_client: QdrantDep,
|
qdrant_client: QdrantDep,
|
||||||
ollama_client: OllamaDep,
|
ollama_client: OllamaDep,
|
||||||
searxng_client: SearXNGDep,
|
searxng_client: SearXNGDep,
|
||||||
|
content_extractor: ContentExtractorDep,
|
||||||
settings: Settings = Depends(get_settings),
|
settings: Settings = Depends(get_settings),
|
||||||
api_key: str = Depends(verify_api_key)
|
api_key: str = Depends(verify_api_key)
|
||||||
):
|
):
|
||||||
@@ -225,9 +226,10 @@ async def smart_create_page(
|
|||||||
graph_service=graph_service,
|
graph_service=graph_service,
|
||||||
searxng_client=searxng_client,
|
searxng_client=searxng_client,
|
||||||
ollama_client=ollama_client,
|
ollama_client=ollama_client,
|
||||||
|
content_extractor=content_extractor,
|
||||||
settings=settings
|
settings=settings
|
||||||
)
|
)
|
||||||
wiki_page_writer = WikiPageWriter(ollama_client=ollama_client)
|
wiki_page_writer = WikiPageWriter(ollama_client=ollama_client, settings=settings)
|
||||||
|
|
||||||
# Step 1-5: Research + Generate + Create page
|
# Step 1-5: Research + Generate + Create page
|
||||||
page, research_data = await wiki_service.smart_create_page(
|
page, research_data = await wiki_service.smart_create_page(
|
||||||
|
|||||||
@@ -47,7 +47,7 @@ class ConsolidationService:
|
|||||||
self.ollama = ollama
|
self.ollama = ollama
|
||||||
self.wiki = wiki
|
self.wiki = wiki
|
||||||
self.settings = settings
|
self.settings = settings
|
||||||
self.wiki_page_writer = WikiPageWriter(ollama_client=ollama)
|
self.wiki_page_writer = WikiPageWriter(ollama_client=ollama, settings=settings)
|
||||||
self.ingestion_service = ingestion_service # Optional to avoid circular dependency
|
self.ingestion_service = ingestion_service # Optional to avoid circular dependency
|
||||||
|
|
||||||
async def consolidate_knowledge(
|
async def consolidate_knowledge(
|
||||||
@@ -465,7 +465,7 @@ JSON:"""
|
|||||||
# Call Ollama for analysis
|
# Call Ollama for analysis
|
||||||
response = await self.ollama.generate_text(
|
response = await self.ollama.generate_text(
|
||||||
prompt=prompt,
|
prompt=prompt,
|
||||||
model=self.settings.reranker_model, # Use mistral-nemo
|
model=self.settings.ollama_model,
|
||||||
stream=False
|
stream=False
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -1077,8 +1077,18 @@ Feel free to expand it with more details!
|
|||||||
search_query,
|
search_query,
|
||||||
{"terms": all_terms, "limit": limit}
|
{"terms": all_terms, "limit": limit}
|
||||||
)
|
)
|
||||||
logger.info(f"Graph search found {len(results)} documents")
|
|
||||||
return results
|
# Deduplicate by page_id (safety net for any edge cases)
|
||||||
|
seen_page_ids = set()
|
||||||
|
unique_results = []
|
||||||
|
for r in results:
|
||||||
|
page_id = r.get("page_id")
|
||||||
|
if page_id and page_id not in seen_page_ids:
|
||||||
|
seen_page_ids.add(page_id)
|
||||||
|
unique_results.append(r)
|
||||||
|
|
||||||
|
logger.info(f"Graph search found {len(unique_results)} unique documents (raw: {len(results)})")
|
||||||
|
return unique_results
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Graph document search failed: {e}", exc_info=True)
|
logger.error(f"Graph document search failed: {e}", exc_info=True)
|
||||||
return []
|
return []
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ HybridRAG service combining vector, graph, and web search.
|
|||||||
1. Parallel Retrieval - Vector + Graph + Web search
|
1. Parallel Retrieval - Vector + Graph + Web search
|
||||||
2. RRF Fusion - Merge results with Reciprocal Rank Fusion
|
2. RRF Fusion - Merge results with Reciprocal Rank Fusion
|
||||||
3. Enrichment - Add related dossiers via graph
|
3. Enrichment - Add related dossiers via graph
|
||||||
4. LLM Re-ranking - Re-rank with mistral-nemo
|
4. LLM Re-ranking - Re-rank with configured Ollama model
|
||||||
5. Context Formatting - Format for LLM consumption
|
5. Context Formatting - Format for LLM consumption
|
||||||
6. Persistence - Store for Librarian processing
|
6. Persistence - Store for Librarian processing
|
||||||
"""
|
"""
|
||||||
@@ -65,7 +65,7 @@ class HybridRAGService:
|
|||||||
self.ollama = ollama_client
|
self.ollama = ollama_client
|
||||||
self.content_extractor = content_extractor
|
self.content_extractor = content_extractor
|
||||||
self.settings = settings
|
self.settings = settings
|
||||||
self.reranker_model = settings.reranker_model
|
self.reranker_model = settings.ollama_model
|
||||||
|
|
||||||
async def search(
|
async def search(
|
||||||
self,
|
self,
|
||||||
@@ -105,14 +105,20 @@ class HybridRAGService:
|
|||||||
timing["graph_ms"] = raw_results.get("timing", {}).get("graph_ms", 0)
|
timing["graph_ms"] = raw_results.get("timing", {}).get("graph_ms", 0)
|
||||||
timing["web_ms"] = raw_results.get("timing", {}).get("web_ms", 0)
|
timing["web_ms"] = raw_results.get("timing", {}).get("web_ms", 0)
|
||||||
|
|
||||||
# Phase 2: RRF Fusion
|
# Phase 2: Two-Stage RRF Fusion
|
||||||
phase2_start = time.time()
|
phase2_start = time.time()
|
||||||
|
|
||||||
|
# Stage 1: Merge wiki sources (vector + graph) into single ranking
|
||||||
|
wiki_merged = self._merge_wiki_sources(
|
||||||
|
vector_results=raw_results.get("vector", []),
|
||||||
|
graph_results=raw_results.get("graph", []),
|
||||||
|
k=config.rrf_k
|
||||||
|
)
|
||||||
|
|
||||||
|
# Stage 2: Final RRF between wiki and web (equal footing)
|
||||||
fused_results = self._reciprocal_rank_fusion(
|
fused_results = self._reciprocal_rank_fusion(
|
||||||
results_by_source={
|
wiki_results=wiki_merged,
|
||||||
"vector": raw_results.get("vector", []),
|
web_results=raw_results.get("web", []),
|
||||||
"graph": raw_results.get("graph", []),
|
|
||||||
"web": raw_results.get("web", [])
|
|
||||||
},
|
|
||||||
k=config.rrf_k
|
k=config.rrf_k
|
||||||
)
|
)
|
||||||
timing["fusion_ms"] = (time.time() - phase2_start) * 1000
|
timing["fusion_ms"] = (time.time() - phase2_start) * 1000
|
||||||
@@ -288,7 +294,8 @@ JSON:"""
|
|||||||
response = await self.vector.search(
|
response = await self.vector.search(
|
||||||
query=query,
|
query=query,
|
||||||
user=user,
|
user=user,
|
||||||
limit=config.vector_limit
|
limit=config.vector_limit,
|
||||||
|
score_threshold=self.settings.vector_similarity_threshold
|
||||||
)
|
)
|
||||||
results = [
|
results = [
|
||||||
{
|
{
|
||||||
@@ -313,11 +320,19 @@ JSON:"""
|
|||||||
async def graph_search():
|
async def graph_search():
|
||||||
start = time.time()
|
start = time.time()
|
||||||
try:
|
try:
|
||||||
|
# Skip synonyms for graph search - only use core keywords
|
||||||
|
# Synonyms like "author" can match unrelated entities like "author2000"
|
||||||
|
graph_keywords = {
|
||||||
|
"core_keywords": keywords_data.get("core_keywords", []),
|
||||||
|
"entities": keywords_data.get("entities", []),
|
||||||
|
"synonyms": {}, # No synonyms for exact entity matching
|
||||||
|
"expansions": {}
|
||||||
|
}
|
||||||
results = await self.graph.search_documents(
|
results = await self.graph.search_documents(
|
||||||
query=query,
|
query=query,
|
||||||
user=user,
|
user=user,
|
||||||
limit=config.graph_limit,
|
limit=config.graph_limit,
|
||||||
keywords_data=keywords_data
|
keywords_data=graph_keywords
|
||||||
)
|
)
|
||||||
formatted = [
|
formatted = [
|
||||||
{
|
{
|
||||||
@@ -394,52 +409,134 @@ JSON:"""
|
|||||||
|
|
||||||
return output
|
return output
|
||||||
|
|
||||||
def _reciprocal_rank_fusion(
|
def _merge_wiki_sources(
|
||||||
self,
|
self,
|
||||||
results_by_source: Dict[str, List],
|
vector_results: List[Dict],
|
||||||
|
graph_results: List[Dict],
|
||||||
k: int = 60
|
k: int = 60
|
||||||
) -> List[Dict[str, Any]]:
|
) -> List[Dict[str, Any]]:
|
||||||
"""
|
"""
|
||||||
Phase 2: Merge results using Reciprocal Rank Fusion.
|
Stage 1: Merge vector and graph into single wiki ranking using RRF.
|
||||||
|
|
||||||
RRF formula: score = sum(1 / (k + rank)) for each source
|
Both sources search the same wiki pool, so we combine them before
|
||||||
|
final RRF with web to avoid double-counting wiki pages.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
results_by_source: Results from each source
|
vector_results: Results from vector search
|
||||||
|
graph_results: Results from graph search
|
||||||
k: RRF constant (default 60)
|
k: RRF constant (default 60)
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Merged and sorted results
|
Merged wiki results sorted by wiki RRF score
|
||||||
|
"""
|
||||||
|
wiki_scores = {}
|
||||||
|
|
||||||
|
# Process vector results
|
||||||
|
for rank, result in enumerate(vector_results, start=1):
|
||||||
|
page_id = result.get("page_id")
|
||||||
|
if not page_id:
|
||||||
|
continue
|
||||||
|
result_id = f"page_{page_id}"
|
||||||
|
|
||||||
|
if result_id not in wiki_scores:
|
||||||
|
wiki_scores[result_id] = {
|
||||||
|
"result": dict(result), # Copy to avoid mutation
|
||||||
|
"wiki_rrf_score": 0.0,
|
||||||
|
"found_by": []
|
||||||
|
}
|
||||||
|
|
||||||
|
wiki_scores[result_id]["wiki_rrf_score"] += 1 / (k + rank)
|
||||||
|
wiki_scores[result_id]["found_by"].append("vector")
|
||||||
|
|
||||||
|
# Process graph results
|
||||||
|
for rank, result in enumerate(graph_results, start=1):
|
||||||
|
page_id = result.get("page_id")
|
||||||
|
if not page_id:
|
||||||
|
continue
|
||||||
|
result_id = f"page_{page_id}"
|
||||||
|
|
||||||
|
if result_id not in wiki_scores:
|
||||||
|
wiki_scores[result_id] = {
|
||||||
|
"result": dict(result),
|
||||||
|
"wiki_rrf_score": 0.0,
|
||||||
|
"found_by": []
|
||||||
|
}
|
||||||
|
|
||||||
|
wiki_scores[result_id]["wiki_rrf_score"] += 1 / (k + rank)
|
||||||
|
wiki_scores[result_id]["found_by"].append("graph")
|
||||||
|
|
||||||
|
# Add graph metadata to existing result
|
||||||
|
wiki_scores[result_id]["result"]["entity_matches"] = result.get("entity_matches")
|
||||||
|
wiki_scores[result_id]["result"]["matched_entities"] = result.get("matched_entities")
|
||||||
|
|
||||||
|
# Sort by wiki RRF score
|
||||||
|
sorted_wiki = sorted(
|
||||||
|
wiki_scores.values(),
|
||||||
|
key=lambda x: x["wiki_rrf_score"],
|
||||||
|
reverse=True
|
||||||
|
)
|
||||||
|
|
||||||
|
# Return merged results with wiki ranking
|
||||||
|
merged = []
|
||||||
|
for wiki_rank, item in enumerate(sorted_wiki, start=1):
|
||||||
|
merged.append({
|
||||||
|
**item["result"],
|
||||||
|
"wiki_rank": wiki_rank,
|
||||||
|
"wiki_rrf_score": item["wiki_rrf_score"],
|
||||||
|
"found_by": item["found_by"],
|
||||||
|
"source": "wiki"
|
||||||
|
})
|
||||||
|
|
||||||
|
logger.info(f"Wiki merge: {len(merged)} unique pages from vector+graph")
|
||||||
|
return merged
|
||||||
|
|
||||||
|
def _reciprocal_rank_fusion(
|
||||||
|
self,
|
||||||
|
wiki_results: List[Dict],
|
||||||
|
web_results: List[Dict],
|
||||||
|
k: int = 60
|
||||||
|
) -> List[Dict[str, Any]]:
|
||||||
|
"""
|
||||||
|
Stage 2: Final RRF between wiki (single source) and web.
|
||||||
|
|
||||||
|
Wiki results are pre-merged from vector+graph, so wiki and web
|
||||||
|
now compete on equal footing.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
wiki_results: Pre-merged wiki results from _merge_wiki_sources()
|
||||||
|
web_results: Results from web search
|
||||||
|
k: RRF constant (default 60)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Final merged and sorted results
|
||||||
"""
|
"""
|
||||||
rrf_scores = {}
|
rrf_scores = {}
|
||||||
|
|
||||||
for source, results in results_by_source.items():
|
# Wiki results (single source, already merged)
|
||||||
for rank, result in enumerate(results, start=1):
|
for rank, result in enumerate(wiki_results, start=1):
|
||||||
# Use page_id for wiki results, url hash for web results
|
page_id = result.get("page_id")
|
||||||
if result.get("page_id"):
|
if not page_id:
|
||||||
result_id = f"page_{result['page_id']}"
|
continue
|
||||||
elif result.get("url"):
|
result_id = f"page_{page_id}"
|
||||||
result_id = f"url_{hash(result['url'])}"
|
rrf_scores[result_id] = {
|
||||||
else:
|
"result": result,
|
||||||
continue # Skip results without ID
|
"rrf_score": 1 / (k + rank),
|
||||||
|
"sources": result.get("found_by", ["wiki"]),
|
||||||
|
"source_type": "wiki"
|
||||||
|
}
|
||||||
|
|
||||||
if result_id not in rrf_scores:
|
# Web results (single source)
|
||||||
rrf_scores[result_id] = {
|
for rank, result in enumerate(web_results, start=1):
|
||||||
"result": result,
|
url = result.get("url")
|
||||||
"rrf_score": 0.0,
|
if not url:
|
||||||
"sources": [],
|
continue
|
||||||
"source_type": source
|
result_id = f"url_{hash(url)}"
|
||||||
}
|
rrf_scores[result_id] = {
|
||||||
|
"result": result,
|
||||||
# RRF formula: sum of 1/(k + rank) across sources
|
"rrf_score": 1 / (k + rank),
|
||||||
rrf_scores[result_id]["rrf_score"] += 1 / (k + rank)
|
"sources": ["web"],
|
||||||
rrf_scores[result_id]["sources"].append(source)
|
"source_type": "web"
|
||||||
|
}
|
||||||
# If result appears in multiple sources, update source_type
|
|
||||||
if len(rrf_scores[result_id]["sources"]) > 1:
|
|
||||||
rrf_scores[result_id]["source_type"] = "+".join(
|
|
||||||
sorted(set(rrf_scores[result_id]["sources"]))
|
|
||||||
)
|
|
||||||
|
|
||||||
# Sort by RRF score descending
|
# Sort by RRF score descending
|
||||||
sorted_results = sorted(
|
sorted_results = sorted(
|
||||||
@@ -448,7 +545,7 @@ JSON:"""
|
|||||||
reverse=True
|
reverse=True
|
||||||
)
|
)
|
||||||
|
|
||||||
logger.info(f"RRF fusion: {len(sorted_results)} unique results from {len(results_by_source)} sources")
|
logger.info(f"Final RRF: {len(sorted_results)} results (wiki + web)")
|
||||||
|
|
||||||
return sorted_results
|
return sorted_results
|
||||||
|
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
"""
|
"""
|
||||||
Intelligent Wiki Page Writer Service
|
Intelligent Wiki Page Writer Service
|
||||||
|
|
||||||
Uses LLM (mistral-nemo) to create and reconstruct wiki pages with:
|
Uses LLM to create and reconstruct wiki pages with:
|
||||||
- Holistic content restructuring
|
- Holistic content restructuring
|
||||||
- Zero fact loss (unless superseded)
|
- Zero fact loss (unless superseded)
|
||||||
- Conflict detection and flagging
|
- Conflict detection and flagging
|
||||||
@@ -25,15 +25,16 @@ class WikiPageWriter:
|
|||||||
Intelligent wiki page writer using LLM for content generation and restructuring.
|
Intelligent wiki page writer using LLM for content generation and restructuring.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, ollama_client):
|
def __init__(self, ollama_client, settings):
|
||||||
"""
|
"""
|
||||||
Initialize wiki page writer.
|
Initialize wiki page writer.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
ollama_client: OllamaClient for LLM operations
|
ollama_client: OllamaClient for LLM operations
|
||||||
|
settings: Application settings
|
||||||
"""
|
"""
|
||||||
self.ollama = ollama_client
|
self.ollama = ollama_client
|
||||||
self.model = "mistral-nemo" # Default model for writing
|
self.model = settings.ollama_model
|
||||||
|
|
||||||
async def create_page(
|
async def create_page(
|
||||||
self,
|
self,
|
||||||
|
|||||||
+38
-37
@@ -219,53 +219,54 @@ async def test_vector_data(vector_service, test_wiki_page):
|
|||||||
# ============================================================================
|
# ============================================================================
|
||||||
|
|
||||||
class TestRRFFusion:
|
class TestRRFFusion:
|
||||||
"""Test Reciprocal Rank Fusion algorithm."""
|
"""Test two-stage Reciprocal Rank Fusion algorithm."""
|
||||||
|
|
||||||
def test_rrf_single_source(self, hybrid_rag_service):
|
def test_wiki_merge_single_source(self, hybrid_rag_service):
|
||||||
"""Test RRF with single source."""
|
"""Test wiki merge with single source (vector only)."""
|
||||||
results_by_source = {
|
vector_results = [
|
||||||
"vector": [
|
{"page_id": 1, "title": "Doc 1", "content": "test"},
|
||||||
{"page_id": 1, "title": "Doc 1", "content": "test"},
|
{"page_id": 2, "title": "Doc 2", "content": "test"}
|
||||||
{"page_id": 2, "title": "Doc 2", "content": "test"}
|
]
|
||||||
]
|
|
||||||
}
|
|
||||||
|
|
||||||
fused = hybrid_rag_service._reciprocal_rank_fusion(results_by_source, k=60)
|
merged = hybrid_rag_service._merge_wiki_sources(vector_results, [], k=60)
|
||||||
|
|
||||||
assert len(fused) == 2
|
assert len(merged) == 2
|
||||||
assert fused[0]["rrf_score"] > fused[1]["rrf_score"] # Rank 1 > Rank 2
|
assert merged[0]["wiki_rrf_score"] > merged[1]["wiki_rrf_score"] # Rank 1 > Rank 2
|
||||||
assert fused[0]["sources"] == ["vector"]
|
assert merged[0]["found_by"] == ["vector"]
|
||||||
|
|
||||||
def test_rrf_multiple_sources_same_doc(self, hybrid_rag_service):
|
def test_wiki_merge_multiple_sources_same_doc(self, hybrid_rag_service):
|
||||||
"""Test RRF with same document from multiple sources."""
|
"""Test wiki merge with same document from vector and graph."""
|
||||||
results_by_source = {
|
vector_results = [{"page_id": 1, "title": "Doc 1", "content": "test"}]
|
||||||
"vector": [{"page_id": 1, "title": "Doc 1", "content": "test"}],
|
graph_results = [{"page_id": 1, "title": "Doc 1", "content": ""}]
|
||||||
"graph": [{"page_id": 1, "title": "Doc 1", "content": ""}],
|
|
||||||
}
|
|
||||||
|
|
||||||
fused = hybrid_rag_service._reciprocal_rank_fusion(results_by_source, k=60)
|
merged = hybrid_rag_service._merge_wiki_sources(vector_results, graph_results, k=60)
|
||||||
|
|
||||||
assert len(fused) == 1 # Deduplicated
|
assert len(merged) == 1 # Deduplicated
|
||||||
assert len(fused[0]["sources"]) == 2 # Both sources
|
assert len(merged[0]["found_by"]) == 2 # Both sources
|
||||||
assert "vector" in fused[0]["sources"]
|
assert "vector" in merged[0]["found_by"]
|
||||||
assert "graph" in fused[0]["sources"]
|
assert "graph" in merged[0]["found_by"]
|
||||||
# RRF score should be sum: 1/(60+1) + 1/(60+1)
|
# Wiki RRF score should be sum: 1/(60+1) + 1/(60+1)
|
||||||
expected_score = 1/61 + 1/61
|
expected_score = 1/61 + 1/61
|
||||||
assert abs(fused[0]["rrf_score"] - expected_score) < 0.001
|
assert abs(merged[0]["wiki_rrf_score"] - expected_score) < 0.001
|
||||||
|
|
||||||
def test_rrf_web_results(self, hybrid_rag_service):
|
def test_final_rrf_wiki_and_web(self, hybrid_rag_service):
|
||||||
"""Test RRF with web results (URL-based)."""
|
"""Test final RRF between wiki and web results."""
|
||||||
results_by_source = {
|
# Pre-merged wiki results
|
||||||
"web": [
|
wiki_results = [
|
||||||
{"url": "https://example.com/1", "title": "Web 1", "content": "test"},
|
{"page_id": 1, "title": "Wiki 1", "content": "test", "found_by": ["vector"]}
|
||||||
{"url": "https://example.com/2", "title": "Web 2", "content": "test"}
|
]
|
||||||
]
|
web_results = [
|
||||||
}
|
{"url": "https://example.com/1", "title": "Web 1", "content": "test"},
|
||||||
|
{"url": "https://example.com/2", "title": "Web 2", "content": "test"}
|
||||||
|
]
|
||||||
|
|
||||||
fused = hybrid_rag_service._reciprocal_rank_fusion(results_by_source, k=60)
|
fused = hybrid_rag_service._reciprocal_rank_fusion(wiki_results, web_results, k=60)
|
||||||
|
|
||||||
assert len(fused) == 2
|
assert len(fused) == 3
|
||||||
assert fused[0]["result"]["url"] == "https://example.com/1"
|
# Wiki rank 1 and web rank 1 should have same RRF score
|
||||||
|
wiki_score = next(r["rrf_score"] for r in fused if r["source_type"] == "wiki")
|
||||||
|
web_score = next(r["rrf_score"] for r in fused if r["source_type"] == "web")
|
||||||
|
assert abs(wiki_score - web_score) < 0.001 # Equal footing
|
||||||
|
|
||||||
|
|
||||||
class TestContextFormatting:
|
class TestContextFormatting:
|
||||||
|
|||||||
Reference in New Issue
Block a user