Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a1832e3245 | ||
|
|
5be31a5a00 | ||
|
|
376284f90e | ||
|
|
f095de1162 |
@@ -23,6 +23,32 @@
|
||||
* **Update `CHANGELOG.md`** with every user-facing change.
|
||||
* Format: `## [Unreleased] - YYYY-MM-DD` followed by `### Added`, `### Changed`, or `### Fixed`.
|
||||
|
||||
### 🚀 Release Flow
|
||||
When changes are ready for deployment:
|
||||
|
||||
1. **Ask user if deploy cycle is desired **
|
||||
|
||||
2. **Update version** in `pyproject.toml`:
|
||||
- Bug fixes: bump patch version (1.8.3 → 1.8.4)
|
||||
- New features: bump minor version (1.8.4 → 1.9.0)
|
||||
|
||||
3. **Update CHANGELOG.md**:
|
||||
- Move items from `[Unreleased]` to new version section
|
||||
- Add release date: `## [1.8.4] - 2025-12-16`
|
||||
|
||||
4. **Commit and tag**:
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "fix: description of changes"
|
||||
git tag v1.8.4
|
||||
git push origin main --tags
|
||||
```
|
||||
|
||||
5. **CI/CD triggers automatically**:
|
||||
- Gitea CI builds Docker image on new tag
|
||||
- Watchtower pulls and deploys to production
|
||||
- Verify deployment: `curl http://192.168.86.149:8000/health`
|
||||
|
||||
---
|
||||
|
||||
## 2. FastAPI Architecture & Best Practices
|
||||
|
||||
@@ -5,6 +5,23 @@ All notable changes to Library Desk will be documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [1.3.2] - 2025-12-22
|
||||
|
||||
### Changed
|
||||
|
||||
- **Consolidated Ollama model configuration** - All LLM operations now use single `OLLAMA_MODEL` environment variable
|
||||
- Removed separate `reranker_model` setting
|
||||
- HybridRAG re-ranking, consolidation analysis, and wiki page writing all use the same model
|
||||
- Improves VRAM efficiency by keeping one model hot
|
||||
- Added `OLLAMA_EMBEDDING_MODEL` environment variable for embedding model (previously overloaded `OLLAMA_MODEL`)
|
||||
- Updated WikiPageWriter to accept settings instead of hardcoded model name
|
||||
|
||||
## [1.3.1] - 2025-12-16
|
||||
|
||||
### Fixed
|
||||
|
||||
- Smart create endpoint missing `content_extractor` dependency causing 500 errors on `POST /wiki/pages/smart-create`
|
||||
|
||||
## [1.3.0] - 2025-12-15
|
||||
|
||||
### Changed
|
||||
|
||||
@@ -49,7 +49,8 @@ QDRANT_PORT=6333
|
||||
WIKIJS_URL=http://wiki:3000
|
||||
SEARXNG_URL=http://searxng:8080
|
||||
OLLAMA_URL=http://ollama:11434
|
||||
OLLAMA_MODEL=nomic-embed-text
|
||||
OLLAMA_MODEL=mistral-nemo-large:latest
|
||||
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
|
||||
REDIS_HOST=redis-shared
|
||||
REDIS_PORT=6379
|
||||
REDIS_DB=2
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
# HybridRAG Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
HybridRAG combines three search sources to provide comprehensive results:
|
||||
- **Vector search** (Qdrant) - Semantic similarity via embeddings
|
||||
- **Graph search** (Neo4j) - Entity relationships in knowledge graph
|
||||
- **Web search** (SearXNG) - External web results via Trafilatura extraction
|
||||
|
||||
## Two-Stage RRF Fusion (v1.3.0+)
|
||||
|
||||
To ensure fair ranking between wiki and web results, we use a two-stage Reciprocal Rank Fusion:
|
||||
|
||||
```
|
||||
Stage 1: Wiki Merge
|
||||
vector results ─┬─→ Mini-RRF ─→ Unified wiki ranking
|
||||
graph results ─┘
|
||||
|
||||
Stage 2: Final RRF
|
||||
wiki (merged) ─┬─→ Final RRF ─→ Combined results
|
||||
web results ─┘
|
||||
```
|
||||
|
||||
**Why two stages?**
|
||||
|
||||
Previously, wiki pages found by BOTH vector and graph received double RRF contribution, giving them an unfair 2x advantage over web results. The two-stage approach:
|
||||
1. Merges vector+graph into a single "wiki" source
|
||||
2. Wiki's internal ranking still benefits from multi-source confirmation
|
||||
3. Wiki and web compete as equals in final ranking
|
||||
|
||||
## Configuration
|
||||
|
||||
| Setting | Default | Description |
|
||||
|---------|---------|-------------|
|
||||
| `VECTOR_SIMILARITY_THRESHOLD` | 0.7 | Minimum similarity score for vector results |
|
||||
| `HYBRID_RAG_VECTOR_LIMIT` | 10 | Max vector results |
|
||||
| `HYBRID_RAG_GRAPH_LIMIT` | 10 | Max graph results |
|
||||
| `HYBRID_RAG_WEB_LIMIT` | 5 | Max web results |
|
||||
|
||||
## Known Limitations & Future Improvements
|
||||
|
||||
### Vector Search Noise
|
||||
|
||||
**Status:** Open for improvement if needed after observation period.
|
||||
|
||||
Vector search may return generic category/index pages (e.g., "Reference", "Projects", "Places") with high similarity scores (~0.86). These pages often have similar boilerplate content leading to uniform scores.
|
||||
|
||||
**Potential solutions if this becomes problematic:**
|
||||
1. **Raise threshold** - Increase `VECTOR_SIMILARITY_THRESHOLD` to 0.85+
|
||||
2. **Page-type filtering** - Exclude pages tagged as category/index/stub
|
||||
3. **Content length signal** - Penalize pages with minimal content
|
||||
4. **Duplicate score detection** - Flag results with suspiciously identical scores
|
||||
|
||||
The LLM re-ranking phase typically demotes these low-quality results, so this may not require immediate action.
|
||||
|
||||
### Graph Search
|
||||
|
||||
Graph search uses only core keywords (no LLM-generated synonyms) to avoid false matches like "author" → "author2000". This is intentional - vector search handles semantic similarity via embeddings.
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "library-desk"
|
||||
version = "1.3.0"
|
||||
version = "1.3.2"
|
||||
description = "Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and knowledge consolidation"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
|
||||
+3
-3
@@ -62,12 +62,12 @@ class Settings(BaseSettings):
|
||||
# SearXNG Configuration
|
||||
searxng_url: str = Field(default="http://searxng:8080", description="SearXNG URL")
|
||||
|
||||
# Ollama Configuration (for embeddings)
|
||||
# Ollama Configuration
|
||||
ollama_url: str = Field(default="http://ollama:11434", description="Ollama URL")
|
||||
ollama_model: str = Field(default="nomic-embed-text", description="Ollama embedding model")
|
||||
ollama_model: str = Field(default="mistral-nemo-large:latest", description="Ollama LLM model")
|
||||
ollama_embedding_model: str = Field(default="nomic-embed-text", description="Ollama embedding model")
|
||||
|
||||
# HybridRAG Configuration
|
||||
reranker_model: str = Field(default="mistral-nemo", description="Model for LLM re-ranking")
|
||||
reranker_enabled: bool = Field(default=True, description="Enable LLM re-ranking")
|
||||
hybrid_rag_vector_limit: int = Field(default=10, ge=1, le=50, description="Vector search limit")
|
||||
hybrid_rag_graph_limit: int = Field(default=10, ge=1, le=50, description="Graph search limit")
|
||||
|
||||
@@ -113,7 +113,7 @@ def get_ollama_client() -> OllamaClient:
|
||||
settings = get_settings()
|
||||
client = OllamaClient(
|
||||
base_url=settings.ollama_url,
|
||||
model=settings.ollama_model
|
||||
model=settings.ollama_embedding_model
|
||||
)
|
||||
logger.debug("Created Ollama client instance")
|
||||
return client
|
||||
|
||||
+4
-2
@@ -24,7 +24,7 @@ from src.clients.neo4j_client import Neo4jClient
|
||||
from src.clients.qdrant_client import QdrantClientWrapper
|
||||
from src.clients.ollama_client import OllamaClient
|
||||
from src.core.dependencies import (
|
||||
WikiJSDep, Neo4jDep, QdrantDep, OllamaDep, SearXNGDep,
|
||||
WikiJSDep, Neo4jDep, QdrantDep, OllamaDep, SearXNGDep, ContentExtractorDep,
|
||||
verify_api_key, get_settings, get_hybrid_rag_service, get_ingestion_service
|
||||
)
|
||||
from src.core.multi_tenancy import DEFAULT_USER
|
||||
@@ -180,6 +180,7 @@ async def smart_create_page(
|
||||
qdrant_client: QdrantDep,
|
||||
ollama_client: OllamaDep,
|
||||
searxng_client: SearXNGDep,
|
||||
content_extractor: ContentExtractorDep,
|
||||
settings: Settings = Depends(get_settings),
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
@@ -225,9 +226,10 @@ async def smart_create_page(
|
||||
graph_service=graph_service,
|
||||
searxng_client=searxng_client,
|
||||
ollama_client=ollama_client,
|
||||
content_extractor=content_extractor,
|
||||
settings=settings
|
||||
)
|
||||
wiki_page_writer = WikiPageWriter(ollama_client=ollama_client)
|
||||
wiki_page_writer = WikiPageWriter(ollama_client=ollama_client, settings=settings)
|
||||
|
||||
# Step 1-5: Research + Generate + Create page
|
||||
page, research_data = await wiki_service.smart_create_page(
|
||||
|
||||
@@ -47,7 +47,7 @@ class ConsolidationService:
|
||||
self.ollama = ollama
|
||||
self.wiki = wiki
|
||||
self.settings = settings
|
||||
self.wiki_page_writer = WikiPageWriter(ollama_client=ollama)
|
||||
self.wiki_page_writer = WikiPageWriter(ollama_client=ollama, settings=settings)
|
||||
self.ingestion_service = ingestion_service # Optional to avoid circular dependency
|
||||
|
||||
async def consolidate_knowledge(
|
||||
@@ -465,7 +465,7 @@ JSON:"""
|
||||
# Call Ollama for analysis
|
||||
response = await self.ollama.generate_text(
|
||||
prompt=prompt,
|
||||
model=self.settings.reranker_model, # Use mistral-nemo
|
||||
model=self.settings.ollama_model,
|
||||
stream=False
|
||||
)
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ HybridRAG service combining vector, graph, and web search.
|
||||
1. Parallel Retrieval - Vector + Graph + Web search
|
||||
2. RRF Fusion - Merge results with Reciprocal Rank Fusion
|
||||
3. Enrichment - Add related dossiers via graph
|
||||
4. LLM Re-ranking - Re-rank with mistral-nemo
|
||||
4. LLM Re-ranking - Re-rank with configured Ollama model
|
||||
5. Context Formatting - Format for LLM consumption
|
||||
6. Persistence - Store for Librarian processing
|
||||
"""
|
||||
@@ -65,7 +65,7 @@ class HybridRAGService:
|
||||
self.ollama = ollama_client
|
||||
self.content_extractor = content_extractor
|
||||
self.settings = settings
|
||||
self.reranker_model = settings.reranker_model
|
||||
self.reranker_model = settings.ollama_model
|
||||
|
||||
async def search(
|
||||
self,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""
|
||||
Intelligent Wiki Page Writer Service
|
||||
|
||||
Uses LLM (mistral-nemo) to create and reconstruct wiki pages with:
|
||||
Uses LLM to create and reconstruct wiki pages with:
|
||||
- Holistic content restructuring
|
||||
- Zero fact loss (unless superseded)
|
||||
- Conflict detection and flagging
|
||||
@@ -25,15 +25,16 @@ class WikiPageWriter:
|
||||
Intelligent wiki page writer using LLM for content generation and restructuring.
|
||||
"""
|
||||
|
||||
def __init__(self, ollama_client):
|
||||
def __init__(self, ollama_client, settings):
|
||||
"""
|
||||
Initialize wiki page writer.
|
||||
|
||||
Args:
|
||||
ollama_client: OllamaClient for LLM operations
|
||||
settings: Application settings
|
||||
"""
|
||||
self.ollama = ollama_client
|
||||
self.model = "mistral-nemo" # Default model for writing
|
||||
self.model = settings.ollama_model
|
||||
|
||||
async def create_page(
|
||||
self,
|
||||
|
||||
Reference in New Issue
Block a user