Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
318636d33d | ||
|
|
b976da0092 | ||
|
|
edfe11f0fb | ||
|
|
8e003bb9e8 | ||
|
|
dfd1f19bf9 | ||
|
|
1aca286703 | ||
|
|
262a58b0d2 | ||
|
|
02d728ac5b | ||
|
|
a1832e3245 | ||
|
|
5be31a5a00 |
@@ -23,6 +23,32 @@
|
||||
* **Update `CHANGELOG.md`** with every user-facing change.
|
||||
* Format: `## [Unreleased] - YYYY-MM-DD` followed by `### Added`, `### Changed`, or `### Fixed`.
|
||||
|
||||
### 🚀 Release Flow
|
||||
When changes are ready for deployment:
|
||||
|
||||
1. **Ask user if deploy cycle is desired **
|
||||
|
||||
2. **Update version** in `pyproject.toml`:
|
||||
- Bug fixes: bump patch version (1.8.3 → 1.8.4)
|
||||
- New features: bump minor version (1.8.4 → 1.9.0)
|
||||
|
||||
3. **Update CHANGELOG.md**:
|
||||
- Move items from `[Unreleased]` to new version section
|
||||
- Add release date: `## [1.8.4] - 2025-12-16`
|
||||
|
||||
4. **Commit and tag**:
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "fix: description of changes"
|
||||
git tag v1.8.4
|
||||
git push origin main --tags
|
||||
```
|
||||
|
||||
5. **CI/CD triggers automatically**:
|
||||
- Gitea CI builds Docker image on new tag
|
||||
- Watchtower pulls and deploys to production
|
||||
- Verify deployment: `curl http://192.168.86.149:8000/health`
|
||||
|
||||
---
|
||||
|
||||
## 2. FastAPI Architecture & Best Practices
|
||||
|
||||
@@ -5,6 +5,43 @@ All notable changes to Library Desk will be documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [1.3.3] - 2025-12-23
|
||||
|
||||
### Added
|
||||
|
||||
- Temperature parameter to `OllamaClient.generate_text()` for controlling output determinism
|
||||
- `TODO.md` tracking remaining stub endpoints to implement
|
||||
- Wired `/query/semantic` endpoint to VectorService
|
||||
- Wired `/query/graph` endpoint to GraphService
|
||||
|
||||
### Changed
|
||||
|
||||
- **Improved LLM prompts** based on llm-findings.md recommendations:
|
||||
- Keyword extraction: temperature 0.0, negative constraints
|
||||
- LLM re-ranking: temperature 0.0, explicit rules
|
||||
- Conflict detection: temperature 0.0, analysis steps (CoT)
|
||||
- Wiki page creation: temperature 0.3, anti-hallucination constraints
|
||||
- Page reconstruction: temperature 0.2, preservation constraints
|
||||
- Web results analysis: temperature 0.0, conservative approach
|
||||
- Test fixtures now use configurable host (TEST_HOST) instead of Docker hostnames
|
||||
|
||||
### Removed
|
||||
|
||||
- Dead code: unused `get_default_user()` function
|
||||
- Unused imports from routers (wiki.py, graph.py, hybrid_rag.py)
|
||||
- Stub endpoints shadowed by real implementations (/stats, /ingest/document, /ingest/batch)
|
||||
|
||||
## [1.3.2] - 2025-12-22
|
||||
|
||||
### Changed
|
||||
|
||||
- **Consolidated Ollama model configuration** - All LLM operations now use single `OLLAMA_MODEL` environment variable
|
||||
- Removed separate `reranker_model` setting
|
||||
- HybridRAG re-ranking, consolidation analysis, and wiki page writing all use the same model
|
||||
- Improves VRAM efficiency by keeping one model hot
|
||||
- Added `OLLAMA_EMBEDDING_MODEL` environment variable for embedding model (previously overloaded `OLLAMA_MODEL`)
|
||||
- Updated WikiPageWriter to accept settings instead of hardcoded model name
|
||||
|
||||
## [1.3.1] - 2025-12-16
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -49,7 +49,8 @@ QDRANT_PORT=6333
|
||||
WIKIJS_URL=http://wiki:3000
|
||||
SEARXNG_URL=http://searxng:8080
|
||||
OLLAMA_URL=http://ollama:11434
|
||||
OLLAMA_MODEL=nomic-embed-text
|
||||
OLLAMA_MODEL=mistral-nemo-large:latest
|
||||
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
|
||||
REDIS_HOST=redis-shared
|
||||
REDIS_PORT=6379
|
||||
REDIS_DB=2
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
# TODO
|
||||
|
||||
Outstanding work items for Library Desk.
|
||||
|
||||
## Stub Endpoints to Implement
|
||||
|
||||
The following endpoints in `src/main.py` return stub responses and need real implementations:
|
||||
|
||||
### Ingestion Status Endpoints
|
||||
|
||||
#### `POST /ingest/check-updates`
|
||||
Check which documents need updating based on content hashes. Used by Scheduler to determine what changed since last sync.
|
||||
|
||||
**Implementation needed:**
|
||||
1. Query existing documents by path
|
||||
2. Compare content hashes
|
||||
3. Return list of updates needed
|
||||
|
||||
#### `GET /ingest/status/{document_id}`
|
||||
Get processing status for a document.
|
||||
|
||||
**Implementation needed:**
|
||||
- Status tracking system (Redis or database)
|
||||
- Track ingestion progress per document
|
||||
|
||||
#### `GET /ingest/repo-status/{repository}`
|
||||
Get indexing status for an entire repository.
|
||||
|
||||
**Implementation needed:**
|
||||
- Repository-level statistics
|
||||
- Track which documents from a repo are indexed
|
||||
|
||||
### Deduplication
|
||||
|
||||
#### `POST /deduplicate/check`
|
||||
Check for duplicate or highly similar documents using vector similarity and graph analysis.
|
||||
|
||||
**Implementation needed:**
|
||||
1. Get document embedding from Qdrant
|
||||
2. Find similar vectors above threshold
|
||||
3. Check graph relationships
|
||||
4. Return candidates with similarity scores
|
||||
|
||||
## System Statistics
|
||||
|
||||
#### `GET /stats`
|
||||
Get system statistics (wiki pages, neo4j nodes, qdrant vectors).
|
||||
|
||||
**Implementation needed:**
|
||||
- Query Neo4j for node count
|
||||
- Query Qdrant for vector count
|
||||
- Query Wiki.js for page count
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "library-desk"
|
||||
version = "1.3.1"
|
||||
version = "1.3.3"
|
||||
description = "Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and knowledge consolidation"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
|
||||
@@ -249,7 +249,8 @@ class OllamaClient:
|
||||
self,
|
||||
prompt: str,
|
||||
model: Optional[str] = None,
|
||||
stream: bool = False
|
||||
stream: bool = False,
|
||||
temperature: Optional[float] = None
|
||||
) -> Optional[str]:
|
||||
"""
|
||||
Generate text completion (for non-embedding use cases).
|
||||
@@ -258,12 +259,15 @@ class OllamaClient:
|
||||
prompt: Input prompt
|
||||
model: Model name (defaults to self.model)
|
||||
stream: Enable streaming response
|
||||
temperature: Sampling temperature (0.0 = deterministic, higher = more creative)
|
||||
None uses model default (~0.7 for mistral-nemo)
|
||||
|
||||
Returns:
|
||||
Generated text or None on failure
|
||||
|
||||
Note: This is primarily for debugging/testing. Use specialized
|
||||
LLM services for production text generation.
|
||||
Note: Use temperature=0.0 for deterministic outputs like JSON parsing,
|
||||
ranking, and factual extraction. Use higher values (0.3-0.7) for
|
||||
creative content generation.
|
||||
"""
|
||||
try:
|
||||
payload = {
|
||||
@@ -272,6 +276,10 @@ class OllamaClient:
|
||||
"stream": stream
|
||||
}
|
||||
|
||||
# Add temperature to options if specified
|
||||
if temperature is not None:
|
||||
payload["options"] = {"temperature": temperature}
|
||||
|
||||
response = await self.client.post(
|
||||
self.generate_url,
|
||||
json=payload
|
||||
|
||||
+3
-3
@@ -62,12 +62,12 @@ class Settings(BaseSettings):
|
||||
# SearXNG Configuration
|
||||
searxng_url: str = Field(default="http://searxng:8080", description="SearXNG URL")
|
||||
|
||||
# Ollama Configuration (for embeddings)
|
||||
# Ollama Configuration
|
||||
ollama_url: str = Field(default="http://ollama:11434", description="Ollama URL")
|
||||
ollama_model: str = Field(default="nomic-embed-text", description="Ollama embedding model")
|
||||
ollama_model: str = Field(default="mistral-nemo-large:latest", description="Ollama LLM model")
|
||||
ollama_embedding_model: str = Field(default="nomic-embed-text", description="Ollama embedding model")
|
||||
|
||||
# HybridRAG Configuration
|
||||
reranker_model: str = Field(default="mistral-nemo", description="Model for LLM re-ranking")
|
||||
reranker_enabled: bool = Field(default=True, description="Enable LLM re-ranking")
|
||||
hybrid_rag_vector_limit: int = Field(default=10, ge=1, le=50, description="Vector search limit")
|
||||
hybrid_rag_graph_limit: int = Field(default=10, ge=1, le=50, description="Graph search limit")
|
||||
|
||||
@@ -113,7 +113,7 @@ def get_ollama_client() -> OllamaClient:
|
||||
settings = get_settings()
|
||||
client = OllamaClient(
|
||||
base_url=settings.ollama_url,
|
||||
model=settings.ollama_model
|
||||
model=settings.ollama_embedding_model
|
||||
)
|
||||
logger.debug("Created Ollama client instance")
|
||||
return client
|
||||
@@ -395,18 +395,6 @@ def get_rag_search_service() -> "RAGSearchService":
|
||||
)
|
||||
|
||||
|
||||
# Utility: Get default user from settings or multi_tenancy
|
||||
def get_default_user() -> str:
|
||||
"""
|
||||
Get default user for operations.
|
||||
|
||||
Returns:
|
||||
Default user identifier
|
||||
"""
|
||||
from src.core.multi_tenancy import DEFAULT_USER
|
||||
return DEFAULT_USER
|
||||
|
||||
|
||||
# Authentication
|
||||
from fastapi import Security, HTTPException
|
||||
from fastapi.security import HTTPBearer
|
||||
|
||||
+69
-100
@@ -8,7 +8,7 @@ Following best practices:
|
||||
- OpenAPI documentation
|
||||
"""
|
||||
|
||||
from fastapi import FastAPI, HTTPException, Depends
|
||||
from fastapi import FastAPI, HTTPException, Depends, Query
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from pydantic import BaseModel
|
||||
@@ -17,7 +17,10 @@ import logging
|
||||
from pathlib import Path
|
||||
|
||||
from src.config import Settings, get_settings, __version__
|
||||
from src.core.dependencies import verify_api_key
|
||||
from src.core.dependencies import (
|
||||
verify_api_key, QdrantDep, WikiJSDep, OllamaDep, Neo4jDep
|
||||
)
|
||||
from src.core.multi_tenancy import DEFAULT_USER
|
||||
|
||||
# Configure logging
|
||||
logging.basicConfig(
|
||||
@@ -78,13 +81,6 @@ class HealthResponse(BaseModel):
|
||||
services: Dict[str, Any]
|
||||
|
||||
|
||||
class StatsResponse(BaseModel):
|
||||
"""Statistics response model."""
|
||||
wiki_pages: int
|
||||
neo4j_nodes: int
|
||||
qdrant_vectors: int
|
||||
|
||||
|
||||
# Routes
|
||||
@app.get("/", tags=["Root"])
|
||||
async def root() -> Dict[str, str]:
|
||||
@@ -141,77 +137,6 @@ async def health(settings: Settings = Depends(get_settings)) -> HealthResponse:
|
||||
)
|
||||
|
||||
|
||||
@app.get("/stats", response_model=StatsResponse, tags=["System"])
|
||||
async def stats(
|
||||
api_key: str = Depends(verify_api_key)
|
||||
) -> StatsResponse:
|
||||
"""
|
||||
Get system statistics.
|
||||
Protected endpoint - requires API key.
|
||||
|
||||
TODO: Implement actual stats gathering from:
|
||||
- Neo4j (node count)
|
||||
- Qdrant (vector count)
|
||||
- Wiki.js (page count)
|
||||
"""
|
||||
return StatsResponse(
|
||||
wiki_pages=0,
|
||||
neo4j_nodes=0,
|
||||
qdrant_vectors=0
|
||||
)
|
||||
|
||||
|
||||
# Ingestion endpoints (for Scheduler integration)
|
||||
@app.post("/ingest/document", tags=["Ingestion"])
|
||||
async def ingest_document(
|
||||
document: Dict[str, Any],
|
||||
api_key: str = Depends(verify_api_key)
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Ingest a single document for indexing.
|
||||
Used by The Scheduler to add mirrored documentation to the knowledge base.
|
||||
|
||||
Expected fields:
|
||||
- source: str (e.g., "github", "gitea")
|
||||
- repository: str (e.g., "anthropic-cookbook")
|
||||
- path: str (file path)
|
||||
- content: str (document content)
|
||||
- metadata: dict (commit, author, tags, etc.)
|
||||
|
||||
TODO: Implement document ingestion pipeline:
|
||||
1. Chunk content
|
||||
2. Generate embeddings (Ollama)
|
||||
3. Extract entities (NLP)
|
||||
4. Index in Qdrant
|
||||
5. Create graph nodes/relationships in Neo4j
|
||||
"""
|
||||
return {
|
||||
"message": "Document ingestion not yet implemented",
|
||||
"document_id": f"doc_{document.get('path', 'unknown')}",
|
||||
"status": "stub"
|
||||
}
|
||||
|
||||
|
||||
@app.post("/ingest/batch", tags=["Ingestion"])
|
||||
async def batch_ingest(
|
||||
batch: Dict[str, Any],
|
||||
api_key: str = Depends(verify_api_key)
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Ingest multiple documents in a batch.
|
||||
More efficient than individual ingestion for large syncs.
|
||||
|
||||
TODO: Implement batch processing with task queue
|
||||
"""
|
||||
document_count = len(batch.get("documents", []))
|
||||
return {
|
||||
"message": "Batch ingestion not yet implemented",
|
||||
"batch_id": "batch_stub",
|
||||
"total_documents": document_count,
|
||||
"status": "stub"
|
||||
}
|
||||
|
||||
|
||||
@app.post("/ingest/check-updates", tags=["Ingestion"])
|
||||
async def check_updates(
|
||||
documents: Dict[str, Any],
|
||||
@@ -269,41 +194,85 @@ async def get_repo_status(
|
||||
}
|
||||
|
||||
|
||||
# Query endpoints (stubs for future implementation)
|
||||
# NOTE: /query/hybrid is now implemented in routers/hybrid_rag.py
|
||||
# Query endpoints
|
||||
# NOTE: /query/hybrid is implemented in routers/hybrid_rag.py
|
||||
|
||||
@app.post("/query/semantic", tags=["Query"])
|
||||
async def semantic_query(
|
||||
query: Dict[str, Any],
|
||||
query: str = Query(..., min_length=1, description="Search query text"),
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
limit: int = Query(default=10, ge=1, le=100, description="Maximum results"),
|
||||
score_threshold: float = Query(default=0.5, ge=0.0, le=1.0, description="Minimum similarity score"),
|
||||
qdrant_client: QdrantDep = None,
|
||||
wiki_client: WikiJSDep = None,
|
||||
ollama_client: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
) -> Dict[str, Any]:
|
||||
):
|
||||
"""
|
||||
Semantic search via Qdrant.
|
||||
Pure vector similarity search.
|
||||
Semantic search via Qdrant vector similarity.
|
||||
|
||||
TODO: Implement semantic search
|
||||
Searches document chunks using embedding similarity. Returns matching
|
||||
chunks with relevance scores, page titles, and paths.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
POST /query/semantic?query=docker%20configuration&user=jpmschweitzer&limit=10
|
||||
```
|
||||
|
||||
**Returns:** List of matching chunks with similarity scores (0-1)
|
||||
"""
|
||||
return {
|
||||
"message": "Semantic search not yet implemented",
|
||||
"query": query
|
||||
}
|
||||
from src.services.vector_service import VectorService
|
||||
|
||||
vector_service = VectorService(qdrant_client, wiki_client, ollama_client)
|
||||
try:
|
||||
return await vector_service.search(
|
||||
query=query,
|
||||
user=user,
|
||||
limit=limit,
|
||||
score_threshold=score_threshold
|
||||
)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.error(f"Semantic search failed: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail="Search failed")
|
||||
|
||||
|
||||
@app.post("/query/graph", tags=["Query"])
|
||||
async def graph_query(
|
||||
query: Dict[str, Any],
|
||||
query: str = Query(..., description="Cypher query to execute"),
|
||||
user: str = Query(default=DEFAULT_USER, description="User for scoping (auto-filters results)"),
|
||||
neo4j_client: Neo4jDep = None,
|
||||
wiki_client: WikiJSDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
) -> Dict[str, Any]:
|
||||
):
|
||||
"""
|
||||
Graph traversal via Neo4j.
|
||||
Execute Cypher queries.
|
||||
Execute a Cypher query against the Neo4j knowledge graph.
|
||||
|
||||
TODO: Implement graph queries
|
||||
Queries are automatically scoped to the user's data for security.
|
||||
Use this for custom graph traversals beyond what /graph/nodes provides.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
POST /query/graph?query=MATCH%20(d:Document)-[:MENTIONS]->(p:Person)%20RETURN%20d,p&user=jpmschweitzer
|
||||
```
|
||||
|
||||
**Security:** All queries are user-scoped to prevent cross-user data access.
|
||||
"""
|
||||
return {
|
||||
"message": "Graph query not yet implemented",
|
||||
"query": query
|
||||
}
|
||||
from src.services.graph_service import GraphService
|
||||
|
||||
graph_service = GraphService(neo4j_client, wiki_client)
|
||||
try:
|
||||
return await graph_service.execute_query(
|
||||
query=query,
|
||||
parameters={},
|
||||
user=user
|
||||
)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.error(f"Graph query failed: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail="Query execution failed")
|
||||
|
||||
|
||||
# Deduplication endpoints
|
||||
|
||||
@@ -15,8 +15,6 @@ from src.models.graph import (
|
||||
MindMapResponse
|
||||
)
|
||||
from src.services.graph_service import GraphService
|
||||
from src.clients.neo4j_client import Neo4jClient
|
||||
from src.clients.wikijs_client import WikiJSClient
|
||||
from src.core.dependencies import Neo4jDep, WikiJSDep, verify_api_key
|
||||
from src.core.multi_tenancy import DEFAULT_USER
|
||||
|
||||
|
||||
@@ -10,10 +10,6 @@ import logging
|
||||
|
||||
from src.models.hybrid_rag import HybridRAGRequest, HybridRAGResponse
|
||||
from src.services.hybrid_rag_service import HybridRAGService
|
||||
from src.services.vector_service import VectorService
|
||||
from src.services.graph_service import GraphService
|
||||
from src.clients.searxng_client import SearXNGClient
|
||||
from src.clients.ollama_client import OllamaClient
|
||||
from src.core.dependencies import (
|
||||
Neo4jDep, WikiJSDep, QdrantDep, OllamaDep,
|
||||
SearXNGDep, ContentExtractorDep, verify_api_key, get_settings
|
||||
|
||||
+2
-3
@@ -5,8 +5,7 @@ Endpoints for wiki page and dossier management.
|
||||
All operations are scoped to user namespaces for multi-tenancy.
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Depends, Query, Security, BackgroundTasks
|
||||
from fastapi.security import HTTPAuthorizationCredentials
|
||||
from fastapi import APIRouter, HTTPException, Depends, Query, BackgroundTasks
|
||||
from typing import Optional
|
||||
import logging
|
||||
|
||||
@@ -229,7 +228,7 @@ async def smart_create_page(
|
||||
content_extractor=content_extractor,
|
||||
settings=settings
|
||||
)
|
||||
wiki_page_writer = WikiPageWriter(ollama_client=ollama_client)
|
||||
wiki_page_writer = WikiPageWriter(ollama_client=ollama_client, settings=settings)
|
||||
|
||||
# Step 1-5: Research + Generate + Create page
|
||||
page, research_data = await wiki_service.smart_create_page(
|
||||
|
||||
@@ -47,7 +47,7 @@ class ConsolidationService:
|
||||
self.ollama = ollama
|
||||
self.wiki = wiki
|
||||
self.settings = settings
|
||||
self.wiki_page_writer = WikiPageWriter(ollama_client=ollama)
|
||||
self.wiki_page_writer = WikiPageWriter(ollama_client=ollama, settings=settings)
|
||||
self.ingestion_service = ingestion_service # Optional to avoid circular dependency
|
||||
|
||||
async def consolidate_knowledge(
|
||||
@@ -410,13 +410,19 @@ This is a PERSONAL knowledge base using Schema.org-aligned taxonomy that capture
|
||||
- Projects: Work projects, personal projects (Schema.org: Project)
|
||||
- Reference: General knowledge, how-tos (Custom extension)
|
||||
|
||||
Identify information worth documenting:
|
||||
1. New topics/people/things that deserve their own wiki page
|
||||
2. Facts that could enhance existing pages
|
||||
3. Entities (people, places, things, concepts) for the knowledge graph
|
||||
ANALYSIS STEPS:
|
||||
1. Read each web result carefully for substantive, factual content
|
||||
2. Identify genuinely novel information not likely already known
|
||||
3. Match topics to appropriate taxonomy categories
|
||||
4. Generate valid paths following the exact format below
|
||||
|
||||
Be INCLUSIVE - if someone searched for it, it's likely worth documenting.
|
||||
Personal information is just as valuable as technical information.
|
||||
RULES:
|
||||
- Do NOT suggest pages for topics with insufficient information in results
|
||||
- Do NOT invent entities not explicitly mentioned in results
|
||||
- Do NOT suggest paths that don't match the taxonomy exactly
|
||||
- Do NOT suggest generic or vague page topics
|
||||
- Be CONSERVATIVE - fewer high-quality suggestions is better than many low-quality ones
|
||||
- ONLY suggest documentation for substantive, specific information
|
||||
|
||||
**CRITICAL: Use ONLY these Schema.org-aligned path prefixes (case-sensitive):**
|
||||
|
||||
@@ -462,11 +468,12 @@ Return ONLY valid JSON:
|
||||
JSON:"""
|
||||
|
||||
try:
|
||||
# Call Ollama for analysis
|
||||
# Call Ollama for analysis (temperature=0.0 for consistent classification)
|
||||
response = await self.ollama.generate_text(
|
||||
prompt=prompt,
|
||||
model=self.settings.reranker_model, # Use mistral-nemo
|
||||
stream=False
|
||||
model=self.settings.ollama_model,
|
||||
stream=False,
|
||||
temperature=0.0
|
||||
)
|
||||
|
||||
if not response:
|
||||
|
||||
@@ -6,7 +6,7 @@ HybridRAG service combining vector, graph, and web search.
|
||||
1. Parallel Retrieval - Vector + Graph + Web search
|
||||
2. RRF Fusion - Merge results with Reciprocal Rank Fusion
|
||||
3. Enrichment - Add related dossiers via graph
|
||||
4. LLM Re-ranking - Re-rank with mistral-nemo
|
||||
4. LLM Re-ranking - Re-rank with configured Ollama model
|
||||
5. Context Formatting - Format for LLM consumption
|
||||
6. Persistence - Store for Librarian processing
|
||||
"""
|
||||
@@ -65,7 +65,7 @@ class HybridRAGService:
|
||||
self.ollama = ollama_client
|
||||
self.content_extractor = content_extractor
|
||||
self.settings = settings
|
||||
self.reranker_model = settings.reranker_model
|
||||
self.reranker_model = settings.ollama_model
|
||||
|
||||
async def search(
|
||||
self,
|
||||
@@ -195,25 +195,21 @@ class HybridRAGService:
|
||||
Returns:
|
||||
Dictionary with keywords, entities, synonyms, expansions
|
||||
"""
|
||||
prompt = f"""Extract search terms from this query. For each important word, provide synonyms and expansions.
|
||||
prompt = f"""Extract search terms from this query.
|
||||
|
||||
Query: "{query}"
|
||||
|
||||
Return ONLY valid JSON:
|
||||
{{
|
||||
"core_keywords": ["key", "words", "from", "query"],
|
||||
"synonyms": {{
|
||||
"word": ["alternative", "terms"]
|
||||
}}
|
||||
}}
|
||||
RULES:
|
||||
- Extract ONLY keywords explicitly present or directly implied in the query
|
||||
- Do NOT invent terms, concepts, or synonyms not clearly related
|
||||
- Do NOT add general knowledge or associations
|
||||
- Provide synonyms ONLY for technical terms with well-known alternatives
|
||||
- Return valid JSON only, no commentary
|
||||
|
||||
Example for "Docker container hosting":
|
||||
Return format:
|
||||
{{
|
||||
"core_keywords": ["docker", "container", "hosting"],
|
||||
"synonyms": {{
|
||||
"docker": ["containerization", "container runtime"],
|
||||
"hosting": ["server", "infrastructure"]
|
||||
}}
|
||||
"core_keywords": ["words", "from", "query"],
|
||||
"synonyms": {{"term": ["direct", "alternatives"]}}
|
||||
}}
|
||||
|
||||
JSON:"""
|
||||
@@ -221,7 +217,8 @@ JSON:"""
|
||||
try:
|
||||
response = await self.ollama.generate_text(
|
||||
prompt=prompt,
|
||||
model=self.reranker_model
|
||||
model=self.reranker_model,
|
||||
temperature=0.0 # Deterministic for consistent extraction
|
||||
)
|
||||
|
||||
# Parse JSON response (handle potential extra text)
|
||||
@@ -623,21 +620,27 @@ JSON:"""
|
||||
for i, r in enumerate(results)
|
||||
])
|
||||
|
||||
prompt = f"""Given this search query and documents, rank them by relevance.
|
||||
prompt = f"""Rank these documents by relevance to the query.
|
||||
|
||||
Query: {query}
|
||||
|
||||
Documents:
|
||||
{docs_text}
|
||||
|
||||
Return only the numbers in order of relevance (most relevant first).
|
||||
Example: 3,1,5,2,4
|
||||
RULES:
|
||||
- Rank ONLY by how well content answers the query
|
||||
- Do NOT consider document length, formatting, or style
|
||||
- Do NOT add explanation or commentary
|
||||
- Return ONLY comma-separated numbers, most relevant first
|
||||
|
||||
Example output: 3,1,5,2,4
|
||||
|
||||
Ranking:"""
|
||||
|
||||
response = await self.ollama.generate_text(
|
||||
prompt=prompt,
|
||||
model=self.reranker_model
|
||||
model=self.reranker_model,
|
||||
temperature=0.0 # Deterministic for consistent rankings
|
||||
)
|
||||
|
||||
# Parse response: "3,1,5,2,4" → [2, 0, 4, 1, 3] (0-indexed)
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""
|
||||
Intelligent Wiki Page Writer Service
|
||||
|
||||
Uses LLM (mistral-nemo) to create and reconstruct wiki pages with:
|
||||
Uses LLM to create and reconstruct wiki pages with:
|
||||
- Holistic content restructuring
|
||||
- Zero fact loss (unless superseded)
|
||||
- Conflict detection and flagging
|
||||
@@ -25,15 +25,16 @@ class WikiPageWriter:
|
||||
Intelligent wiki page writer using LLM for content generation and restructuring.
|
||||
"""
|
||||
|
||||
def __init__(self, ollama_client):
|
||||
def __init__(self, ollama_client, settings):
|
||||
"""
|
||||
Initialize wiki page writer.
|
||||
|
||||
Args:
|
||||
ollama_client: OllamaClient for LLM operations
|
||||
settings: Application settings
|
||||
"""
|
||||
self.ollama = ollama_client
|
||||
self.model = "mistral-nemo" # Default model for writing
|
||||
self.model = settings.ollama_model
|
||||
|
||||
async def create_page(
|
||||
self,
|
||||
@@ -130,8 +131,8 @@ class WikiPageWriter:
|
||||
conflicts=conflicts
|
||||
)
|
||||
|
||||
# Reconstruct with LLM
|
||||
reconstructed = await self._call_llm(prompt)
|
||||
# Reconstruct with LLM (lower temperature for precise merging)
|
||||
reconstructed = await self._call_llm(prompt, temperature=0.2)
|
||||
|
||||
# Ensure standard sections are present
|
||||
reconstructed = self._ensure_standard_sections(
|
||||
@@ -154,7 +155,7 @@ class WikiPageWriter:
|
||||
Returns:
|
||||
List of conflicts with: {fact_a, fact_b, confidence, context}
|
||||
"""
|
||||
prompt = f"""Analyze these two pieces of content for factual conflicts.
|
||||
prompt = f"""Analyze these contents for direct factual conflicts.
|
||||
|
||||
EXISTING CONTENT:
|
||||
{existing_content[:2000]}
|
||||
@@ -162,25 +163,26 @@ EXISTING CONTENT:
|
||||
NEW INFORMATION:
|
||||
{new_information[:2000]}
|
||||
|
||||
Identify any facts that contradict each other. For each conflict, provide:
|
||||
1. The fact from existing content
|
||||
2. The contradicting fact from new information
|
||||
3. Confidence level (low/medium/high)
|
||||
4. Context/explanation
|
||||
ANALYSIS STEPS:
|
||||
1. Identify specific factual claims in existing content (dates, numbers, names, states)
|
||||
2. Identify specific factual claims in new content
|
||||
3. Compare ONLY for direct contradictions (X says A, Y says not-A)
|
||||
|
||||
Return ONLY valid JSON:
|
||||
RULES:
|
||||
- Do NOT flag differences in wording or phrasing as conflicts
|
||||
- Do NOT flag new/additional information as conflicts
|
||||
- Do NOT flag opinion differences as conflicts
|
||||
- ONLY flag direct factual contradictions
|
||||
- Return valid JSON only, no commentary
|
||||
|
||||
Return format:
|
||||
{{
|
||||
"conflicts": [
|
||||
{{
|
||||
"existing_fact": "fact from old content",
|
||||
"new_fact": "contradicting fact",
|
||||
"confidence": "medium",
|
||||
"context": "explanation of why these conflict"
|
||||
}}
|
||||
{{"existing_fact": "...", "new_fact": "...", "confidence": "low/medium/high", "context": "..."}}
|
||||
]
|
||||
}}
|
||||
|
||||
If no conflicts, return: {{"conflicts": []}}
|
||||
If no conflicts: {{"conflicts": []}}
|
||||
|
||||
JSON:"""
|
||||
|
||||
@@ -188,7 +190,8 @@ JSON:"""
|
||||
response = await self.ollama.generate_text(
|
||||
prompt=prompt,
|
||||
model=self.model,
|
||||
stream=False
|
||||
stream=False,
|
||||
temperature=0.0 # Deterministic for consistent conflict detection
|
||||
)
|
||||
|
||||
# Extract JSON
|
||||
@@ -347,6 +350,13 @@ FORMATTING RULES:
|
||||
- Keep sections focused and scannable
|
||||
- Adapt structure to content - not all sections apply to all topics
|
||||
|
||||
CRITICAL CONSTRAINTS:
|
||||
- Do NOT invent facts not present in the source information above
|
||||
- Do NOT add speculative information or assumptions
|
||||
- Do NOT fill sections with placeholder text or generic statements
|
||||
- If information for a section is not available, OMIT the section entirely
|
||||
- Base ALL content strictly on provided source information
|
||||
|
||||
Generate ONLY the markdown content (do not include Sources, Knowledge Graph, or Mind Map sections - those are added automatically).
|
||||
|
||||
MARKDOWN:"""
|
||||
@@ -402,6 +412,13 @@ FORMATTING RULES:
|
||||
- Bold important terms
|
||||
- Add subsections (###) where it improves clarity
|
||||
|
||||
CRITICAL CONSTRAINTS:
|
||||
- Do NOT rephrase facts in ways that change their meaning
|
||||
- Do NOT remove ANY information unless explicitly superseded by newer facts
|
||||
- Do NOT add information not present in existing content or new information
|
||||
- Preserve exact quotes, dates, numbers, and names verbatim
|
||||
- Do NOT fill gaps with assumptions or general knowledge
|
||||
|
||||
OUTPUT INSTRUCTIONS:
|
||||
- Return complete page content (do not include Sources, Knowledge Graph, Mind Map - those are added automatically)
|
||||
- Include updated "Changes & Updates" section noting what was changed today
|
||||
@@ -409,13 +426,21 @@ OUTPUT INSTRUCTIONS:
|
||||
|
||||
RECONSTRUCTED MARKDOWN:"""
|
||||
|
||||
async def _call_llm(self, prompt: str) -> str:
|
||||
"""Call LLM with prompt and return response."""
|
||||
async def _call_llm(self, prompt: str, temperature: float = 0.3) -> str:
|
||||
"""
|
||||
Call LLM with prompt and return response.
|
||||
|
||||
Args:
|
||||
prompt: The prompt text
|
||||
temperature: Sampling temperature (0.0=deterministic, higher=creative)
|
||||
Default 0.3 for controlled but natural content generation
|
||||
"""
|
||||
try:
|
||||
response = await self.ollama.generate_text(
|
||||
prompt=prompt,
|
||||
model=self.model,
|
||||
stream=False
|
||||
stream=False,
|
||||
temperature=temperature
|
||||
)
|
||||
|
||||
if not response:
|
||||
|
||||
+20
-9
@@ -1,5 +1,6 @@
|
||||
"""Pytest configuration and shared fixtures for Library Desk tests."""
|
||||
|
||||
import os
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from typing import AsyncGenerator
|
||||
@@ -7,6 +8,9 @@ from typing import AsyncGenerator
|
||||
# Test configuration
|
||||
pytest_plugins = ("pytest_asyncio",)
|
||||
|
||||
# Use real host for tests (services available at this IP)
|
||||
TEST_HOST = os.environ.get("TEST_HOST", "192.168.86.149")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def test_user() -> str:
|
||||
@@ -17,49 +21,56 @@ def test_user() -> str:
|
||||
@pytest.fixture
|
||||
def neo4j_test_uri() -> str:
|
||||
"""Test Neo4j URI."""
|
||||
return "bolt://neo4j:7687"
|
||||
return f"bolt://{TEST_HOST}:7687"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def neo4j_test_auth() -> tuple:
|
||||
"""Test Neo4j authentication."""
|
||||
return ("neo4j", "test_password")
|
||||
from src.config import get_settings
|
||||
settings = get_settings()
|
||||
return ("neo4j", settings.neo4j_password)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def qdrant_test_url() -> str:
|
||||
"""Test Qdrant URL."""
|
||||
return "http://qdrant:6333"
|
||||
return f"http://{TEST_HOST}:6333"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def wikijs_test_config() -> dict:
|
||||
"""Test Wiki.js configuration."""
|
||||
from src.config import get_settings
|
||||
settings = get_settings()
|
||||
return {
|
||||
"base_url": "http://wiki:3000",
|
||||
"api_key": "test_api_key"
|
||||
"base_url": f"http://{TEST_HOST}:3000",
|
||||
"username": settings.wikijs_username,
|
||||
"password": settings.wikijs_password
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def searxng_test_url() -> str:
|
||||
"""Test SearXNG URL."""
|
||||
return "http://searxng:8080"
|
||||
return f"http://{TEST_HOST}:8080"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def ollama_test_config() -> dict:
|
||||
"""Test Ollama configuration."""
|
||||
from src.config import get_settings
|
||||
settings = get_settings()
|
||||
return {
|
||||
"base_url": "http://ollama:11434",
|
||||
"model": "nomic-embed-text"
|
||||
"base_url": f"http://{TEST_HOST}:11434",
|
||||
"model": settings.ollama_embedding_model
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def redis_test_url() -> str:
|
||||
"""Test Redis URL."""
|
||||
return "redis://redis-shared:6379/4"
|
||||
return f"redis://{TEST_HOST}:6379/4"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
|
||||
@@ -38,10 +38,10 @@ def settings():
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def neo4j_client(settings) -> AsyncGenerator[Neo4jClient, None]:
|
||||
async def neo4j_client(settings, neo4j_test_uri) -> AsyncGenerator[Neo4jClient, None]:
|
||||
"""Get connected Neo4j client."""
|
||||
client = Neo4jClient(
|
||||
uri=settings.neo4j_uri,
|
||||
uri=neo4j_test_uri,
|
||||
user=settings.neo4j_user,
|
||||
password=settings.neo4j_password
|
||||
)
|
||||
@@ -51,12 +51,12 @@ async def neo4j_client(settings) -> AsyncGenerator[Neo4jClient, None]:
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def wiki_client(settings) -> AsyncGenerator[WikiJSClient, None]:
|
||||
async def wiki_client(wikijs_test_config) -> AsyncGenerator[WikiJSClient, None]:
|
||||
"""Get Wiki.js client."""
|
||||
client = WikiJSClient(
|
||||
base_url=settings.wikijs_url,
|
||||
username=settings.wikijs_username,
|
||||
password=settings.wikijs_password
|
||||
base_url=wikijs_test_config["base_url"],
|
||||
username=wikijs_test_config["username"],
|
||||
password=wikijs_test_config["password"]
|
||||
)
|
||||
yield client
|
||||
|
||||
|
||||
+15
-12
@@ -43,10 +43,10 @@ def settings():
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def neo4j_client(settings) -> AsyncGenerator[Neo4jClient, None]:
|
||||
async def neo4j_client(settings, neo4j_test_uri) -> AsyncGenerator[Neo4jClient, None]:
|
||||
"""Get connected Neo4j client."""
|
||||
client = Neo4jClient(
|
||||
uri=settings.neo4j_uri,
|
||||
uri=neo4j_test_uri,
|
||||
user=settings.neo4j_user,
|
||||
password=settings.neo4j_password
|
||||
)
|
||||
@@ -56,32 +56,35 @@ async def neo4j_client(settings) -> AsyncGenerator[Neo4jClient, None]:
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def qdrant_client(settings) -> QdrantClientWrapper:
|
||||
def qdrant_client(qdrant_test_url) -> QdrantClientWrapper:
|
||||
"""Get Qdrant client."""
|
||||
return QdrantClientWrapper(url=settings.qdrant_url)
|
||||
return QdrantClientWrapper(url=qdrant_test_url)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def wiki_client(settings) -> AsyncGenerator[WikiJSClient, None]:
|
||||
async def wiki_client(wikijs_test_config) -> AsyncGenerator[WikiJSClient, None]:
|
||||
"""Get Wiki.js client."""
|
||||
client = WikiJSClient(
|
||||
base_url=settings.wikijs_url,
|
||||
username=settings.wikijs_username,
|
||||
password=settings.wikijs_password
|
||||
base_url=wikijs_test_config["base_url"],
|
||||
username=wikijs_test_config["username"],
|
||||
password=wikijs_test_config["password"]
|
||||
)
|
||||
yield client
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def searxng_client(settings) -> SearXNGClient:
|
||||
def searxng_client(searxng_test_url) -> SearXNGClient:
|
||||
"""Get SearXNG client."""
|
||||
return SearXNGClient(base_url=settings.searxng_url)
|
||||
return SearXNGClient(base_url=searxng_test_url)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def ollama_client(settings) -> OllamaClient:
|
||||
def ollama_client(ollama_test_config) -> OllamaClient:
|
||||
"""Get Ollama client."""
|
||||
return OllamaClient(base_url=settings.ollama_url)
|
||||
return OllamaClient(
|
||||
base_url=ollama_test_config["base_url"],
|
||||
model=ollama_test_config["model"]
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
|
||||
+16
-15
@@ -30,10 +30,10 @@ def settings():
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def neo4j_client(settings) -> AsyncGenerator[Neo4jClient, None]:
|
||||
async def neo4j_client(settings, neo4j_test_uri) -> AsyncGenerator[Neo4jClient, None]:
|
||||
"""Get connected Neo4j client."""
|
||||
client = Neo4jClient(
|
||||
uri=settings.neo4j_uri,
|
||||
uri=neo4j_test_uri,
|
||||
user=settings.neo4j_user,
|
||||
password=settings.neo4j_password
|
||||
)
|
||||
@@ -43,45 +43,46 @@ async def neo4j_client(settings) -> AsyncGenerator[Neo4jClient, None]:
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def qdrant_client(settings) -> QdrantClientWrapper:
|
||||
def qdrant_client(settings, qdrant_test_url) -> QdrantClientWrapper:
|
||||
"""Get Qdrant client."""
|
||||
return QdrantClientWrapper(url=settings.qdrant_url)
|
||||
return QdrantClientWrapper(url=qdrant_test_url)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def wikijs_client(settings) -> AsyncGenerator[WikiJSClient, None]:
|
||||
async def wikijs_client(wikijs_test_config) -> AsyncGenerator[WikiJSClient, None]:
|
||||
"""Get Wiki.js client."""
|
||||
client = WikiJSClient(
|
||||
base_url=settings.wikijs_url,
|
||||
api_key=settings.wikijs_api_key
|
||||
base_url=wikijs_test_config["base_url"],
|
||||
username=wikijs_test_config["username"],
|
||||
password=wikijs_test_config["password"]
|
||||
)
|
||||
yield client
|
||||
await client.close()
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def searxng_client(settings) -> AsyncGenerator[SearXNGClient, None]:
|
||||
async def searxng_client(searxng_test_url) -> AsyncGenerator[SearXNGClient, None]:
|
||||
"""Get SearXNG client."""
|
||||
client = SearXNGClient(base_url=settings.searxng_url)
|
||||
client = SearXNGClient(base_url=searxng_test_url)
|
||||
yield client
|
||||
await client.close()
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def ollama_client(settings) -> AsyncGenerator[OllamaClient, None]:
|
||||
"""Get Ollama client."""
|
||||
async def ollama_client(ollama_test_config) -> AsyncGenerator[OllamaClient, None]:
|
||||
"""Get Ollama client for embeddings."""
|
||||
client = OllamaClient(
|
||||
base_url=settings.ollama_url,
|
||||
model=settings.ollama_model
|
||||
base_url=ollama_test_config["base_url"],
|
||||
model=ollama_test_config["model"]
|
||||
)
|
||||
yield client
|
||||
await client.close()
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def job_manager(settings) -> AsyncGenerator[JobManager, None]:
|
||||
async def job_manager(redis_test_url) -> AsyncGenerator[JobManager, None]:
|
||||
"""Get job manager."""
|
||||
manager = JobManager(redis_url=settings.redis_url)
|
||||
manager = JobManager(redis_url=redis_test_url)
|
||||
await manager.connect()
|
||||
yield manager
|
||||
await manager.close()
|
||||
|
||||
Reference in New Issue
Block a user