fix: add content_extractor to smart-create endpoint

The POST /wiki/pages/smart-create endpoint was failing with 500
Internal Server Error because HybridRAGService.__init__() was
missing the required content_extractor parameter.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

.gitea/workflows/build.yml

@@ -13,7 +13,7 @@ jobs:
       - name: Login to Gitea Registry
         uses: docker/login-action@v3
         with:
-          registry: git.schweitz.net
+          registry: git.schweitz.internal
           username: ${{ secrets.REGISTRY_USER }}
           password: ${{ secrets.REGISTRY_PASSWORD }}
 
@@ -23,5 +23,11 @@ jobs:
           context: .
           push: true
           tags: |
-            git.schweitz.net/jpmschweitzer/library-desk:latest
-            git.schweitz.net/jpmschweitzer/library-desk:${{ github.ref_name }}
+            git.schweitz.internal/jpmschweitzer/library-desk:latest
+            git.schweitz.internal/jpmschweitzer/library-desk:${{ github.ref_name }}
+
+      - name: Trigger Watchtower update
+        if: success()
+        run: |
+          curl -sf -H "Authorization: Bearer ${{ secrets.WATCHTOWER_TOKEN }}" \
+            http://watchtower:8080/v1/update

AGENTS.md

@@ -0,0 +1,46 @@
+
+# AGENTS.md
+
+> **Start every session by reading this file.**
+> This file outlines the operational protocols, coding standards, and architectural decisions for this FastAPI project.
+
+## 1. Agent Operational Protocols
+
+### 🧠 Work Patterns (Plan-Act-Reflect)
+* **Plan:** Before writing code, briefly outline your plan. Identify which files you will touch and what the side effects might be.
+* **Act:** Execute the changes in small, atomic steps.
+* **Reflect:** After coding, verify your work. Did you break existing tests? Did you add new tests?
+
+### 🛡️ Git Discipline
+* **NEVER commit to `main` or `master` directly.** Always create a feature branch: `feature/your-feature-name` or `fix/issue-description`.
+* **Commit Messages:** Use the [Conventional Commits](https://www.conventionalcommits.org/) format.
+    * `feat: add user login endpoint`
+    * `fix: resolve database connection timeout`
+    * `refactor: split monolith dependency file`
+* **Atomic Commits:** Keep commits small. One logical change = one commit.
+
+### 📝 Changelog Maintenance
+* **Update `CHANGELOG.md`** with every user-facing change.
+* Format: `## [Unreleased] - YYYY-MM-DD` followed by `### Added`, `### Changed`, or `### Fixed`.
+
+---
+
+## 2. FastAPI Architecture & Best Practices
+*Reference: [FastAPI Best Practices](https://github.com/zhanymkanov/fastapi-best-practices)*
+
+### 📂 Project Structure (Directory-based, NOT File-type based)
+Do **not** group files by type (e.g., one huge `routers` folder). Group by **domain/module** inside a `src/` directory.
+
+**Correct Structure:**
+```text
+src/
+├── auth/
+│   ├── router.py      # Endpoints
+│   ├── schemas.py     # Pydantic models
+│   ├── service.py     # Business logic (CRUD, etc.)
+│   ├── dependencies.py# Module-specific dependencies
+│   └── config.py      # Module-specific settings
+├── posts/
+│   ├── router.py
+│   └── ...
+└── main.py            # App entry point

CHANGELOG.md

@@ -0,0 +1,143 @@
+# Changelog
+
+All notable changes to Library Desk will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.3.1] - 2025-12-16
+
+### Fixed
+
+- Smart create endpoint missing `content_extractor` dependency causing 500 errors on `POST /wiki/pages/smart-create`
+
+## [1.3.0] - 2025-12-15
+
+### Changed
+
+- **Two-Stage RRF Architecture** - Major refactor to level the playing field between wiki and web results
+  - Stage 1: Vector and graph results merged into single "wiki" ranking using mini-RRF
+  - Stage 2: Final RRF between wiki (single source) and web (single source)
+  - Wiki pages no longer get 2x advantage from appearing in both vector and graph searches
+  - Multi-source confirmation still determines wiki internal ranking
+
+- **Skip synonyms in graph search** - LLM-generated synonyms (e.g., "author") no longer match unrelated graph entities (e.g., "author2000")
+  - Vector search still uses synonyms for semantic similarity
+  - Graph search uses only core keywords for exact entity matching
+
+### Added
+
+- `VECTOR_SIMILARITY_THRESHOLD` config setting (default: 0.7) to filter weak vector matches
+- Deduplication in graph search to prevent same document appearing multiple times
+
+### Fixed
+
+- Graph search duplicate entity bug where same document could appear twice if entity linked multiple times
+
+## [1.2.1] - 2025-12-15
+
+### Fixed
+
+- HybridRAG router missing `content_extractor` dependency causing 500 errors on `/query/hybrid` endpoint
+
+## [1.2.0] - 2025-12-15
+
+### Added
+
+- **RAG Search Endpoint** (`POST /rag/search`)
+  - Web, news, and image search via SearXNG
+  - Full content extraction using Trafilatura (F1 score 0.958)
+  - Redis caching with configurable TTL
+  - Markdown sources summary for LLM consumption
+  - Returns both extracted content and original snippets
+
+- **Content Extraction Endpoints** (`/content/*`)
+  - `POST /content/extract` - Extract content from a single URL
+  - `POST /content/extract/batch` - Batch extraction (up to 20 URLs)
+  - Reusable ContentExtractor client for use across the codebase
+
+- **HybridRAG Content Extraction Enhancement**
+  - Web search results now include full extracted content via Trafilatura
+  - Falls back to original snippets if extraction fails
+  - Improves context quality for LLM re-ranking and consumption
+
+### Changed
+
+- Added new configuration options:
+  - `SEARCH_CACHE_TTL` - Search cache TTL in seconds (default: 300)
+  - `SEARCH_TIMEOUT` - SearXNG timeout (default: 10s)
+  - `CONTENT_EXTRACTION_TIMEOUT` - Per-URL extraction timeout (default: 5s)
+  - `CONTENT_MAX_LENGTH` - Max extracted content length (default: 2000)
+  - `SEARCH_DEFAULT_LIMIT` - Default search results (default: 10)
+
+### Dependencies
+
+- Added `trafilatura~=1.12.0` for content extraction
+
+## [1.1.3] - 2025-12-14
+
+### Added
+
+- Watchtower update trigger in Gitea workflow after successful build
+
+## [1.1.2] - 2025-12-14
+
+### Fixed
+
+- Updated registry login URL in Gitea workflow (git.schweitz.net → git.schweitz.internal)
+
+## [1.1.1] - 2025-12-14
+
+### Fixed
+
+- Updated container registry tag URLs in Gitea workflow (git.schweitz.net → git.schweitz.internal)
+
+### Added
+
+- Tests for Smart Page Creation feature (`test_smart_create.py`)
+  - Model validation tests for WikiSmartCreateRequest/Response
+  - WikiService.smart_create_page method tests
+  - Bidirectional entity linking utility tests
+  - Endpoint validation tests
+
+## [1.1.0] - 2025-12-11
+
+### Added
+
+- **Smart Page Creation Endpoint** (`POST /wiki/pages/smart-create`)
+  - Combines HybridRAG research with LLM content generation
+  - Searches existing wiki, knowledge graph, and web for topic context
+  - Uses WikiPageWriter to synthesize findings into structured wiki content
+  - Auto-generates page path from topic if not provided
+  - Returns research summary with source counts
+
+- **Bidirectional Entity Linking**
+  - New shared utility (`entity_linking_utils.py`) for reusable entity linking
+  - Forward links: Links entities mentioned in new pages to existing entity pages
+  - Backward links: Updates existing pages that mention the new entity
+  - Runs automatically in background after smart page creation
+
+- **Version Management**
+  - Added `pyproject.toml` with project metadata and version
+  - Version is now read from `pyproject.toml` (single source of truth)
+  - Health check endpoint returns current version
+  - FastAPI docs show current version
+
+### Changed
+
+- Updated `config.py` to read version from `pyproject.toml`
+- Updated `main.py` to use centralized version
+
+## [1.0.0] - 2025-12-10
+
+### Added
+
+- Initial release extracted from portainer-core
+- Wiki page management (`/wiki/pages` CRUD endpoints)
+- HybridRAG search (`/query/hybrid`) with vector, graph, and web search
+- Knowledge graph operations (`/graph/*`)
+- Vector search operations (`/vector/*`)
+- Knowledge consolidation from search results (`/consolidate/knowledge`)
+- Entity linking and extraction
+- Wiki.js change listener for auto-processing user edits
+- Multi-tenant architecture with user namespace isolation

Dockerfile

@@ -12,6 +12,7 @@ COPY requirements.txt .
 RUN pip install --no-cache-dir -r requirements.txt
 
 # Copy application
+COPY pyproject.toml .
 COPY src/ ./src/
 COPY static/ ./static/
 

docs/HYBRIDRAG.md

@@ -0,0 +1,58 @@
+# HybridRAG Architecture
+
+## Overview
+
+HybridRAG combines three search sources to provide comprehensive results:
+- **Vector search** (Qdrant) - Semantic similarity via embeddings
+- **Graph search** (Neo4j) - Entity relationships in knowledge graph
+- **Web search** (SearXNG) - External web results via Trafilatura extraction
+
+## Two-Stage RRF Fusion (v1.3.0+)
+
+To ensure fair ranking between wiki and web results, we use a two-stage Reciprocal Rank Fusion:
+
+```
+Stage 1: Wiki Merge
+  vector results ─┬─→ Mini-RRF ─→ Unified wiki ranking
+  graph results  ─┘
+
+Stage 2: Final RRF
+  wiki (merged)  ─┬─→ Final RRF ─→ Combined results
+  web results    ─┘
+```
+
+**Why two stages?**
+
+Previously, wiki pages found by BOTH vector and graph received double RRF contribution, giving them an unfair 2x advantage over web results. The two-stage approach:
+1. Merges vector+graph into a single "wiki" source
+2. Wiki's internal ranking still benefits from multi-source confirmation
+3. Wiki and web compete as equals in final ranking
+
+## Configuration
+
+| Setting | Default | Description |
+|---------|---------|-------------|
+| `VECTOR_SIMILARITY_THRESHOLD` | 0.7 | Minimum similarity score for vector results |
+| `HYBRID_RAG_VECTOR_LIMIT` | 10 | Max vector results |
+| `HYBRID_RAG_GRAPH_LIMIT` | 10 | Max graph results |
+| `HYBRID_RAG_WEB_LIMIT` | 5 | Max web results |
+
+## Known Limitations & Future Improvements
+
+### Vector Search Noise
+
+**Status:** Open for improvement if needed after observation period.
+
+Vector search may return generic category/index pages (e.g., "Reference", "Projects", "Places") with high similarity scores (~0.86). These pages often have similar boilerplate content leading to uniform scores.
+
+**Potential solutions if this becomes problematic:**
+1. **Raise threshold** - Increase `VECTOR_SIMILARITY_THRESHOLD` to 0.85+
+2. **Page-type filtering** - Exclude pages tagged as category/index/stub
+3. **Content length signal** - Penalize pages with minimal content
+4. **Duplicate score detection** - Flag results with suspiciously identical scores
+
+The LLM re-ranking phase typically demotes these low-quality results, so this may not require immediate action.
+
+### Graph Search
+
+Graph search uses only core keywords (no LLM-generated synonyms) to avoid false matches like "author" → "author2000". This is intentional - vector search handles semantic similarity via embeddings.

pyproject.toml

@@ -0,0 +1,31 @@
+[project]
+name = "library-desk"
+version = "1.3.1"
+description = "Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and knowledge consolidation"
+readme = "README.md"
+requires-python = ">=3.12"
+license = {text = "MIT"}
+authors = [
+    {name = "JP Schweitzer"}
+]
+keywords = ["rag", "knowledge-graph", "wiki", "semantic-search", "neo4j", "qdrant"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Framework :: FastAPI",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+]
+
+[project.urls]
+Homepage = "https://github.com/jpmschweitzer/library-desk"
+Documentation = "https://github.com/jpmschweitzer/library-desk#readme"
+
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["src*"]

requirements.txt

@@ -25,6 +25,9 @@ python-multipart~=0.0.20
 # Utilities
 python-dateutil~=2.9.0
 
+# Content Extraction
+trafilatura~=1.12.0
+
 # Testing
 pytest~=8.3.0
 pytest-asyncio~=0.24.0

src/clients/content_extractor.py

@@ -0,0 +1,310 @@
+"""
+Content extraction client for Library Desk.
+
+A reusable Trafilatura wrapper that can be used throughout library-desk:
+- RAG search service (extract content from search results)
+- Ingestion service (extract content from URLs)
+- Standalone endpoint (ad-hoc content extraction)
+"""
+
+import asyncio
+import logging
+from concurrent.futures import ThreadPoolExecutor
+from typing import List, Optional
+
+import trafilatura
+
+from src.models.content import ContentExtractionResult
+
+logger = logging.getLogger(__name__)
+
+
+class ContentExtractor:
+    """
+    Generic content extraction client using Trafilatura.
+
+    Provides async wrappers around Trafilatura's synchronous extraction,
+    with support for parallel batch processing and configurable timeouts.
+    """
+
+    def __init__(
+        self,
+        timeout: int = 5,
+        max_length: int = 2000,
+        max_workers: int = 10
+    ):
+        """
+        Initialize ContentExtractor.
+
+        Args:
+            timeout: Per-URL timeout in seconds
+            max_length: Maximum content length to return (truncated if longer)
+            max_workers: Max concurrent extractions for batch operations
+        """
+        self.timeout = timeout
+        self.max_length = max_length
+        self._executor = ThreadPoolExecutor(max_workers=max_workers)
+        logger.info(
+            f"Initialized ContentExtractor: timeout={timeout}s, "
+            f"max_length={max_length}, max_workers={max_workers}"
+        )
+
+    def _extract_sync(
+        self,
+        url: str,
+        include_metadata: bool = True,
+        max_length: Optional[int] = None
+    ) -> ContentExtractionResult:
+        """
+        Synchronous extraction (runs in thread pool).
+
+        Args:
+            url: URL to extract content from
+            include_metadata: Whether to extract title, author, date
+            max_length: Override default max length
+
+        Returns:
+            ContentExtractionResult with extracted content or error
+        """
+        effective_max_length = max_length or self.max_length
+
+        try:
+            # Fetch the URL
+            downloaded = trafilatura.fetch_url(url)
+            if not downloaded:
+                return ContentExtractionResult(
+                    url=url,
+                    content="",
+                    success=False,
+                    error="Failed to fetch URL"
+                )
+
+            # Extract content
+            content = trafilatura.extract(
+                downloaded,
+                include_comments=False,
+                include_tables=True,
+                output_format='txt'
+            )
+
+            if not content:
+                return ContentExtractionResult(
+                    url=url,
+                    content="",
+                    success=False,
+                    error="No content extracted"
+                )
+
+            # Truncate if needed
+            if len(content) > effective_max_length:
+                content = content[:effective_max_length] + "..."
+
+            # Extract metadata if requested
+            title = None
+            author = None
+            date = None
+            language = None
+
+            if include_metadata:
+                metadata = trafilatura.extract(
+                    downloaded,
+                    output_format='xml',
+                    include_comments=False
+                )
+                # Parse metadata from XML if available
+                # trafilatura.extract with output_format='xml' returns XML with metadata
+                # For simplicity, we'll use bare_extraction which returns a dict
+                try:
+                    meta_dict = trafilatura.bare_extraction(
+                        downloaded,
+                        include_comments=False
+                    )
+                    if meta_dict:
+                        title = meta_dict.get('title')
+                        author = meta_dict.get('author')
+                        date = meta_dict.get('date')
+                        language = meta_dict.get('language')
+                except Exception as e:
+                    logger.debug(f"Metadata extraction failed for {url}: {e}")
+
+            return ContentExtractionResult(
+                url=url,
+                title=title,
+                content=content,
+                author=author,
+                date=date,
+                language=language,
+                success=True,
+                error=None
+            )
+
+        except Exception as e:
+            logger.error(f"Content extraction failed for {url}: {e}")
+            return ContentExtractionResult(
+                url=url,
+                content="",
+                success=False,
+                error=str(e)
+            )
+
+    async def extract(
+        self,
+        url: str,
+        include_metadata: bool = True,
+        max_length: Optional[int] = None
+    ) -> ContentExtractionResult:
+        """
+        Extract content from a single URL asynchronously.
+
+        Args:
+            url: URL to extract content from
+            include_metadata: Whether to extract title, author, date
+            max_length: Override default max length
+
+        Returns:
+            ContentExtractionResult with extracted content or error
+        """
+        loop = asyncio.get_event_loop()
+
+        try:
+            result = await asyncio.wait_for(
+                loop.run_in_executor(
+                    self._executor,
+                    self._extract_sync,
+                    url,
+                    include_metadata,
+                    max_length
+                ),
+                timeout=self.timeout
+            )
+            return result
+        except asyncio.TimeoutError:
+            logger.warning(f"Content extraction timed out for {url}")
+            return ContentExtractionResult(
+                url=url,
+                content="",
+                success=False,
+                error=f"Extraction timed out after {self.timeout}s"
+            )
+        except Exception as e:
+            logger.error(f"Unexpected error extracting {url}: {e}")
+            return ContentExtractionResult(
+                url=url,
+                content="",
+                success=False,
+                error=str(e)
+            )
+
+    async def extract_batch(
+        self,
+        urls: List[str],
+        include_metadata: bool = True,
+        max_length: Optional[int] = None
+    ) -> List[ContentExtractionResult]:
+        """
+        Extract content from multiple URLs in parallel.
+
+        Args:
+            urls: List of URLs to extract content from
+            include_metadata: Whether to extract title, author, date
+            max_length: Override default max length
+
+        Returns:
+            List of ContentExtractionResult in same order as input URLs
+        """
+        tasks = [
+            self.extract(url, include_metadata, max_length)
+            for url in urls
+        ]
+        results = await asyncio.gather(*tasks)
+        return list(results)
+
+    async def extract_from_html(
+        self,
+        html: str,
+        url: str = "",
+        include_metadata: bool = True,
+        max_length: Optional[int] = None
+    ) -> ContentExtractionResult:
+        """
+        Extract content from raw HTML string.
+
+        Args:
+            html: Raw HTML content
+            url: Optional URL for reference (not fetched)
+            include_metadata: Whether to extract title, author, date
+            max_length: Override default max length
+
+        Returns:
+            ContentExtractionResult with extracted content or error
+        """
+        effective_max_length = max_length or self.max_length
+
+        def _extract():
+            try:
+                content = trafilatura.extract(
+                    html,
+                    include_comments=False,
+                    include_tables=True,
+                    output_format='txt'
+                )
+
+                if not content:
+                    return ContentExtractionResult(
+                        url=url,
+                        content="",
+                        success=False,
+                        error="No content extracted from HTML"
+                    )
+
+                # Truncate if needed
+                if len(content) > effective_max_length:
+                    content = content[:effective_max_length] + "..."
+
+                # Extract metadata
+                title = None
+                author = None
+                date = None
+                language = None
+
+                if include_metadata:
+                    try:
+                        meta_dict = trafilatura.bare_extraction(
+                            html,
+                            include_comments=False
+                        )
+                        if meta_dict:
+                            title = meta_dict.get('title')
+                            author = meta_dict.get('author')
+                            date = meta_dict.get('date')
+                            language = meta_dict.get('language')
+                    except Exception as e:
+                        logger.debug(f"Metadata extraction failed: {e}")
+
+                return ContentExtractionResult(
+                    url=url,
+                    title=title,
+                    content=content,
+                    author=author,
+                    date=date,
+                    language=language,
+                    success=True,
+                    error=None
+                )
+
+            except Exception as e:
+                logger.error(f"HTML content extraction failed: {e}")
+                return ContentExtractionResult(
+                    url=url,
+                    content="",
+                    success=False,
+                    error=str(e)
+                )
+
+        loop = asyncio.get_event_loop()
+        return await loop.run_in_executor(self._executor, _extract)
+
+    async def close(self):
+        """Shutdown the thread pool executor."""
+        self._executor.shutdown(wait=False)
+        logger.info("ContentExtractor closed")

src/config.py

@@ -4,9 +4,20 @@ Following best practices: modular settings, environment-based config.
 """
 
 from functools import lru_cache
+from pathlib import Path
 from pydantic import Field
 from pydantic_settings import BaseSettings, SettingsConfigDict
 
+# Read version from pyproject.toml
+try:
+    import tomllib
+    _pyproject_path = Path(__file__).parent.parent / "pyproject.toml"
+    with open(_pyproject_path, "rb") as f:
+        _pyproject = tomllib.load(f)
+    __version__ = _pyproject["project"]["version"]
+except Exception:
+    __version__ = "0.0.0"  # Fallback if pyproject.toml not found
+
 
 class Settings(BaseSettings):
     """Application settings loaded from environment variables."""
@@ -61,6 +72,7 @@ class Settings(BaseSettings):
     hybrid_rag_vector_limit: int = Field(default=10, ge=1, le=50, description="Vector search limit")
     hybrid_rag_graph_limit: int = Field(default=10, ge=1, le=50, description="Graph search limit")
     hybrid_rag_web_limit: int = Field(default=5, ge=1, le=20, description="Web search limit")
+    vector_similarity_threshold: float = Field(default=0.7, ge=0.0, le=1.0, description="Minimum similarity score for vector results")
 
     # Entity Linking Fuzzy Matching Configuration
     entity_linking_min_confidence: float = Field(default=0.70, ge=0.0, le=1.0, description="Minimum confidence for entity-document matching")
@@ -75,9 +87,18 @@ class Settings(BaseSettings):
 
     # Application
     app_name: str = Field(default="Library Desk", description="Application name")
-    app_version: str = Field(default="1.0.0", description="Application version")
+    app_version: str = Field(default=__version__, description="Application version")
     debug: bool = Field(default=False, description="Debug mode")
 
+    # RAG Search Configuration
+    search_cache_ttl: int = Field(default=300, ge=0, le=3600, description="Search cache TTL in seconds")
+    search_timeout: int = Field(default=10, ge=1, le=60, description="SearXNG timeout in seconds")
+    search_default_limit: int = Field(default=10, ge=1, le=20, description="Default number of search results")
+
+    # Content Extraction Configuration
+    content_extraction_timeout: int = Field(default=5, ge=1, le=30, description="Trafilatura per-URL timeout in seconds")
+    content_max_length: int = Field(default=2000, ge=500, le=10000, description="Max extracted content length per result")
+
     @property
     def qdrant_url(self) -> str:
         """Computed Qdrant URL."""

src/core/dependencies.py

@@ -13,12 +13,15 @@ from typing import Annotated
 from fastapi import Depends
 import logging
 
+import redis.asyncio as aioredis
+
 from src.config import Settings, get_settings
 from src.clients.neo4j_client import Neo4jClient
 from src.clients.qdrant_client import QdrantClientWrapper
 from src.clients.wikijs_client import WikiJSClient
 from src.clients.searxng_client import SearXNGClient
 from src.clients.ollama_client import OllamaClient
+from src.clients.content_extractor import ContentExtractor
 
 logger = logging.getLogger(__name__)
 
@@ -116,6 +119,43 @@ def get_ollama_client() -> OllamaClient:
     return client
 
 
+@lru_cache
+def get_redis_client() -> aioredis.Redis:
+    """
+    Get Redis client singleton for caching.
+
+    Returns:
+        Async Redis client connected to the configured database
+
+    Note: Uses Redis DB 4 (configured for library-desk)
+    """
+    settings = get_settings()
+    client = aioredis.from_url(
+        settings.redis_url,
+        encoding="utf-8",
+        decode_responses=True
+    )
+    logger.debug(f"Created Redis client: {settings.redis_url}")
+    return client
+
+
+@lru_cache
+def get_content_extractor() -> ContentExtractor:
+    """
+    Get ContentExtractor singleton.
+
+    Returns:
+        Initialized content extraction client using Trafilatura
+    """
+    settings = get_settings()
+    extractor = ContentExtractor(
+        timeout=settings.content_extraction_timeout,
+        max_length=settings.content_max_length
+    )
+    logger.debug("Created ContentExtractor instance")
+    return extractor
+
+
 # Type aliases for FastAPI endpoint dependencies
 # Usage: def my_endpoint(neo4j: Neo4jDep):
 Neo4jDep = Annotated[Neo4jClient, Depends(get_neo4j_client)]
@@ -123,6 +163,8 @@ QdrantDep = Annotated[QdrantClientWrapper, Depends(get_qdrant_client)]
 WikiJSDep = Annotated[WikiJSClient, Depends(get_wikijs_client)]
 SearXNGDep = Annotated[SearXNGClient, Depends(get_searxng_client)]
 OllamaDep = Annotated[OllamaClient, Depends(get_ollama_client)]
+RedisDep = Annotated[aioredis.Redis, Depends(get_redis_client)]
+ContentExtractorDep = Annotated[ContentExtractor, Depends(get_content_extractor)]
 
 
 # Lifecycle management functions
@@ -336,6 +378,19 @@ def get_hybrid_rag_service() -> "HybridRAGService":
         graph_service=get_graph_service(),
         searxng_client=get_searxng_client(),
         ollama_client=get_ollama_client(),
+        content_extractor=get_content_extractor(),
+        settings=get_settings()
+    )
+
+
+@lru_cache
+def get_rag_search_service() -> "RAGSearchService":
+    """Get RAGSearchService singleton."""
+    from src.services.rag_search_service import RAGSearchService
+    return RAGSearchService(
+        searxng_client=get_searxng_client(),
+        content_extractor=get_content_extractor(),
+        redis_client=get_redis_client(),
         settings=get_settings()
     )
 

src/main.py

@@ -16,7 +16,7 @@ from typing import Dict, Any
 import logging
 from pathlib import Path
 
-from src.config import Settings, get_settings
+from src.config import Settings, get_settings, __version__
 from src.core.dependencies import verify_api_key
 
 # Configure logging
@@ -30,7 +30,7 @@ logger = logging.getLogger(__name__)
 app = FastAPI(
     title="Library Desk API",
     description="Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and mind map generation",
-    version="1.0.0",
+    version=__version__,
     docs_url="/docs",
     redoc_url="/redoc",
 )
@@ -45,7 +45,10 @@ app.add_middleware(
 )
 
 # Register routers
-from src.routers import wiki, tools, graph, vector, hybrid_rag, consolidation, ingestion, entity_linking, webhooks
+from src.routers import (
+    wiki, tools, graph, vector, hybrid_rag, consolidation,
+    ingestion, entity_linking, webhooks, rag_search, content
+)
 
 app.include_router(wiki.router)
 app.include_router(tools.router)
@@ -56,6 +59,8 @@ app.include_router(consolidation.router)
 app.include_router(ingestion.router)
 app.include_router(entity_linking.router)
 app.include_router(webhooks.router)
+app.include_router(rag_search.router)
+app.include_router(content.router)
 
 # Mount static files directory for Wiki.js integration scripts
 static_dir = Path(__file__).parent.parent / "static"

src/models/content.py

@@ -0,0 +1,69 @@
+"""
+Content extraction models for Library Desk.
+
+Pydantic models for content extraction requests and responses.
+"""
+
+from typing import Optional, List
+from pydantic import BaseModel, Field
+
+
+class ContentExtractionResult(BaseModel):
+    """Result of extracting content from a single URL."""
+
+    url: str = Field(..., description="The URL that was processed")
+    title: Optional[str] = Field(None, description="Page title if extracted")
+    content: str = Field("", description="Extracted main text content")
+    author: Optional[str] = Field(None, description="Author if available")
+    date: Optional[str] = Field(None, description="Publication date if available (ISO format)")
+    language: Optional[str] = Field(None, description="Detected language code")
+    success: bool = Field(..., description="Whether extraction succeeded")
+    error: Optional[str] = Field(None, description="Error message if extraction failed")
+
+
+class ContentExtractionRequest(BaseModel):
+    """Request to extract content from a single URL."""
+
+    url: str = Field(..., min_length=1, description="URL to extract content from")
+    include_metadata: bool = Field(default=True, description="Include title, author, date metadata")
+    max_length: Optional[int] = Field(
+        None,
+        ge=100,
+        le=50000,
+        description="Override default max content length"
+    )
+
+
+class ContentExtractionResponse(BaseModel):
+    """Response for single URL extraction."""
+
+    result: ContentExtractionResult
+    extraction_time_ms: int = Field(..., ge=0, description="Time taken to extract content")
+
+
+class BatchContentExtractionRequest(BaseModel):
+    """Request to extract content from multiple URLs."""
+
+    urls: List[str] = Field(
+        ...,
+        min_length=1,
+        max_length=20,
+        description="URLs to extract content from (max 20)"
+    )
+    include_metadata: bool = Field(default=True, description="Include title, author, date metadata")
+    max_length: Optional[int] = Field(
+        None,
+        ge=100,
+        le=50000,
+        description="Override default max content length"
+    )
+
+
+class BatchContentExtractionResponse(BaseModel):
+    """Response for batch URL extraction."""
+
+    results: List[ContentExtractionResult]
+    total_urls: int = Field(..., ge=0, description="Total number of URLs processed")
+    successful: int = Field(..., ge=0, description="Number of successful extractions")
+    failed: int = Field(..., ge=0, description="Number of failed extractions")
+    extraction_time_ms: int = Field(..., ge=0, description="Total time for batch extraction")

src/models/rag_search.py

@@ -0,0 +1,88 @@
+"""
+RAG search models for Library Desk.
+
+Pydantic models for web/news/image search requests and responses.
+"""
+
+from enum import Enum
+from typing import Optional, List
+from pydantic import BaseModel, Field
+
+from src.core.multi_tenancy import DEFAULT_USER
+
+
+class SearchType(str, Enum):
+    """Supported search types."""
+    WEB = "web"
+    NEWS = "news"
+    IMAGES = "images"
+
+
+class RAGSearchRequest(BaseModel):
+    """Request for RAG search endpoint."""
+
+    query: str = Field(
+        ...,
+        min_length=1,
+        max_length=500,
+        description="The search query"
+    )
+    search_type: SearchType = Field(
+        default=SearchType.WEB,
+        description="Type of search: web, news, or images"
+    )
+    limit: int = Field(
+        default=10,
+        ge=1,
+        le=20,
+        description="Maximum number of results (1-20)"
+    )
+    user: str = Field(
+        default=DEFAULT_USER,
+        description="User identifier for rate limiting/personalization"
+    )
+
+
+class RAGSearchResult(BaseModel):
+    """A single search result with extracted content."""
+
+    title: str = Field(..., description="Title of the result")
+    url: str = Field(..., description="URL of the source")
+    content: str = Field(
+        "",
+        description="Full extracted text via Trafilatura (max ~2000 chars)"
+    )
+    snippet: str = Field(
+        "",
+        description="Original search engine snippet (150-300 chars)"
+    )
+    source: str = Field(..., description="Domain name of the source")
+    published_date: Optional[str] = Field(
+        None,
+        description="Publication date in ISO format if available"
+    )
+
+
+class RAGSearchResponse(BaseModel):
+    """Response from RAG search endpoint."""
+
+    query: str = Field(..., description="Echo of the original query")
+    search_type: SearchType = Field(..., description="Type of search performed")
+    results: List[RAGSearchResult] = Field(
+        default_factory=list,
+        description="List of search results with extracted content"
+    )
+    total_results: int = Field(
+        ...,
+        ge=0,
+        description="Number of results returned"
+    )
+    search_time_ms: int = Field(
+        ...,
+        ge=0,
+        description="Total time for search and content extraction"
+    )
+    sources_summary: str = Field(
+        "",
+        description="Markdown-formatted list of all source URLs"
+    )

src/models/wiki.py

@@ -8,7 +8,7 @@ Models for:
 """
 
 from pydantic import BaseModel, Field, field_validator
-from typing import Optional, List
+from typing import Optional, List, Dict, Any
 from datetime import datetime
 
 
@@ -190,3 +190,44 @@ class DossierOperationResponse(BaseModel):
     dossier_name: str = Field(..., description="Dossier name")
     index_page_id: Optional[int] = Field(None, description="Index page ID (if created)")
     index_page_path: Optional[str] = Field(None, description="Index page path (if created)")
+
+
+# Smart create models (HybridRAG-powered page creation)
+class WikiSmartCreateRequest(BaseModel):
+    """Request model for smart page creation with research."""
+    topic: str = Field(..., min_length=1, max_length=500, description="Topic to research and create page about")
+    path: Optional[str] = Field(None, description="Page path (auto-generated from topic if not provided)")
+    tags: List[str] = Field(default_factory=list, description="Tags for the page")
+    user: Optional[str] = Field(None, description="User identifier")
+    include_web_research: bool = Field(default=True, description="Include web search results")
+    include_wiki_search: bool = Field(default=True, description="Include existing wiki knowledge")
+
+    @field_validator("tags")
+    @classmethod
+    def validate_tags(cls, v: List[str]) -> List[str]:
+        """Validate and clean tags."""
+        cleaned = [tag.strip() for tag in v if tag.strip()]
+        return list(set(cleaned))
+
+    @field_validator("path")
+    @classmethod
+    def validate_path(cls, v: Optional[str]) -> Optional[str]:
+        """Validate page path if provided."""
+        if v is None:
+            return None
+        # Ensure path starts with /
+        if not v.startswith("/"):
+            v = f"/{v}"
+        # Remove trailing slash
+        if v.endswith("/") and v != "/":
+            v = v.rstrip("/")
+        return v
+
+
+class WikiSmartCreateResponse(BaseModel):
+    """Response model for smart page creation."""
+    page: WikiPage = Field(..., description="Created wiki page")
+    research_summary: Dict[str, Any] = Field(..., description="Summary of research used")
+    sources_used: int = Field(..., description="Number of sources incorporated")
+    search_id: Optional[str] = Field(None, description="HybridRAG search ID for reference")
+    entity_linking: Dict[str, int] = Field(default_factory=dict, description="Entity linking statistics")

src/routers/content.py

@@ -0,0 +1,132 @@
+"""
+Content extraction router for Library Desk API.
+
+Endpoints for extracting main content from web URLs using Trafilatura.
+"""
+
+import time
+from fastapi import APIRouter, HTTPException, Depends
+import logging
+
+from src.models.content import (
+    ContentExtractionRequest,
+    ContentExtractionResponse,
+    BatchContentExtractionRequest,
+    BatchContentExtractionResponse,
+)
+from src.clients.content_extractor import ContentExtractor
+from src.core.dependencies import verify_api_key
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/content", tags=["Content Extraction"])
+
+
+# Lazy import to avoid circular dependency
+def get_content_extractor() -> ContentExtractor:
+    """Get content extractor instance."""
+    from src.core.dependencies import get_content_extractor as _get_extractor
+    return _get_extractor()
+
+
+@router.post("/extract", response_model=ContentExtractionResponse)
+async def extract_content(
+    request: ContentExtractionRequest,
+    api_key: str = Depends(verify_api_key)
+):
+    """
+    Extract main content from a single URL.
+
+    Uses Trafilatura to fetch the URL and extract the main text content,
+    removing navigation, ads, and other boilerplate.
+
+    **Example Request:**
+    ```json
+    {
+        "url": "https://example.com/article",
+        "include_metadata": true,
+        "max_length": 2000
+    }
+    ```
+
+    **Returns:** Extracted content with optional metadata (title, author, date)
+    """
+    start_time = time.time()
+
+    try:
+        extractor = get_content_extractor()
+        result = await extractor.extract(
+            url=request.url,
+            include_metadata=request.include_metadata,
+            max_length=request.max_length
+        )
+
+        extraction_time_ms = int((time.time() - start_time) * 1000)
+
+        return ContentExtractionResponse(
+            result=result,
+            extraction_time_ms=extraction_time_ms
+        )
+
+    except ValueError as e:
+        raise HTTPException(status_code=400, detail=str(e))
+    except Exception as e:
+        logger.error(f"Content extraction failed: {e}", exc_info=True)
+        raise HTTPException(status_code=500, detail="Content extraction failed")
+
+
+@router.post("/extract/batch", response_model=BatchContentExtractionResponse)
+async def extract_content_batch(
+    request: BatchContentExtractionRequest,
+    api_key: str = Depends(verify_api_key)
+):
+    """
+    Extract content from multiple URLs in parallel.
+
+    Processes up to 20 URLs concurrently with per-URL timeouts.
+    Failed extractions are included in results with success=false.
+
+    **Example Request:**
+    ```json
+    {
+        "urls": [
+            "https://example.com/article1",
+            "https://example.com/article2"
+        ],
+        "include_metadata": true,
+        "max_length": 2000
+    }
+    ```
+
+    **Returns:** List of extraction results with success/failure counts
+    """
+    start_time = time.time()
+
+    if not request.urls:
+        raise HTTPException(status_code=400, detail="URLs list cannot be empty")
+
+    try:
+        extractor = get_content_extractor()
+        results = await extractor.extract_batch(
+            urls=request.urls,
+            include_metadata=request.include_metadata,
+            max_length=request.max_length
+        )
+
+        extraction_time_ms = int((time.time() - start_time) * 1000)
+        successful = sum(1 for r in results if r.success)
+        failed = len(results) - successful
+
+        return BatchContentExtractionResponse(
+            results=results,
+            total_urls=len(request.urls),
+            successful=successful,
+            failed=failed,
+            extraction_time_ms=extraction_time_ms
+        )
+
+    except ValueError as e:
+        raise HTTPException(status_code=400, detail=str(e))
+    except Exception as e:
+        logger.error(f"Batch content extraction failed: {e}", exc_info=True)
+        raise HTTPException(status_code=500, detail="Batch extraction failed")

src/routers/hybrid_rag.py

@@ -16,7 +16,7 @@ from src.clients.searxng_client import SearXNGClient
 from src.clients.ollama_client import OllamaClient
 from src.core.dependencies import (
     Neo4jDep, WikiJSDep, QdrantDep, OllamaDep,
-    SearXNGDep, verify_api_key, get_settings
+    SearXNGDep, ContentExtractorDep, verify_api_key, get_settings
 )
 from src.config import Settings
 
@@ -32,6 +32,7 @@ def get_hybrid_rag_service(
     qdrant_client: QdrantDep,
     ollama_client: OllamaDep,
     searxng_client: SearXNGDep,
+    content_extractor: ContentExtractorDep,
     settings: Settings = Depends(get_settings)
 ) -> HybridRAGService:
     """Get HybridRAG service instance with all dependencies."""
@@ -48,6 +49,7 @@ def get_hybrid_rag_service(
         graph_service=graph_service,
         searxng_client=searxng_client,
         ollama_client=ollama_client,
+        content_extractor=content_extractor,
         settings=settings
     )
 

src/routers/rag_search.py

@@ -0,0 +1,87 @@
+"""
+RAG search router for Library Desk API.
+
+Endpoints for web, news, and image search with content extraction.
+"""
+
+import httpx
+from fastapi import APIRouter, HTTPException, Depends
+import logging
+
+from src.models.rag_search import RAGSearchRequest, RAGSearchResponse
+from src.services.rag_search_service import RAGSearchService
+from src.core.dependencies import verify_api_key
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/rag", tags=["RAG Search"])
+
+
+# Lazy import to avoid circular dependency
+def get_rag_search_service() -> RAGSearchService:
+    """Get RAG search service instance."""
+    from src.core.dependencies import get_rag_search_service as _get_service
+    return _get_service()
+
+
+@router.post("/search", response_model=RAGSearchResponse)
+async def search(
+    request: RAGSearchRequest,
+    api_key: str = Depends(verify_api_key)
+):
+    """
+    Execute RAG-optimized web search with content extraction.
+
+    Searches via SearXNG and extracts full content from results using
+    Trafilatura. Results are cached in Redis for efficiency.
+
+    **Search Types:**
+    - `web`: General web search (default)
+    - `news`: News articles with recency filtering
+    - `images`: Image search results
+
+    **Example Request:**
+    ```json
+    {
+        "query": "Python async programming best practices",
+        "search_type": "web",
+        "limit": 10,
+        "user": "default"
+    }
+    ```
+
+    **Response includes:**
+    - Full extracted text content per result
+    - Original search snippets
+    - Source domain names
+    - Markdown sources summary for LLM consumption
+
+    **Error Codes:**
+    - 400: Invalid query (empty or too long)
+    - 502: Search provider (SearXNG) error
+    - 504: Search timeout
+    """
+    try:
+        service = get_rag_search_service()
+        response = await service.search(
+            query=request.query,
+            search_type=request.search_type,
+            limit=request.limit,
+            user=request.user
+        )
+        return response
+
+    except ValueError as e:
+        raise HTTPException(status_code=400, detail=str(e))
+
+    except httpx.TimeoutException:
+        logger.error(f"Search timed out for query: {request.query}")
+        raise HTTPException(status_code=504, detail="Search timed out")
+
+    except httpx.HTTPError as e:
+        logger.error(f"Search provider error: {e}")
+        raise HTTPException(status_code=502, detail="Search provider error")
+
+    except Exception as e:
+        logger.error(f"RAG search failed: {e}", exc_info=True)
+        raise HTTPException(status_code=500, detail="Search failed")

src/routers/wiki.py

@@ -13,7 +13,8 @@ import logging
 from src.models.wiki import (
     WikiPage, WikiPageList, WikiPageCreate, WikiPageUpdate, WikiPageMove,
     WikiOperationResponse, WikiSearchResponse,
-    DossierList, WikiSearchResult
+    DossierList, WikiSearchResult,
+    WikiSmartCreateRequest, WikiSmartCreateResponse
 )
 from src.services.wiki_service import WikiService
 from src.services.graph_service import GraphService
@@ -22,8 +23,15 @@ from src.clients.wikijs_client import WikiJSClient
 from src.clients.neo4j_client import Neo4jClient
 from src.clients.qdrant_client import QdrantClientWrapper
 from src.clients.ollama_client import OllamaClient
-from src.core.dependencies import WikiJSDep, Neo4jDep, QdrantDep, OllamaDep, verify_api_key
+from src.core.dependencies import (
+    WikiJSDep, Neo4jDep, QdrantDep, OllamaDep, SearXNGDep, ContentExtractorDep,
+    verify_api_key, get_settings, get_hybrid_rag_service, get_ingestion_service
+)
 from src.core.multi_tenancy import DEFAULT_USER
+from src.services.hybrid_rag_service import HybridRAGService
+from src.services.wiki_page_writer import WikiPageWriter
+from src.services.entity_linking_utils import apply_bidirectional_entity_linking
+from src.config import Settings
 
 logger = logging.getLogger(__name__)
 
@@ -163,6 +171,125 @@ async def create_page(
         raise HTTPException(status_code=500, detail="Internal server error")
 
 
+@router.post("/pages/smart-create", response_model=WikiSmartCreateResponse, status_code=201)
+async def smart_create_page(
+    request: WikiSmartCreateRequest,
+    background_tasks: BackgroundTasks,
+    wiki_client: WikiJSDep,
+    neo4j_client: Neo4jDep,
+    qdrant_client: QdrantDep,
+    ollama_client: OllamaDep,
+    searxng_client: SearXNGDep,
+    content_extractor: ContentExtractorDep,
+    settings: Settings = Depends(get_settings),
+    api_key: str = Depends(verify_api_key)
+):
+    """
+    Create wiki page with intelligent research.
+
+    Combines HybridRAG search with LLM content generation to create
+    rich, well-researched wiki pages in a single API call.
+
+    **Process:**
+    1. Runs HybridRAG search on the topic (wiki + graph + web)
+    2. Uses LLM to synthesize findings into structured wiki content
+    3. Creates the page with proper attribution/sources
+    4. Indexes into vectors + knowledge graph (background)
+    5. Applies bidirectional entity linking (background)
+
+    **Example Request:**
+    ```json
+    {
+        "topic": "Docker orchestration patterns",
+        "path": "/technology/containers/docker-orchestration",
+        "tags": ["technology", "devops", "containers"],
+        "user": "jpmschweitzer",
+        "include_web_research": true,
+        "include_wiki_search": true
+    }
+    ```
+
+    **Returns:**
+    - Created page with ID, path, content
+    - Research summary (wiki/web/graph result counts)
+    - Entity linking statistics (forward/backward links)
+    """
+    try:
+        user = request.user or DEFAULT_USER
+
+        # Build services
+        wiki_service = WikiService(wiki_client)
+        vector_service = VectorService(qdrant_client, wiki_client, ollama_client)
+        graph_service = GraphService(neo4j_client, wiki_client)
+        hybrid_rag_service = HybridRAGService(
+            vector_service=vector_service,
+            graph_service=graph_service,
+            searxng_client=searxng_client,
+            ollama_client=ollama_client,
+            content_extractor=content_extractor,
+            settings=settings
+        )
+        wiki_page_writer = WikiPageWriter(ollama_client=ollama_client)
+
+        # Step 1-5: Research + Generate + Create page
+        page, research_data = await wiki_service.smart_create_page(
+            topic=request.topic,
+            user=user,
+            path=request.path,
+            tags=request.tags,
+            hybrid_rag_service=hybrid_rag_service,
+            wiki_page_writer=wiki_page_writer,
+            include_web=request.include_web_research,
+            include_wiki=request.include_wiki_search
+        )
+
+        # Schedule graph and vector updates in background
+        background_tasks.add_task(
+            graph_service.update_from_page,
+            page_id=page.id,
+            user=user
+        )
+        background_tasks.add_task(
+            vector_service.update_from_page,
+            page_id=page.id,
+            user=user
+        )
+
+        # Schedule bidirectional entity linking in background
+        async def run_entity_linking():
+            ingestion_service = get_ingestion_service()
+            return await apply_bidirectional_entity_linking(
+                page_id=page.id,
+                page_title=page.title,
+                user=user,
+                neo4j_client=neo4j_client,
+                wiki_service=wiki_service,
+                ingestion_service=ingestion_service
+            )
+
+        background_tasks.add_task(run_entity_linking)
+
+        logger.info(
+            f"Smart page created: id={page.id}, path={page.path}, "
+            f"sources={research_data['sources_used']}"
+        )
+
+        return WikiSmartCreateResponse(
+            page=page,
+            research_summary=research_data["research_summary"],
+            sources_used=research_data["sources_used"],
+            search_id=research_data["search_id"],
+            entity_linking={"forward_links": 0, "backward_links": 0, "pages_updated": 0}
+            # Note: entity_linking stats are 0 here as it runs in background
+        )
+
+    except ValueError as e:
+        raise HTTPException(status_code=400, detail=str(e))
+    except Exception as e:
+        logger.error(f"Failed to smart create page: {e}", exc_info=True)
+        raise HTTPException(status_code=500, detail="Internal server error")
+
+
 @router.put("/pages/{page_id}", response_model=WikiPage)
 async def update_page(
     page_id: int,

src/services/entity_linking_utils.py

@@ -0,0 +1,160 @@
+"""
+Shared entity linking utilities for Library Desk.
+
+Provides bidirectional entity linking functionality that can be used by:
+- Consolidation service (knowledge consolidation)
+- Wiki router (smart page creation)
+- Any other service that creates wiki pages
+"""
+import logging
+from typing import Dict, Any, Optional
+
+from src.core.multi_tenancy import get_neo4j_user_base_label
+
+logger = logging.getLogger(__name__)
+
+
+async def apply_bidirectional_entity_linking(
+    page_id: int,
+    page_title: str,
+    user: str,
+    neo4j_client: "Neo4jClient",
+    wiki_service: "WikiService",
+    ingestion_service: Optional["IngestionService"] = None
+) -> Dict[str, int]:
+    """
+    Apply bidirectional entity linking after page creation/update.
+
+    This runs AFTER ingestion so entities are extracted and in the graph.
+
+    Steps:
+    1. Link entities in the new page (forward links to existing entities)
+    2. Find pages that mention the new entity (reverse references)
+    3. Link entities in those pages (backward links to the new entity)
+
+    Args:
+        page_id: Wiki page ID
+        page_title: Page title (used to find reverse references)
+        user: User identifier
+        neo4j_client: Neo4j client for graph queries
+        wiki_service: Wiki service for page operations
+        ingestion_service: Optional ingestion service for re-indexing
+
+    Returns:
+        Dict with link counts: {
+            "forward_links": int,  # Links added to the new page
+            "backward_links": int,  # Links added to other pages pointing to new page
+            "pages_updated": int   # Number of other pages updated
+        }
+    """
+    from src.routers.entity_linking import (
+        link_entities_in_page,
+        EntityLinkingRequest,
+        get_entities_with_paths,
+        add_entity_links_to_content
+    )
+    from src.core.dependencies import get_graph_service, get_wiki_service, get_ingestion_service
+    from src.models.wiki import WikiPageUpdate
+
+    forward_links = 0
+    backward_links = 0
+    pages_updated = 0
+
+    try:
+        graph_service = get_graph_service()
+
+        # Use provided services or get defaults
+        wiki_svc = wiki_service
+        ingestion_svc = ingestion_service or get_ingestion_service()
+
+        # STEP 1: Forward linking - link entities in the new page
+        logger.info(f"Step 1/3: Linking entities in page {page_id} ('{page_title}')")
+        try:
+            forward_result = await link_entities_in_page(
+                request=EntityLinkingRequest(
+                    user=user,
+                    page_id=page_id,
+                    create_relationships=True,
+                    re_index_if_changed=False  # Already indexed, no need to re-index
+                ),
+                wiki_service=wiki_svc,
+                graph_service=graph_service,
+                ingestion_service=ingestion_svc,
+                api_key=""  # Internal call, no auth needed
+            )
+            forward_links = forward_result.content_links_added
+            logger.info(f"Added {forward_links} forward links in page {page_id}")
+        except Exception as e:
+            logger.error(f"Failed to add forward links: {e}")
+
+        # STEP 2: Find reverse references - which pages mention this new entity?
+        logger.info(f"Step 2/3: Finding pages that mention '{page_title}'")
+        user_base_label = get_neo4j_user_base_label(user)
+
+        # Query to find documents that mention entities with this page's title
+        reverse_query = f"""
+        // Find entities with the same name as the page title
+        MATCH (e:{user_base_label})
+        WHERE toLower(e.name) = toLower($title)
+        AND NOT e:Document
+
+        // Find documents that mention those entities
+        MATCH (d:Document)-[r:MENTIONS]->(e)
+        WHERE d.page_id <> $page_id  // Exclude the page itself
+
+        RETURN DISTINCT d.page_id as page_id, d.title as title
+        LIMIT 50
+        """
+
+        try:
+            reverse_refs = await neo4j_client.execute_query(
+                reverse_query,
+                {"title": page_title, "page_id": page_id}
+            )
+            logger.info(f"Found {len(reverse_refs)} pages that mention '{page_title}'")
+        except Exception as e:
+            logger.error(f"Failed to find reverse references: {e}")
+            reverse_refs = []
+
+        # STEP 3: Backward linking - add links in those pages to the new entity
+        if reverse_refs:
+            logger.info(f"Step 3/3: Adding backward links in {len(reverse_refs)} pages")
+            for ref in reverse_refs:
+                try:
+                    backward_result = await link_entities_in_page(
+                        request=EntityLinkingRequest(
+                            user=user,
+                            page_id=ref['page_id'],
+                            create_relationships=False,  # Relationships already exist
+                            re_index_if_changed=False  # Don't re-index for link updates
+                        ),
+                        wiki_service=wiki_svc,
+                        graph_service=graph_service,
+                        ingestion_service=ingestion_svc,
+                        api_key=""
+                    )
+                    if backward_result.content_links_added > 0:
+                        backward_links += backward_result.content_links_added
+                        pages_updated += 1
+                        logger.info(
+                            f"Added {backward_result.content_links_added} links "
+                            f"in page {ref['page_id']} ('{ref['title']}')"
+                        )
+                except Exception as e:
+                    logger.error(f"Failed to add backward links in page {ref['page_id']}: {e}")
+        else:
+            logger.info("Step 3/3: No reverse references found, skipping backward linking")
+
+        return {
+            "forward_links": forward_links,
+            "backward_links": backward_links,
+            "pages_updated": pages_updated
+        }
+
+    except Exception as e:
+        logger.error(f"Bidirectional entity linking failed: {e}", exc_info=True)
+        return {
+            "forward_links": 0,
+            "backward_links": 0,
+            "pages_updated": 0
+        }

src/services/graph_service.py

@@ -1077,8 +1077,18 @@ Feel free to expand it with more details!
                 search_query,
                 {"terms": all_terms, "limit": limit}
             )
-            logger.info(f"Graph search found {len(results)} documents")
-            return results
+
+            # Deduplicate by page_id (safety net for any edge cases)
+            seen_page_ids = set()
+            unique_results = []
+            for r in results:
+                page_id = r.get("page_id")
+                if page_id and page_id not in seen_page_ids:
+                    seen_page_ids.add(page_id)
+                    unique_results.append(r)
+
+            logger.info(f"Graph search found {len(unique_results)} unique documents (raw: {len(results)})")
+            return unique_results
         except Exception as e:
             logger.error(f"Graph document search failed: {e}", exc_info=True)
             return []

src/services/hybrid_rag_service.py

@@ -22,6 +22,7 @@ from src.services.vector_service import VectorService
 from src.services.graph_service import GraphService
 from src.clients.searxng_client import SearXNGClient
 from src.clients.ollama_client import OllamaClient
+from src.clients.content_extractor import ContentExtractor
 from src.config import Settings
 from src.models.hybrid_rag import (
     HybridRAGConfig, HybridRAGRequest, HybridRAGResponse,
@@ -44,6 +45,7 @@ class HybridRAGService:
         graph_service: GraphService,
         searxng_client: SearXNGClient,
         ollama_client: OllamaClient,
+        content_extractor: ContentExtractor,
         settings: Settings
     ):
         """
@@ -54,12 +56,14 @@ class HybridRAGService:
             graph_service: Service for Neo4j graph search
             searxng_client: Client for web search
             ollama_client: Client for LLM (keyword extraction, re-ranking)
+            content_extractor: Client for extracting full content from URLs
             settings: Application settings
         """
         self.vector = vector_service
         self.graph = graph_service
         self.searxng = searxng_client
         self.ollama = ollama_client
+        self.content_extractor = content_extractor
         self.settings = settings
         self.reranker_model = settings.reranker_model
 
@@ -101,14 +105,20 @@ class HybridRAGService:
         timing["graph_ms"] = raw_results.get("timing", {}).get("graph_ms", 0)
         timing["web_ms"] = raw_results.get("timing", {}).get("web_ms", 0)
 
-        # Phase 2: RRF Fusion
+        # Phase 2: Two-Stage RRF Fusion
         phase2_start = time.time()
+
+        # Stage 1: Merge wiki sources (vector + graph) into single ranking
+        wiki_merged = self._merge_wiki_sources(
+            vector_results=raw_results.get("vector", []),
+            graph_results=raw_results.get("graph", []),
+            k=config.rrf_k
+        )
+
+        # Stage 2: Final RRF between wiki and web (equal footing)
         fused_results = self._reciprocal_rank_fusion(
-            results_by_source={
-                "vector": raw_results.get("vector", []),
-                "graph": raw_results.get("graph", []),
-                "web": raw_results.get("web", [])
-            },
+            wiki_results=wiki_merged,
+            web_results=raw_results.get("web", []),
             k=config.rrf_k
         )
         timing["fusion_ms"] = (time.time() - phase2_start) * 1000
@@ -284,7 +294,8 @@ JSON:"""
                     response = await self.vector.search(
                         query=query,
                         user=user,
-                        limit=config.vector_limit
+                        limit=config.vector_limit,
+                        score_threshold=self.settings.vector_similarity_threshold
                     )
                     results = [
                         {
@@ -309,11 +320,19 @@ JSON:"""
             async def graph_search():
                 start = time.time()
                 try:
+                    # Skip synonyms for graph search - only use core keywords
+                    # Synonyms like "author" can match unrelated entities like "author2000"
+                    graph_keywords = {
+                        "core_keywords": keywords_data.get("core_keywords", []),
+                        "entities": keywords_data.get("entities", []),
+                        "synonyms": {},  # No synonyms for exact entity matching
+                        "expansions": {}
+                    }
                     results = await self.graph.search_documents(
                         query=query,
                         user=user,
                         limit=config.graph_limit,
-                        keywords_data=keywords_data
+                        keywords_data=graph_keywords
                     )
                     formatted = [
                         {
@@ -334,7 +353,7 @@ JSON:"""
 
             tasks["graph"] = graph_search()
 
-        # Web search
+        # Web search with content extraction
         if config.enable_web:
             async def web_search():
                 start = time.time()
@@ -343,11 +362,24 @@ JSON:"""
                         query=query,
                         limit=config.web_limit
                     )
+
+                    # Extract full content from URLs using Trafilatura
+                    urls = [r.get("url") for r in results if r.get("url")]
+                    extraction_results = await self.content_extractor.extract_batch(urls)
+
+                    # Map extracted content back to results by URL
+                    url_to_content = {
+                        ext.url: ext.content
+                        for ext in extraction_results
+                        if ext.success and ext.content
+                    }
+
                     formatted = [
                         {
                             "url": r.get("url"),
                             "title": r.get("title", ""),
-                            "content": r.get("content", ""),
+                            "content": url_to_content.get(r.get("url"), r.get("content", "")),
+                            "snippet": r.get("content", ""),  # Keep original snippet
                             "engine": r.get("engine", ""),
                             "source": "web"
                         }
@@ -377,52 +409,134 @@ JSON:"""
 
         return output
 
-    def _reciprocal_rank_fusion(
+    def _merge_wiki_sources(
         self,
-        results_by_source: Dict[str, List],
+        vector_results: List[Dict],
+        graph_results: List[Dict],
         k: int = 60
     ) -> List[Dict[str, Any]]:
         """
-        Phase 2: Merge results using Reciprocal Rank Fusion.
+        Stage 1: Merge vector and graph into single wiki ranking using RRF.
 
-        RRF formula: score = sum(1 / (k + rank)) for each source
+        Both sources search the same wiki pool, so we combine them before
+        final RRF with web to avoid double-counting wiki pages.
 
         Args:
-            results_by_source: Results from each source
+            vector_results: Results from vector search
+            graph_results: Results from graph search
             k: RRF constant (default 60)
 
         Returns:
-            Merged and sorted results
+            Merged wiki results sorted by wiki RRF score
+        """
+        wiki_scores = {}
+
+        # Process vector results
+        for rank, result in enumerate(vector_results, start=1):
+            page_id = result.get("page_id")
+            if not page_id:
+                continue
+            result_id = f"page_{page_id}"
+
+            if result_id not in wiki_scores:
+                wiki_scores[result_id] = {
+                    "result": dict(result),  # Copy to avoid mutation
+                    "wiki_rrf_score": 0.0,
+                    "found_by": []
+                }
+
+            wiki_scores[result_id]["wiki_rrf_score"] += 1 / (k + rank)
+            wiki_scores[result_id]["found_by"].append("vector")
+
+        # Process graph results
+        for rank, result in enumerate(graph_results, start=1):
+            page_id = result.get("page_id")
+            if not page_id:
+                continue
+            result_id = f"page_{page_id}"
+
+            if result_id not in wiki_scores:
+                wiki_scores[result_id] = {
+                    "result": dict(result),
+                    "wiki_rrf_score": 0.0,
+                    "found_by": []
+                }
+
+            wiki_scores[result_id]["wiki_rrf_score"] += 1 / (k + rank)
+            wiki_scores[result_id]["found_by"].append("graph")
+
+            # Add graph metadata to existing result
+            wiki_scores[result_id]["result"]["entity_matches"] = result.get("entity_matches")
+            wiki_scores[result_id]["result"]["matched_entities"] = result.get("matched_entities")
+
+        # Sort by wiki RRF score
+        sorted_wiki = sorted(
+            wiki_scores.values(),
+            key=lambda x: x["wiki_rrf_score"],
+            reverse=True
+        )
+
+        # Return merged results with wiki ranking
+        merged = []
+        for wiki_rank, item in enumerate(sorted_wiki, start=1):
+            merged.append({
+                **item["result"],
+                "wiki_rank": wiki_rank,
+                "wiki_rrf_score": item["wiki_rrf_score"],
+                "found_by": item["found_by"],
+                "source": "wiki"
+            })
+
+        logger.info(f"Wiki merge: {len(merged)} unique pages from vector+graph")
+        return merged
+
+    def _reciprocal_rank_fusion(
+        self,
+        wiki_results: List[Dict],
+        web_results: List[Dict],
+        k: int = 60
+    ) -> List[Dict[str, Any]]:
+        """
+        Stage 2: Final RRF between wiki (single source) and web.
+
+        Wiki results are pre-merged from vector+graph, so wiki and web
+        now compete on equal footing.
+
+        Args:
+            wiki_results: Pre-merged wiki results from _merge_wiki_sources()
+            web_results: Results from web search
+            k: RRF constant (default 60)
+
+        Returns:
+            Final merged and sorted results
         """
         rrf_scores = {}
 
-        for source, results in results_by_source.items():
-            for rank, result in enumerate(results, start=1):
-                # Use page_id for wiki results, url hash for web results
-                if result.get("page_id"):
-                    result_id = f"page_{result['page_id']}"
-                elif result.get("url"):
-                    result_id = f"url_{hash(result['url'])}"
-                else:
-                    continue  # Skip results without ID
+        # Wiki results (single source, already merged)
+        for rank, result in enumerate(wiki_results, start=1):
+            page_id = result.get("page_id")
+            if not page_id:
+                continue
+            result_id = f"page_{page_id}"
+            rrf_scores[result_id] = {
+                "result": result,
+                "rrf_score": 1 / (k + rank),
+                "sources": result.get("found_by", ["wiki"]),
+                "source_type": "wiki"
+            }
 
-                if result_id not in rrf_scores:
-                    rrf_scores[result_id] = {
-                        "result": result,
-                        "rrf_score": 0.0,
-                        "sources": [],
-                        "source_type": source
-                    }
-
-                # RRF formula: sum of 1/(k + rank) across sources
-                rrf_scores[result_id]["rrf_score"] += 1 / (k + rank)
-                rrf_scores[result_id]["sources"].append(source)
-
-                # If result appears in multiple sources, update source_type
-                if len(rrf_scores[result_id]["sources"]) > 1:
-                    rrf_scores[result_id]["source_type"] = "+".join(
-                        sorted(set(rrf_scores[result_id]["sources"]))
-                    )
+        # Web results (single source)
+        for rank, result in enumerate(web_results, start=1):
+            url = result.get("url")
+            if not url:
+                continue
+            result_id = f"url_{hash(url)}"
+            rrf_scores[result_id] = {
+                "result": result,
+                "rrf_score": 1 / (k + rank),
+                "sources": ["web"],
+                "source_type": "web"
+            }
 
         # Sort by RRF score descending
         sorted_results = sorted(
@@ -431,7 +545,7 @@ JSON:"""
             reverse=True
         )
 
-        logger.info(f"RRF fusion: {len(sorted_results)} unique results from {len(results_by_source)} sources")
+        logger.info(f"Final RRF: {len(sorted_results)} results (wiki + web)")
 
         return sorted_results
 

src/services/rag_search_service.py

@@ -0,0 +1,265 @@
+"""
+RAG Search service for Library Desk.
+
+Provides web, news, and image search with content extraction:
+- Uses SearXNG for search queries
+- Uses Trafilatura for content extraction
+- Caches results in Redis
+"""
+
+import hashlib
+import json
+import logging
+import time
+from typing import List, Optional
+from urllib.parse import urlparse
+
+import redis.asyncio as aioredis
+
+from src.clients.searxng_client import SearXNGClient
+from src.clients.content_extractor import ContentExtractor
+from src.config import Settings
+from src.models.rag_search import (
+    SearchType,
+    RAGSearchRequest,
+    RAGSearchResult,
+    RAGSearchResponse,
+)
+
+logger = logging.getLogger(__name__)
+
+
+def extract_domain(url: str) -> str:
+    """Extract domain name from URL, removing 'www.' prefix."""
+    try:
+        parsed = urlparse(url)
+        domain = parsed.netloc
+        return domain.removeprefix("www.")
+    except Exception:
+        return url
+
+
+class RAGSearchService:
+    """
+    Service for RAG-optimized web search with content extraction.
+
+    Combines SearXNG search with Trafilatura content extraction
+    and Redis caching for efficient RAG pipeline integration.
+    """
+
+    def __init__(
+        self,
+        searxng_client: SearXNGClient,
+        content_extractor: ContentExtractor,
+        redis_client: aioredis.Redis,
+        settings: Settings
+    ):
+        """
+        Initialize RAG search service.
+
+        Args:
+            searxng_client: SearXNG search client
+            content_extractor: Trafilatura content extractor
+            redis_client: Async Redis client for caching
+            settings: Application settings
+        """
+        self.searxng = searxng_client
+        self.extractor = content_extractor
+        self.redis = redis_client
+        self.settings = settings
+
+        self.cache_ttl = settings.search_cache_ttl
+        self.default_limit = settings.search_default_limit
+
+        logger.info(
+            f"Initialized RAGSearchService: cache_ttl={self.cache_ttl}s, "
+            f"default_limit={self.default_limit}"
+        )
+
+    def _cache_key(self, query: str, search_type: str, limit: int) -> str:
+        """Generate cache key from search parameters."""
+        key_data = f"{query}:{search_type}:{limit}"
+        key_hash = hashlib.md5(key_data.encode()).hexdigest()
+        return f"rag_search:{key_hash}"
+
+    async def _get_cached_result(self, cache_key: str) -> Optional[RAGSearchResponse]:
+        """Try to get cached search result."""
+        try:
+            cached = await self.redis.get(cache_key)
+            if cached:
+                data = json.loads(cached)
+                logger.debug(f"Cache hit: {cache_key}")
+                return RAGSearchResponse(**data)
+        except Exception as e:
+            logger.warning(f"Cache read failed: {e}")
+        return None
+
+    async def _set_cached_result(self, cache_key: str, result: RAGSearchResponse):
+        """Cache search result."""
+        try:
+            await self.redis.setex(
+                cache_key,
+                self.cache_ttl,
+                result.model_dump_json()
+            )
+            logger.debug(f"Cached result: {cache_key} (TTL={self.cache_ttl}s)")
+        except Exception as e:
+            logger.warning(f"Cache write failed: {e}")
+
+    async def _search_searxng(
+        self,
+        query: str,
+        search_type: SearchType,
+        limit: int
+    ) -> List[dict]:
+        """Execute search via SearXNG based on search type."""
+        try:
+            if search_type == SearchType.WEB:
+                results = await self.searxng.search_general(
+                    query=query,
+                    limit=limit
+                )
+            elif search_type == SearchType.NEWS:
+                results = await self.searxng.search_news(
+                    query=query,
+                    limit=limit
+                )
+            elif search_type == SearchType.IMAGES:
+                results = await self.searxng.search_images(
+                    query=query,
+                    limit=limit
+                )
+            else:
+                results = await self.searxng.search_general(
+                    query=query,
+                    limit=limit
+                )
+
+            return results
+        except Exception as e:
+            logger.error(f"SearXNG search failed: {e}")
+            raise
+
+    async def _extract_content_for_results(
+        self,
+        results: List[dict]
+    ) -> List[RAGSearchResult]:
+        """Extract full content from search result URLs."""
+        # Get URLs for extraction
+        urls = [r.get("url", "") for r in results if r.get("url")]
+
+        # Extract content in parallel
+        extraction_results = await self.extractor.extract_batch(urls)
+
+        # Build result objects
+        search_results = []
+        for i, raw_result in enumerate(results):
+            url = raw_result.get("url", "")
+
+            # Find matching extraction result
+            extracted_content = ""
+            for ext_result in extraction_results:
+                if ext_result.url == url and ext_result.success:
+                    extracted_content = ext_result.content
+                    break
+
+            # Get original snippet
+            snippet = raw_result.get("content", "")
+            if len(snippet) > 300:
+                snippet = snippet[:300] + "..."
+
+            # Build result
+            search_results.append(RAGSearchResult(
+                title=raw_result.get("title", ""),
+                url=url,
+                content=extracted_content,
+                snippet=snippet,
+                source=extract_domain(url),
+                published_date=raw_result.get("publishedDate")
+            ))
+
+        return search_results
+
+    def _generate_sources_summary(self, results: List[RAGSearchResult]) -> str:
+        """Generate markdown list of source URLs."""
+        if not results:
+            return ""
+
+        lines = ["## Sources"]
+        for i, r in enumerate(results, 1):
+            lines.append(f"{i}. [{r.title}]({r.url})")
+
+        return "\n".join(lines)
+
+    async def search(
+        self,
+        query: str,
+        search_type: SearchType = SearchType.WEB,
+        limit: Optional[int] = None,
+        user: str = "default"
+    ) -> RAGSearchResponse:
+        """
+        Execute RAG-optimized search.
+
+        Args:
+            query: Search query string
+            search_type: Type of search (web, news, images)
+            limit: Maximum results to return (default from settings)
+            user: User identifier for logging/rate limiting
+
+        Returns:
+            RAGSearchResponse with extracted content and sources
+
+        Raises:
+            ValueError: If query is empty
+            Exception: If search fails
+        """
+        start_time = time.time()
+
+        if not query or not query.strip():
+            raise ValueError("Query cannot be empty")
+
+        effective_limit = limit or self.default_limit
+
+        # Check cache
+        cache_key = self._cache_key(query, search_type.value, effective_limit)
+        cached = await self._get_cached_result(cache_key)
+        if cached:
+            return cached
+
+        logger.info(
+            f"RAG search: '{query}' type={search_type.value} "
+            f"limit={effective_limit} user={user}"
+        )
+
+        # Execute search
+        raw_results = await self._search_searxng(query, search_type, effective_limit)
+
+        # Extract content from results
+        search_results = await self._extract_content_for_results(raw_results)
+
+        # Generate sources summary
+        sources_summary = self._generate_sources_summary(search_results)
+
+        # Calculate timing
+        search_time_ms = int((time.time() - start_time) * 1000)
+
+        # Build response
+        response = RAGSearchResponse(
+            query=query,
+            search_type=search_type,
+            results=search_results,
+            total_results=len(search_results),
+            search_time_ms=search_time_ms,
+            sources_summary=sources_summary
+        )
+
+        # Cache result
+        await self._set_cached_result(cache_key, response)
+
+        logger.info(
+            f"RAG search completed: {len(search_results)} results "
+            f"in {search_time_ms}ms"
+        )
+
+        return response

src/services/wiki_service.py

@@ -431,3 +431,166 @@ class WikiService:
             WikiPageList filtered by dossier tag
         """
         return await self.list_pages(user, tag=dossier_name, limit=limit)
+
+    async def smart_create_page(
+        self,
+        topic: str,
+        user: str,
+        path: Optional[str],
+        tags: List[str],
+        hybrid_rag_service: "HybridRAGService",
+        wiki_page_writer: "WikiPageWriter",
+        include_web: bool = True,
+        include_wiki: bool = True
+    ) -> tuple["WikiPage", Dict[str, Any]]:
+        """
+        Create wiki page with research from HybridRAG.
+
+        This method combines research + content generation + page creation:
+        1. Run HybridRAG search on topic
+        2. Format results for WikiPageWriter
+        3. Generate page content with LLM
+        4. Create page in Wiki.js
+        5. Return page + research summary
+
+        Args:
+            topic: Topic to research and create page about
+            user: User identifier
+            path: Optional page path (auto-generated from topic if not provided)
+            tags: Tags for the page
+            hybrid_rag_service: HybridRAG service for multi-source search
+            wiki_page_writer: WikiPageWriter for LLM content generation
+            include_web: Include web search results
+            include_wiki: Include existing wiki knowledge
+
+        Returns:
+            Tuple of (created WikiPage, research summary dict)
+        """
+        from src.models.hybrid_rag import HybridRAGConfig
+
+        logger.info(f"Smart create page: topic='{topic}', user='{user}'")
+
+        # Step 1: Run HybridRAG search on the topic
+        config = HybridRAGConfig(
+            enable_vector=include_wiki,
+            enable_graph=include_wiki,
+            enable_web=include_web,
+            enable_reranking=True,
+            enable_enrichment=True,
+            final_result_count=15  # Get more results for rich content
+        )
+
+        search_response = await hybrid_rag_service.search(
+            query=topic,
+            user=user,
+            config=config
+        )
+
+        logger.info(
+            f"HybridRAG search completed: {search_response.total_results} results, "
+            f"search_id={search_response.search_id}"
+        )
+
+        # Step 2: Format results for WikiPageWriter
+        source_information = []
+        wiki_results_count = 0
+        web_results_count = 0
+        graph_entities_count = 0
+
+        for result in search_response.results:
+            source_type = result.source_type
+
+            if "web" in source_type:
+                web_results_count += 1
+                source_information.append({
+                    "title": result.title,
+                    "url": result.url or "",
+                    "content": result.content[:500] if result.content else ""
+                })
+            elif "vector" in source_type or "graph" in source_type:
+                wiki_results_count += 1
+                # For wiki results, use page path as URL
+                source_information.append({
+                    "title": result.title,
+                    "url": f"/{result.page_path}" if result.page_path else "",
+                    "content": result.content[:500] if result.content else ""
+                })
+
+            # Count entities from related dossiers
+            if result.related_dossiers:
+                graph_entities_count += len(result.related_dossiers)
+
+        # Step 3: Generate page content with LLM
+        # Use topic as summary and let WikiPageWriter create structured content
+        topic_summary = f"Research findings about: {topic}"
+        if search_response.keywords:
+            topic_summary += f"\n\nKey concepts: {', '.join(search_response.keywords.core_keywords)}"
+
+        # Extract entities from search results for knowledge graph linking
+        entities = []
+        if search_response.keywords and search_response.keywords.core_keywords:
+            entities = search_response.keywords.core_keywords[:10]
+
+        # Get related documents for cross-linking
+        related_docs = []
+        for result in search_response.results[:5]:
+            if result.page_path:
+                related_docs.append(f"[{result.title}](/{result.page_path})")
+
+        content = await wiki_page_writer.create_page(
+            title=topic,
+            topic_summary=topic_summary,
+            source_information=source_information[:10],  # Limit sources
+            entities=entities,
+            related_docs=related_docs
+        )
+
+        logger.info(f"Generated page content: {len(content)} characters")
+
+        # Step 4: Auto-generate path from topic if not provided
+        if not path:
+            # Convert topic to kebab-case path
+            import re
+            path_slug = topic.lower()
+            path_slug = re.sub(r'[^\w\s-]', '', path_slug)  # Remove special chars
+            path_slug = re.sub(r'\s+', '-', path_slug)  # Spaces to hyphens
+            path_slug = re.sub(r'-+', '-', path_slug)  # Multiple hyphens to single
+            path_slug = path_slug.strip('-')
+
+            # Infer category from tags or use reference
+            category = "reference"
+            if tags:
+                category = tags[0].lower()
+
+            path = f"/{category}/{path_slug}"
+
+        # Step 5: Create page using existing create_page method
+        from src.models.wiki import WikiPageCreate
+
+        page_data = WikiPageCreate(
+            title=topic,
+            path=path,
+            content=content,
+            description=f"Research summary about {topic}",
+            tags=tags,
+            user=user
+        )
+
+        page = await self.create_page(page_data)
+
+        logger.info(f"Created page: id={page.id}, path={page.path}")
+
+        # Build research summary
+        research_summary = {
+            "wiki_results": wiki_results_count,
+            "web_results": web_results_count,
+            "graph_entities": graph_entities_count,
+            "keywords_extracted": len(search_response.keywords.core_keywords) if search_response.keywords else 0,
+            "timing_ms": search_response.timing.total_ms if search_response.timing else 0
+        }
+
+        return page, {
+            "research_summary": research_summary,
+            "sources_used": len(source_information),
+            "search_id": search_response.search_id
+        }

tests/test_content_extractor.py

@@ -0,0 +1,183 @@
+"""Tests for ContentExtractor client."""
+
+import pytest
+from unittest.mock import AsyncMock, MagicMock, patch
+
+from src.clients.content_extractor import ContentExtractor
+from src.models.content import ContentExtractionResult
+
+
+@pytest.fixture
+def content_extractor():
+    """Create ContentExtractor with test configuration."""
+    return ContentExtractor(timeout=5, max_length=2000)
+
+
+class TestContentExtractor:
+    """Tests for ContentExtractor client."""
+
+    def test_init(self, content_extractor):
+        """Test ContentExtractor initialization."""
+        assert content_extractor.timeout == 5
+        assert content_extractor.max_length == 2000
+
+    @pytest.mark.asyncio
+    async def test_extract_success(self, content_extractor):
+        """Test successful content extraction."""
+        test_url = "https://example.com/article"
+        test_content = "This is the extracted article content."
+
+        with patch('src.clients.content_extractor.trafilatura') as mock_traf:
+            mock_traf.fetch_url.return_value = "<html><body>Test</body></html>"
+            mock_traf.extract.return_value = test_content
+            mock_traf.bare_extraction.return_value = {
+                "title": "Test Article",
+                "author": "John Doe",
+                "date": "2024-01-15",
+                "language": "en"
+            }
+
+            result = await content_extractor.extract(test_url)
+
+            assert result.success is True
+            assert result.url == test_url
+            assert result.content == test_content
+            assert result.error is None
+
+    @pytest.mark.asyncio
+    async def test_extract_fetch_failure(self, content_extractor):
+        """Test extraction when URL fetch fails."""
+        test_url = "https://example.com/nonexistent"
+
+        with patch('src.clients.content_extractor.trafilatura') as mock_traf:
+            mock_traf.fetch_url.return_value = None
+
+            result = await content_extractor.extract(test_url)
+
+            assert result.success is False
+            assert result.url == test_url
+            assert result.content == ""
+            assert "Failed to fetch URL" in result.error
+
+    @pytest.mark.asyncio
+    async def test_extract_no_content(self, content_extractor):
+        """Test extraction when page has no extractable content."""
+        test_url = "https://example.com/empty"
+
+        with patch('src.clients.content_extractor.trafilatura') as mock_traf:
+            mock_traf.fetch_url.return_value = "<html><body></body></html>"
+            mock_traf.extract.return_value = None
+
+            result = await content_extractor.extract(test_url)
+
+            assert result.success is False
+            assert "No content extracted" in result.error
+
+    @pytest.mark.asyncio
+    async def test_extract_max_length_truncation(self, content_extractor):
+        """Test that content is truncated to max length."""
+        test_url = "https://example.com/long-article"
+        # Content longer than max_length (2000)
+        long_content = "x" * 3000
+
+        with patch('src.clients.content_extractor.trafilatura') as mock_traf:
+            mock_traf.fetch_url.return_value = "<html><body>Test</body></html>"
+            mock_traf.extract.return_value = long_content
+            mock_traf.bare_extraction.return_value = {}
+
+            result = await content_extractor.extract(test_url)
+
+            assert result.success is True
+            assert len(result.content) <= content_extractor.max_length + 3  # +3 for "..."
+            assert result.content.endswith("...")
+
+    @pytest.mark.asyncio
+    async def test_extract_batch(self, content_extractor):
+        """Test batch extraction of multiple URLs."""
+        test_urls = [
+            "https://example.com/article1",
+            "https://example.com/article2",
+            "https://example.com/article3"
+        ]
+
+        with patch('src.clients.content_extractor.trafilatura') as mock_traf:
+            mock_traf.fetch_url.return_value = "<html><body>Test</body></html>"
+            mock_traf.extract.return_value = "Extracted content"
+            mock_traf.bare_extraction.return_value = {}
+
+            results = await content_extractor.extract_batch(test_urls)
+
+            assert len(results) == 3
+            for i, result in enumerate(results):
+                assert result.url == test_urls[i]
+                assert result.success is True
+
+    @pytest.mark.asyncio
+    async def test_extract_timeout(self):
+        """Test extraction timeout handling."""
+        import time
+
+        test_url = "https://example.com/slow"
+
+        # Create an extractor with very short timeout
+        fast_extractor = ContentExtractor(timeout=0.001, max_length=2000)
+
+        def slow_fetch(url):
+            time.sleep(1)  # Sleep synchronously (this runs in thread pool)
+            return "<html></html>"
+
+        with patch('src.clients.content_extractor.trafilatura') as mock_traf:
+            mock_traf.fetch_url = slow_fetch
+
+            result = await fast_extractor.extract(test_url)
+
+            assert result.success is False
+            assert "timed out" in result.error.lower()
+
+    @pytest.mark.asyncio
+    async def test_extract_from_html(self, content_extractor):
+        """Test extraction from raw HTML."""
+        test_html = "<html><body><article>Article content here.</article></body></html>"
+
+        with patch('src.clients.content_extractor.trafilatura') as mock_traf:
+            mock_traf.extract.return_value = "Article content here."
+            mock_traf.bare_extraction.return_value = {"title": "Test"}
+
+            result = await content_extractor.extract_from_html(test_html, url="https://example.com")
+
+            assert result.success is True
+            assert result.content == "Article content here."
+
+
+class TestContentExtractionResult:
+    """Tests for ContentExtractionResult model."""
+
+    def test_success_result(self):
+        """Test creating a successful result."""
+        result = ContentExtractionResult(
+            url="https://example.com",
+            title="Test Article",
+            content="Article content",
+            author="John Doe",
+            date="2024-01-15",
+            language="en",
+            success=True,
+            error=None
+        )
+
+        assert result.url == "https://example.com"
+        assert result.success is True
+        assert result.error is None
+
+    def test_failure_result(self):
+        """Test creating a failure result."""
+        result = ContentExtractionResult(
+            url="https://example.com/error",
+            content="",
+            success=False,
+            error="Failed to fetch URL"
+        )
+
+        assert result.url == "https://example.com/error"
+        assert result.success is False
+        assert result.error == "Failed to fetch URL"

tests/test_hybrid_rag.py

@@ -25,6 +25,7 @@ from src.clients.qdrant_client import QdrantClientWrapper
 from src.clients.wikijs_client import WikiJSClient
 from src.clients.searxng_client import SearXNGClient
 from src.clients.ollama_client import OllamaClient
+from src.clients.content_extractor import ContentExtractor
 from src.services.hybrid_rag_service import HybridRAGService
 from src.services.vector_service import VectorService
 from src.services.graph_service import GraphService
@@ -83,6 +84,15 @@ def ollama_client(settings) -> OllamaClient:
     return OllamaClient(base_url=settings.ollama_url)
 
 
+@pytest.fixture
+def content_extractor(settings) -> ContentExtractor:
+    """Get ContentExtractor client."""
+    return ContentExtractor(
+        timeout=settings.content_extraction_timeout,
+        max_length=settings.content_max_length
+    )
+
+
 @pytest_asyncio.fixture
 async def vector_service(qdrant_client, wiki_client, ollama_client):
     """Get VectorService instance."""
@@ -101,6 +111,7 @@ async def hybrid_rag_service(
     graph_service,
     searxng_client,
     ollama_client,
+    content_extractor,
     settings
 ):
     """Get HybridRAGService instance."""
@@ -109,6 +120,7 @@ async def hybrid_rag_service(
         graph_service=graph_service,
         searxng_client=searxng_client,
         ollama_client=ollama_client,
+        content_extractor=content_extractor,
         settings=settings
     )
 
@@ -207,53 +219,54 @@ async def test_vector_data(vector_service, test_wiki_page):
 # ============================================================================
 
 class TestRRFFusion:
-    """Test Reciprocal Rank Fusion algorithm."""
+    """Test two-stage Reciprocal Rank Fusion algorithm."""
 
-    def test_rrf_single_source(self, hybrid_rag_service):
-        """Test RRF with single source."""
-        results_by_source = {
-            "vector": [
-                {"page_id": 1, "title": "Doc 1", "content": "test"},
-                {"page_id": 2, "title": "Doc 2", "content": "test"}
-            ]
-        }
+    def test_wiki_merge_single_source(self, hybrid_rag_service):
+        """Test wiki merge with single source (vector only)."""
+        vector_results = [
+            {"page_id": 1, "title": "Doc 1", "content": "test"},
+            {"page_id": 2, "title": "Doc 2", "content": "test"}
+        ]
 
-        fused = hybrid_rag_service._reciprocal_rank_fusion(results_by_source, k=60)
+        merged = hybrid_rag_service._merge_wiki_sources(vector_results, [], k=60)
 
-        assert len(fused) == 2
-        assert fused[0]["rrf_score"] > fused[1]["rrf_score"]  # Rank 1 > Rank 2
-        assert fused[0]["sources"] == ["vector"]
+        assert len(merged) == 2
+        assert merged[0]["wiki_rrf_score"] > merged[1]["wiki_rrf_score"]  # Rank 1 > Rank 2
+        assert merged[0]["found_by"] == ["vector"]
 
-    def test_rrf_multiple_sources_same_doc(self, hybrid_rag_service):
-        """Test RRF with same document from multiple sources."""
-        results_by_source = {
-            "vector": [{"page_id": 1, "title": "Doc 1", "content": "test"}],
-            "graph": [{"page_id": 1, "title": "Doc 1", "content": ""}],
-        }
+    def test_wiki_merge_multiple_sources_same_doc(self, hybrid_rag_service):
+        """Test wiki merge with same document from vector and graph."""
+        vector_results = [{"page_id": 1, "title": "Doc 1", "content": "test"}]
+        graph_results = [{"page_id": 1, "title": "Doc 1", "content": ""}]
 
-        fused = hybrid_rag_service._reciprocal_rank_fusion(results_by_source, k=60)
+        merged = hybrid_rag_service._merge_wiki_sources(vector_results, graph_results, k=60)
 
-        assert len(fused) == 1  # Deduplicated
-        assert len(fused[0]["sources"]) == 2  # Both sources
-        assert "vector" in fused[0]["sources"]
-        assert "graph" in fused[0]["sources"]
-        # RRF score should be sum: 1/(60+1) + 1/(60+1)
+        assert len(merged) == 1  # Deduplicated
+        assert len(merged[0]["found_by"]) == 2  # Both sources
+        assert "vector" in merged[0]["found_by"]
+        assert "graph" in merged[0]["found_by"]
+        # Wiki RRF score should be sum: 1/(60+1) + 1/(60+1)
         expected_score = 1/61 + 1/61
-        assert abs(fused[0]["rrf_score"] - expected_score) < 0.001
+        assert abs(merged[0]["wiki_rrf_score"] - expected_score) < 0.001
 
-    def test_rrf_web_results(self, hybrid_rag_service):
-        """Test RRF with web results (URL-based)."""
-        results_by_source = {
-            "web": [
-                {"url": "https://example.com/1", "title": "Web 1", "content": "test"},
-                {"url": "https://example.com/2", "title": "Web 2", "content": "test"}
-            ]
-        }
+    def test_final_rrf_wiki_and_web(self, hybrid_rag_service):
+        """Test final RRF between wiki and web results."""
+        # Pre-merged wiki results
+        wiki_results = [
+            {"page_id": 1, "title": "Wiki 1", "content": "test", "found_by": ["vector"]}
+        ]
+        web_results = [
+            {"url": "https://example.com/1", "title": "Web 1", "content": "test"},
+            {"url": "https://example.com/2", "title": "Web 2", "content": "test"}
+        ]
 
-        fused = hybrid_rag_service._reciprocal_rank_fusion(results_by_source, k=60)
+        fused = hybrid_rag_service._reciprocal_rank_fusion(wiki_results, web_results, k=60)
 
-        assert len(fused) == 2
-        assert fused[0]["result"]["url"] == "https://example.com/1"
+        assert len(fused) == 3
+        # Wiki rank 1 and web rank 1 should have same RRF score
+        wiki_score = next(r["rrf_score"] for r in fused if r["source_type"] == "wiki")
+        web_score = next(r["rrf_score"] for r in fused if r["source_type"] == "web")
+        assert abs(wiki_score - web_score) < 0.001  # Equal footing
 
 
 class TestContextFormatting:

tests/test_rag_search.py

@@ -0,0 +1,369 @@
+"""Tests for RAG search service and endpoints."""
+
+import pytest
+from unittest.mock import AsyncMock, MagicMock, patch
+
+from src.models.rag_search import (
+    SearchType,
+    RAGSearchRequest,
+    RAGSearchResult,
+    RAGSearchResponse,
+)
+from src.services.rag_search_service import RAGSearchService, extract_domain
+
+
+class TestExtractDomain:
+    """Tests for domain extraction utility."""
+
+    def test_extract_simple_domain(self):
+        """Test extracting domain from simple URL."""
+        assert extract_domain("https://example.com/page") == "example.com"
+
+    def test_extract_domain_with_www(self):
+        """Test extracting domain removes www prefix."""
+        assert extract_domain("https://www.example.com/page") == "example.com"
+
+    def test_extract_domain_with_subdomain(self):
+        """Test extracting domain preserves subdomains."""
+        assert extract_domain("https://blog.example.com/post") == "blog.example.com"
+
+    def test_extract_domain_invalid_url(self):
+        """Test extracting domain from invalid URL returns empty string."""
+        # urlparse returns empty netloc for invalid URLs
+        assert extract_domain("not-a-url") == ""
+
+
+class TestRAGSearchModels:
+    """Tests for RAG search Pydantic models."""
+
+    def test_search_request_defaults(self):
+        """Test RAGSearchRequest with default values."""
+        request = RAGSearchRequest(query="test query")
+
+        assert request.query == "test query"
+        assert request.search_type == SearchType.WEB
+        assert request.limit == 10
+
+    def test_search_request_custom_values(self):
+        """Test RAGSearchRequest with custom values."""
+        request = RAGSearchRequest(
+            query="news about AI",
+            search_type=SearchType.NEWS,
+            limit=5,
+            user="custom_user"
+        )
+
+        assert request.query == "news about AI"
+        assert request.search_type == SearchType.NEWS
+        assert request.limit == 5
+        assert request.user == "custom_user"
+
+    def test_search_result(self):
+        """Test RAGSearchResult model."""
+        result = RAGSearchResult(
+            title="Test Article",
+            url="https://example.com/article",
+            content="Full article content",
+            snippet="Article snippet...",
+            source="example.com",
+            published_date="2024-01-15"
+        )
+
+        assert result.title == "Test Article"
+        assert result.source == "example.com"
+        assert result.published_date == "2024-01-15"
+
+    def test_search_response(self):
+        """Test RAGSearchResponse model."""
+        response = RAGSearchResponse(
+            query="test",
+            search_type=SearchType.WEB,
+            results=[],
+            total_results=0,
+            search_time_ms=100,
+            sources_summary=""
+        )
+
+        assert response.query == "test"
+        assert response.total_results == 0
+        assert response.search_time_ms == 100
+
+
+class TestRAGSearchService:
+    """Tests for RAGSearchService."""
+
+    @pytest.fixture
+    def mock_searxng_client(self):
+        """Create mock SearXNG client."""
+        client = MagicMock()
+        client.search_general = AsyncMock(return_value=[
+            {
+                "title": "Test Result 1",
+                "url": "https://example.com/1",
+                "content": "Snippet 1",
+                "publishedDate": "2024-01-15"
+            },
+            {
+                "title": "Test Result 2",
+                "url": "https://example.com/2",
+                "content": "Snippet 2",
+                "publishedDate": None
+            }
+        ])
+        client.search_news = AsyncMock(return_value=[])
+        client.search_images = AsyncMock(return_value=[])
+        return client
+
+    @pytest.fixture
+    def mock_content_extractor(self):
+        """Create mock ContentExtractor."""
+        from src.models.content import ContentExtractionResult
+
+        extractor = MagicMock()
+        extractor.extract_batch = AsyncMock(return_value=[
+            ContentExtractionResult(
+                url="https://example.com/1",
+                content="Full extracted content 1",
+                success=True
+            ),
+            ContentExtractionResult(
+                url="https://example.com/2",
+                content="Full extracted content 2",
+                success=True
+            )
+        ])
+        return extractor
+
+    @pytest.fixture
+    def mock_redis_client(self):
+        """Create mock Redis client."""
+        redis = MagicMock()
+        redis.get = AsyncMock(return_value=None)  # No cache hit
+        redis.setex = AsyncMock()
+        return redis
+
+    @pytest.fixture
+    def mock_settings(self):
+        """Create mock settings."""
+        settings = MagicMock()
+        settings.search_cache_ttl = 300
+        settings.search_default_limit = 10
+        return settings
+
+    @pytest.fixture
+    def rag_search_service(
+        self,
+        mock_searxng_client,
+        mock_content_extractor,
+        mock_redis_client,
+        mock_settings
+    ):
+        """Create RAGSearchService with mocked dependencies."""
+        return RAGSearchService(
+            searxng_client=mock_searxng_client,
+            content_extractor=mock_content_extractor,
+            redis_client=mock_redis_client,
+            settings=mock_settings
+        )
+
+    @pytest.mark.asyncio
+    async def test_search_basic(self, rag_search_service, mock_searxng_client):
+        """Test basic web search."""
+        response = await rag_search_service.search(
+            query="test query",
+            search_type=SearchType.WEB,
+            limit=10
+        )
+
+        assert response.query == "test query"
+        assert response.search_type == SearchType.WEB
+        assert len(response.results) == 2
+        assert response.total_results == 2
+        assert response.search_time_ms >= 0
+
+        mock_searxng_client.search_general.assert_called_once()
+
+    @pytest.mark.asyncio
+    async def test_search_news(self, rag_search_service, mock_searxng_client):
+        """Test news search type."""
+        mock_searxng_client.search_news.return_value = [
+            {"title": "News", "url": "https://news.com/1", "content": "News content"}
+        ]
+
+        response = await rag_search_service.search(
+            query="latest news",
+            search_type=SearchType.NEWS
+        )
+
+        assert response.search_type == SearchType.NEWS
+        mock_searxng_client.search_news.assert_called_once()
+
+    @pytest.mark.asyncio
+    async def test_search_images(self, rag_search_service, mock_searxng_client):
+        """Test image search type."""
+        mock_searxng_client.search_images.return_value = [
+            {"title": "Image", "url": "https://images.com/1.jpg", "content": ""}
+        ]
+
+        response = await rag_search_service.search(
+            query="cat photos",
+            search_type=SearchType.IMAGES
+        )
+
+        assert response.search_type == SearchType.IMAGES
+        mock_searxng_client.search_images.assert_called_once()
+
+    @pytest.mark.asyncio
+    async def test_search_empty_query(self, rag_search_service):
+        """Test search with empty query raises ValueError."""
+        with pytest.raises(ValueError, match="Query cannot be empty"):
+            await rag_search_service.search(query="", search_type=SearchType.WEB)
+
+    @pytest.mark.asyncio
+    async def test_search_caching_miss(
+        self,
+        rag_search_service,
+        mock_redis_client,
+        mock_searxng_client
+    ):
+        """Test search caches results on cache miss."""
+        mock_redis_client.get.return_value = None  # Cache miss
+
+        await rag_search_service.search(query="test", search_type=SearchType.WEB)
+
+        # Should call SearXNG (cache miss)
+        mock_searxng_client.search_general.assert_called_once()
+        # Should cache result
+        mock_redis_client.setex.assert_called_once()
+
+    @pytest.mark.asyncio
+    async def test_search_caching_hit(
+        self,
+        rag_search_service,
+        mock_redis_client,
+        mock_searxng_client
+    ):
+        """Test search returns cached results on cache hit."""
+        # Simulate cache hit
+        cached_response = RAGSearchResponse(
+            query="test",
+            search_type=SearchType.WEB,
+            results=[],
+            total_results=0,
+            search_time_ms=50,
+            sources_summary=""
+        )
+        mock_redis_client.get.return_value = cached_response.model_dump_json()
+
+        response = await rag_search_service.search(query="test", search_type=SearchType.WEB)
+
+        # Should NOT call SearXNG (cache hit)
+        mock_searxng_client.search_general.assert_not_called()
+        assert response.query == "test"
+
+    @pytest.mark.asyncio
+    async def test_search_content_extraction(
+        self,
+        rag_search_service,
+        mock_content_extractor
+    ):
+        """Test search extracts content from result URLs."""
+        response = await rag_search_service.search(
+            query="test",
+            search_type=SearchType.WEB
+        )
+
+        # Should have called content extractor
+        mock_content_extractor.extract_batch.assert_called_once()
+
+        # Results should have extracted content
+        for result in response.results:
+            assert result.content  # Content should be populated
+
+    @pytest.mark.asyncio
+    async def test_search_sources_summary(self, rag_search_service):
+        """Test search generates sources summary."""
+        response = await rag_search_service.search(
+            query="test",
+            search_type=SearchType.WEB
+        )
+
+        assert response.sources_summary
+        assert "## Sources" in response.sources_summary
+        assert "[Test Result 1]" in response.sources_summary
+
+    @pytest.mark.asyncio
+    async def test_search_limit(self, rag_search_service, mock_searxng_client):
+        """Test search respects limit parameter."""
+        await rag_search_service.search(
+            query="test",
+            search_type=SearchType.WEB,
+            limit=5
+        )
+
+        # Check limit was passed to SearXNG
+        mock_searxng_client.search_general.assert_called_once_with(
+            query="test",
+            limit=5
+        )
+
+
+class TestRAGSearchServiceIntegration:
+    """Integration-style tests (still mocked but test more of the flow)."""
+
+    @pytest.mark.asyncio
+    async def test_full_search_flow(self):
+        """Test full search flow with all components mocked."""
+        from src.models.content import ContentExtractionResult
+
+        # Setup mocks
+        mock_searxng = MagicMock()
+        mock_searxng.search_general = AsyncMock(return_value=[
+            {
+                "title": "Python Tutorial",
+                "url": "https://python.org/tutorial",
+                "content": "Learn Python programming",
+                "publishedDate": "2024-01-10"
+            }
+        ])
+
+        mock_extractor = MagicMock()
+        mock_extractor.extract_batch = AsyncMock(return_value=[
+            ContentExtractionResult(
+                url="https://python.org/tutorial",
+                title="Python Tutorial",
+                content="This is a comprehensive Python tutorial covering basics to advanced topics.",
+                success=True
+            )
+        ])
+
+        mock_redis = MagicMock()
+        mock_redis.get = AsyncMock(return_value=None)
+        mock_redis.setex = AsyncMock()
+
+        mock_settings = MagicMock()
+        mock_settings.search_cache_ttl = 300
+        mock_settings.search_default_limit = 10
+
+        # Create service and execute search
+        service = RAGSearchService(
+            searxng_client=mock_searxng,
+            content_extractor=mock_extractor,
+            redis_client=mock_redis,
+            settings=mock_settings
+        )
+
+        response = await service.search(
+            query="python tutorial",
+            search_type=SearchType.WEB,
+            limit=10,
+            user="test_user"
+        )
+
+        # Verify response
+        assert response.query == "python tutorial"
+        assert len(response.results) == 1
+        assert response.results[0].title == "Python Tutorial"
+        assert response.results[0].source == "python.org"
+        assert "comprehensive Python tutorial" in response.results[0].content
+        assert response.results[0].snippet == "Learn Python programming"

tests/test_smart_create.py

@@ -0,0 +1,553 @@
+"""
+Tests for Smart Page Creation functionality.
+
+Tests the new smart-create feature including:
+- WikiSmartCreateRequest/Response models
+- smart_create_page() method in WikiService
+- Bidirectional entity linking utilities
+- POST /wiki/pages/smart-create endpoint
+"""
+
+import pytest
+from unittest.mock import AsyncMock, MagicMock, patch
+
+from src.models.wiki import (
+    WikiSmartCreateRequest,
+    WikiSmartCreateResponse,
+    WikiPage
+)
+
+
+# =============================================================================
+# Model Tests
+# =============================================================================
+
+class TestWikiSmartCreateRequest:
+    """Tests for WikiSmartCreateRequest model validation."""
+
+    def test_minimal_request(self):
+        """Test request with only required field."""
+        request = WikiSmartCreateRequest(topic="Docker containers")
+        assert request.topic == "Docker containers"
+        assert request.path is None
+        assert request.tags == []
+        assert request.user is None
+        assert request.include_web_research is True
+        assert request.include_wiki_search is True
+
+    def test_full_request(self):
+        """Test request with all fields."""
+        request = WikiSmartCreateRequest(
+            topic="Kubernetes orchestration",
+            path="/technology/kubernetes",
+            tags=["devops", "containers"],
+            user="testuser",
+            include_web_research=False,
+            include_wiki_search=True
+        )
+        assert request.topic == "Kubernetes orchestration"
+        assert request.path == "/technology/kubernetes"
+        # Tags are deduplicated via set, so order is not guaranteed
+        assert set(request.tags) == {"devops", "containers"}
+        assert request.user == "testuser"
+        assert request.include_web_research is False
+        assert request.include_wiki_search is True
+
+    def test_topic_min_length(self):
+        """Test that topic requires at least 1 character."""
+        with pytest.raises(ValueError):
+            WikiSmartCreateRequest(topic="")
+
+    def test_topic_max_length(self):
+        """Test that topic is limited to 500 characters."""
+        long_topic = "x" * 501
+        with pytest.raises(ValueError):
+            WikiSmartCreateRequest(topic=long_topic)
+
+    def test_path_validation_adds_leading_slash(self):
+        """Test that path without leading slash gets one added."""
+        request = WikiSmartCreateRequest(
+            topic="Test",
+            path="technology/test"
+        )
+        assert request.path == "/technology/test"
+
+    def test_path_validation_removes_trailing_slash(self):
+        """Test that trailing slash is removed."""
+        request = WikiSmartCreateRequest(
+            topic="Test",
+            path="/technology/test/"
+        )
+        assert request.path == "/technology/test"
+
+    def test_tags_deduplication(self):
+        """Test that duplicate tags are removed."""
+        request = WikiSmartCreateRequest(
+            topic="Test",
+            tags=["devops", "devops", "containers", "devops"]
+        )
+        assert len(request.tags) == 2
+        assert "devops" in request.tags
+        assert "containers" in request.tags
+
+    def test_tags_whitespace_cleanup(self):
+        """Test that tag whitespace is cleaned."""
+        request = WikiSmartCreateRequest(
+            topic="Test",
+            tags=["  devops  ", "containers", "  ", ""]
+        )
+        assert "devops" in request.tags
+        assert "containers" in request.tags
+        assert "" not in request.tags
+        assert "  " not in request.tags
+
+
+class TestWikiSmartCreateResponse:
+    """Tests for WikiSmartCreateResponse model."""
+
+    def test_response_structure(self):
+        """Test response model with all fields."""
+        page = WikiPage(
+            id=123,
+            path="/users/test/technology/docker",
+            title="Docker",
+            content="# Docker\n\nContent here",
+            tags=["technology"],
+            is_published=True,
+            created_at="2024-01-15T10:00:00Z",
+            updated_at="2024-01-15T10:00:00Z"
+        )
+
+        response = WikiSmartCreateResponse(
+            page=page,
+            research_summary={
+                "wiki_results": 3,
+                "web_results": 5,
+                "graph_entities": 2
+            },
+            sources_used=8,
+            search_id="test-uuid-123",
+            entity_linking={
+                "forward_links": 4,
+                "backward_links": 2,
+                "pages_updated": 1
+            }
+        )
+
+        assert response.page.id == 123
+        assert response.sources_used == 8
+        assert response.research_summary["wiki_results"] == 3
+        assert response.entity_linking["forward_links"] == 4
+
+    def test_response_default_entity_linking(self):
+        """Test that entity_linking defaults to empty dict."""
+        page = WikiPage(
+            id=1,
+            path="/test",
+            title="Test",
+            content="Content",
+            tags=[],
+            is_published=True,
+            created_at="2024-01-15T10:00:00Z",
+            updated_at="2024-01-15T10:00:00Z"
+        )
+
+        response = WikiSmartCreateResponse(
+            page=page,
+            research_summary={},
+            sources_used=0
+        )
+
+        assert response.entity_linking == {}
+        assert response.search_id is None
+
+
+# =============================================================================
+# WikiService.smart_create_page Tests
+# =============================================================================
+
+class TestSmartCreatePage:
+    """Tests for WikiService.smart_create_page method."""
+
+    @pytest.fixture
+    def mock_hybrid_rag_service(self):
+        """Mock HybridRAG service."""
+        service = AsyncMock()
+
+        # Create mock response
+        mock_response = MagicMock()
+        mock_response.total_results = 5
+        mock_response.search_id = "search-123"
+        mock_response.results = [
+            MagicMock(
+                source_type="vector",
+                title="Existing Docker Page",
+                url=None,
+                page_path="users/testuser/docker-basics",
+                content="Docker is a containerization platform...",
+                related_dossiers=["containers"]
+            ),
+            MagicMock(
+                source_type="web",
+                title="Docker Documentation",
+                url="https://docs.docker.com",
+                page_path=None,
+                content="Official Docker documentation...",
+                related_dossiers=None
+            )
+        ]
+        mock_response.keywords = MagicMock()
+        mock_response.keywords.core_keywords = ["docker", "containers", "virtualization"]
+        mock_response.timing = MagicMock()
+        mock_response.timing.total_ms = 1500
+
+        service.search = AsyncMock(return_value=mock_response)
+        return service
+
+    @pytest.fixture
+    def mock_wiki_page_writer(self):
+        """Mock WikiPageWriter."""
+        writer = AsyncMock()
+        writer.create_page = AsyncMock(return_value="# Docker Containers\n\n## Overview\n\nGenerated content about Docker...")
+        return writer
+
+    @pytest.mark.asyncio
+    async def test_smart_create_basic(
+        self,
+        mock_hybrid_rag_service,
+        mock_wiki_page_writer
+    ):
+        """Test basic smart page creation flow."""
+        from src.services.wiki_service import WikiService
+
+        mock_wiki_client = AsyncMock()
+        wiki_service = WikiService(mock_wiki_client)
+
+        # Mock the create_page method on the service itself
+        mock_page = WikiPage(
+            id=42,
+            path="/users/testuser/technology/docker",
+            title="Docker containers",
+            content="# Docker\n\nGenerated content",
+            tags=["technology"],
+            is_published=True,
+            created_at="2024-01-15T10:00:00Z",
+            updated_at="2024-01-15T10:00:00Z"
+        )
+
+        with patch.object(wiki_service, 'create_page', new_callable=AsyncMock) as mock_create:
+            mock_create.return_value = mock_page
+
+            page, research_data = await wiki_service.smart_create_page(
+                topic="Docker containers",
+                user="testuser",
+                path="/technology/docker",
+                tags=["technology"],
+                hybrid_rag_service=mock_hybrid_rag_service,
+                wiki_page_writer=mock_wiki_page_writer,
+                include_web=True,
+                include_wiki=True
+            )
+
+            # Verify HybridRAG was called
+            mock_hybrid_rag_service.search.assert_called_once()
+
+            # Verify WikiPageWriter was called
+            mock_wiki_page_writer.create_page.assert_called_once()
+
+            # Verify page was created
+            mock_create.assert_called_once()
+
+            # Verify research data
+            assert "research_summary" in research_data
+            assert "sources_used" in research_data
+            assert "search_id" in research_data
+            assert research_data["search_id"] == "search-123"
+
+    @pytest.mark.asyncio
+    async def test_smart_create_auto_generates_path(
+        self,
+        mock_hybrid_rag_service,
+        mock_wiki_page_writer
+    ):
+        """Test that path is auto-generated from topic when not provided."""
+        from src.services.wiki_service import WikiService
+
+        mock_wiki_client = AsyncMock()
+        wiki_service = WikiService(mock_wiki_client)
+
+        mock_page = WikiPage(
+            id=42,
+            path="/users/testuser/tutorials/docker-compose-tutorial",
+            title="Docker Compose Tutorial",
+            content="# Docker Compose\n\nContent",
+            tags=["tutorials"],
+            is_published=True,
+            created_at="2024-01-15T10:00:00Z",
+            updated_at="2024-01-15T10:00:00Z"
+        )
+
+        with patch.object(wiki_service, 'create_page', new_callable=AsyncMock) as mock_create:
+            mock_create.return_value = mock_page
+
+            await wiki_service.smart_create_page(
+                topic="Docker Compose Tutorial",
+                user="testuser",
+                path=None,  # No path provided
+                tags=["tutorials"],
+                hybrid_rag_service=mock_hybrid_rag_service,
+                wiki_page_writer=mock_wiki_page_writer
+            )
+
+            # Check that create_page was called
+            mock_create.assert_called_once()
+            # The WikiPageCreate passed should have auto-generated path
+            call_args = mock_create.call_args[0][0]  # First positional arg
+            assert "docker-compose-tutorial" in call_args.path.lower()
+
+    @pytest.mark.asyncio
+    async def test_smart_create_respects_web_flag(
+        self,
+        mock_hybrid_rag_service,
+        mock_wiki_page_writer
+    ):
+        """Test that include_web flag is passed to HybridRAG."""
+        from src.services.wiki_service import WikiService
+
+        mock_wiki_client = AsyncMock()
+        wiki_service = WikiService(mock_wiki_client)
+
+        mock_page = WikiPage(
+            id=1,
+            path="/test",
+            title="Test",
+            content="Content",
+            tags=[],
+            is_published=True,
+            created_at="2024-01-15T10:00:00Z",
+            updated_at="2024-01-15T10:00:00Z"
+        )
+
+        with patch.object(wiki_service, 'create_page', new_callable=AsyncMock) as mock_create:
+            mock_create.return_value = mock_page
+
+            await wiki_service.smart_create_page(
+                topic="Test",
+                user="testuser",
+                path="/test",
+                tags=[],
+                hybrid_rag_service=mock_hybrid_rag_service,
+                wiki_page_writer=mock_wiki_page_writer,
+                include_web=False,
+                include_wiki=True
+            )
+
+            # Check HybridRAG config
+            call_args = mock_hybrid_rag_service.search.call_args
+            config = call_args[1]["config"]
+            assert config.enable_web is False
+            assert config.enable_vector is True
+
+    @pytest.mark.asyncio
+    async def test_smart_create_counts_sources(
+        self,
+        mock_hybrid_rag_service,
+        mock_wiki_page_writer
+    ):
+        """Test that sources are counted correctly."""
+        from src.services.wiki_service import WikiService
+
+        mock_wiki_client = AsyncMock()
+        wiki_service = WikiService(mock_wiki_client)
+
+        mock_page = WikiPage(
+            id=1,
+            path="/test",
+            title="Test",
+            content="Content",
+            tags=[],
+            is_published=True,
+            created_at="2024-01-15T10:00:00Z",
+            updated_at="2024-01-15T10:00:00Z"
+        )
+
+        with patch.object(wiki_service, 'create_page', new_callable=AsyncMock) as mock_create:
+            mock_create.return_value = mock_page
+
+            page, research_data = await wiki_service.smart_create_page(
+                topic="Test",
+                user="testuser",
+                path="/test",
+                tags=[],
+                hybrid_rag_service=mock_hybrid_rag_service,
+                wiki_page_writer=mock_wiki_page_writer
+            )
+
+            # Should have 2 sources (1 wiki + 1 web from mock)
+            assert research_data["sources_used"] == 2
+            assert research_data["research_summary"]["wiki_results"] == 1
+            assert research_data["research_summary"]["web_results"] == 1
+
+
+# =============================================================================
+# Entity Linking Utils Tests
+# =============================================================================
+
+class TestBidirectionalEntityLinking:
+    """Tests for entity_linking_utils.apply_bidirectional_entity_linking."""
+
+    @pytest.fixture
+    def mock_neo4j_client(self):
+        """Mock Neo4j client."""
+        client = AsyncMock()
+        client.execute_query = AsyncMock(return_value=[])
+        return client
+
+    @pytest.fixture
+    def mock_wiki_service(self):
+        """Mock WikiService."""
+        service = AsyncMock()
+        return service
+
+    @pytest.fixture
+    def mock_ingestion_service(self):
+        """Mock IngestionService."""
+        service = AsyncMock()
+        return service
+
+    @pytest.mark.asyncio
+    async def test_returns_link_counts(
+        self,
+        mock_neo4j_client,
+        mock_wiki_service,
+        mock_ingestion_service
+    ):
+        """Test that function returns proper link count structure."""
+        from src.services.entity_linking_utils import apply_bidirectional_entity_linking
+
+        # Patch at the import location within the module
+        with patch('src.routers.entity_linking.link_entities_in_page') as mock_link:
+            mock_result = MagicMock()
+            mock_result.content_links_added = 3
+            mock_link.return_value = mock_result
+
+            with patch('src.core.dependencies.get_graph_service'):
+                with patch('src.core.dependencies.get_ingestion_service', return_value=mock_ingestion_service):
+                    result = await apply_bidirectional_entity_linking(
+                        page_id=42,
+                        page_title="Docker",
+                        user="testuser",
+                        neo4j_client=mock_neo4j_client,
+                        wiki_service=mock_wiki_service,
+                        ingestion_service=mock_ingestion_service
+                    )
+
+        assert "forward_links" in result
+        assert "backward_links" in result
+        assert "pages_updated" in result
+
+    @pytest.mark.asyncio
+    async def test_handles_no_reverse_references(
+        self,
+        mock_neo4j_client,
+        mock_wiki_service,
+        mock_ingestion_service
+    ):
+        """Test graceful handling when no reverse references found."""
+        from src.services.entity_linking_utils import apply_bidirectional_entity_linking
+
+        # No reverse references
+        mock_neo4j_client.execute_query = AsyncMock(return_value=[])
+
+        with patch('src.routers.entity_linking.link_entities_in_page') as mock_link:
+            mock_result = MagicMock()
+            mock_result.content_links_added = 2
+            mock_link.return_value = mock_result
+
+            with patch('src.core.dependencies.get_graph_service'):
+                with patch('src.core.dependencies.get_ingestion_service', return_value=mock_ingestion_service):
+                    result = await apply_bidirectional_entity_linking(
+                        page_id=42,
+                        page_title="NewEntity",
+                        user="testuser",
+                        neo4j_client=mock_neo4j_client,
+                        wiki_service=mock_wiki_service,
+                        ingestion_service=mock_ingestion_service
+                    )
+
+        assert result["backward_links"] == 0
+        assert result["pages_updated"] == 0
+
+    @pytest.mark.asyncio
+    async def test_handles_errors_gracefully(
+        self,
+        mock_neo4j_client,
+        mock_wiki_service,
+        mock_ingestion_service
+    ):
+        """Test that errors don't crash the function."""
+        from src.services.entity_linking_utils import apply_bidirectional_entity_linking
+
+        with patch('src.routers.entity_linking.link_entities_in_page') as mock_link:
+            mock_link.side_effect = Exception("Test error")
+
+            with patch('src.core.dependencies.get_graph_service'):
+                with patch('src.core.dependencies.get_ingestion_service', return_value=mock_ingestion_service):
+                    result = await apply_bidirectional_entity_linking(
+                        page_id=42,
+                        page_title="Test",
+                        user="testuser",
+                        neo4j_client=mock_neo4j_client,
+                        wiki_service=mock_wiki_service,
+                        ingestion_service=mock_ingestion_service
+                    )
+
+        # Should return zeros, not raise
+        assert result["forward_links"] == 0
+        assert result["backward_links"] == 0
+        assert result["pages_updated"] == 0
+
+
+# =============================================================================
+# Endpoint Tests
+# =============================================================================
+
+class TestSmartCreateEndpoint:
+    """Tests for POST /wiki/pages/smart-create endpoint."""
+
+    @pytest.fixture
+    def mock_clients(self):
+        """Create all mock clients needed for the endpoint."""
+        return {
+            "wiki_client": AsyncMock(),
+            "neo4j_client": AsyncMock(),
+            "qdrant_client": MagicMock(),
+            "ollama_client": AsyncMock(),
+            "searxng_client": AsyncMock()
+        }
+
+    @pytest.mark.asyncio
+    async def test_endpoint_returns_201(self, mock_clients):
+        """Test that successful creation returns 201 status."""
+        from fastapi.testclient import TestClient
+        from unittest.mock import patch
+
+        # This test would require more setup with FastAPI TestClient
+        # For now, we test the model validation
+        request = WikiSmartCreateRequest(
+            topic="Test Topic",
+            tags=["test"]
+        )
+        assert request.topic == "Test Topic"
+
+    def test_request_validation_rejects_empty_topic(self):
+        """Test that empty topic is rejected."""
+        with pytest.raises(ValueError):
+            WikiSmartCreateRequest(topic="")
+
+    def test_request_accepts_minimal_input(self):
+        """Test that only topic is required."""
+        request = WikiSmartCreateRequest(topic="Minimal test")
+        assert request.topic == "Minimal test"
+        assert request.include_web_research is True  # default
+        assert request.include_wiki_search is True  # default