Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0f224b460e | ||
|
|
3deed7cbcb | ||
|
|
e05e7aeae3 |
@@ -0,0 +1,45 @@
|
||||
# AGENTS.md
|
||||
|
||||
> **Start every session by reading this file.**
|
||||
> This file outlines the operational protocols, coding standards, and architectural decisions for this FastAPI project.
|
||||
|
||||
## 1. Agent Operational Protocols
|
||||
|
||||
### 🧠 Work Patterns (Plan-Act-Reflect)
|
||||
* **Plan:** Before writing code, briefly outline your plan. Identify which files you will touch and what the side effects might be.
|
||||
* **Act:** Execute the changes in small, atomic steps.
|
||||
* **Reflect:** After coding, verify your work. Did you break existing tests? Did you add new tests?
|
||||
|
||||
### 🛡️ Git Discipline
|
||||
* **NEVER commit to `main` or `master` directly.** Always create a feature branch: `feature/your-feature-name` or `fix/issue-description`.
|
||||
* **Commit Messages:** Use the [Conventional Commits](https://www.conventionalcommits.org/) format.
|
||||
* `feat: add user login endpoint`
|
||||
* `fix: resolve database connection timeout`
|
||||
* `refactor: split monolith dependency file`
|
||||
* **Atomic Commits:** Keep commits small. One logical change = one commit.
|
||||
|
||||
### 📝 Changelog Maintenance
|
||||
* **Update `CHANGELOG.md`** with every user-facing change.
|
||||
* Format: `## [Unreleased] - YYYY-MM-DD` followed by `### Added`, `### Changed`, or `### Fixed`.
|
||||
|
||||
---
|
||||
|
||||
## 2. FastAPI Architecture & Best Practices
|
||||
*Reference: [FastAPI Best Practices](https://github.com/zhanymkanov/fastapi-best-practices)*
|
||||
|
||||
### 📂 Project Structure (Directory-based, NOT File-type based)
|
||||
Do **not** group files by type (e.g., one huge `routers` folder). Group by **domain/module** inside a `src/` directory.
|
||||
|
||||
**Correct Structure:**
|
||||
```text
|
||||
src/
|
||||
├── auth/
|
||||
│ ├── router.py # Endpoints
|
||||
│ ├── schemas.py # Pydantic models
|
||||
│ ├── service.py # Business logic (CRUD, etc.)
|
||||
│ ├── dependencies.py# Module-specific dependencies
|
||||
│ └── config.py # Module-specific settings
|
||||
├── posts/
|
||||
│ ├── router.py
|
||||
│ └── ...
|
||||
└── main.py # App entry point
|
||||
@@ -0,0 +1,48 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to Library Desk will be documented in this file.
|
||||
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [1.1.0] - 2025-12-11
|
||||
|
||||
### Added
|
||||
|
||||
- **Smart Page Creation Endpoint** (`POST /wiki/pages/smart-create`)
|
||||
- Combines HybridRAG research with LLM content generation
|
||||
- Searches existing wiki, knowledge graph, and web for topic context
|
||||
- Uses WikiPageWriter to synthesize findings into structured wiki content
|
||||
- Auto-generates page path from topic if not provided
|
||||
- Returns research summary with source counts
|
||||
|
||||
- **Bidirectional Entity Linking**
|
||||
- New shared utility (`entity_linking_utils.py`) for reusable entity linking
|
||||
- Forward links: Links entities mentioned in new pages to existing entity pages
|
||||
- Backward links: Updates existing pages that mention the new entity
|
||||
- Runs automatically in background after smart page creation
|
||||
|
||||
- **Version Management**
|
||||
- Added `pyproject.toml` with project metadata and version
|
||||
- Version is now read from `pyproject.toml` (single source of truth)
|
||||
- Health check endpoint returns current version
|
||||
- FastAPI docs show current version
|
||||
|
||||
### Changed
|
||||
|
||||
- Updated `config.py` to read version from `pyproject.toml`
|
||||
- Updated `main.py` to use centralized version
|
||||
|
||||
## [1.0.0] - 2025-12-10
|
||||
|
||||
### Added
|
||||
|
||||
- Initial release extracted from portainer-core
|
||||
- Wiki page management (`/wiki/pages` CRUD endpoints)
|
||||
- HybridRAG search (`/query/hybrid`) with vector, graph, and web search
|
||||
- Knowledge graph operations (`/graph/*`)
|
||||
- Vector search operations (`/vector/*`)
|
||||
- Knowledge consolidation from search results (`/consolidate/knowledge`)
|
||||
- Entity linking and extraction
|
||||
- Wiki.js change listener for auto-processing user edits
|
||||
- Multi-tenant architecture with user namespace isolation
|
||||
@@ -0,0 +1,31 @@
|
||||
[project]
|
||||
name = "library-desk"
|
||||
version = "1.1.0"
|
||||
description = "Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and knowledge consolidation"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
license = {text = "MIT"}
|
||||
authors = [
|
||||
{name = "JP Schweitzer"}
|
||||
]
|
||||
keywords = ["rag", "knowledge-graph", "wiki", "semantic-search", "neo4j", "qdrant"]
|
||||
classifiers = [
|
||||
"Development Status :: 4 - Beta",
|
||||
"Framework :: FastAPI",
|
||||
"Intended Audience :: Developers",
|
||||
"License :: OSI Approved :: MIT License",
|
||||
"Programming Language :: Python :: 3",
|
||||
"Programming Language :: Python :: 3.12",
|
||||
]
|
||||
|
||||
[project.urls]
|
||||
Homepage = "https://github.com/jpmschweitzer/library-desk"
|
||||
Documentation = "https://github.com/jpmschweitzer/library-desk#readme"
|
||||
|
||||
[build-system]
|
||||
requires = ["setuptools>=61.0"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["."]
|
||||
include = ["src*"]
|
||||
+12
-1
@@ -4,9 +4,20 @@ Following best practices: modular settings, environment-based config.
|
||||
"""
|
||||
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from pydantic import Field
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
# Read version from pyproject.toml
|
||||
try:
|
||||
import tomllib
|
||||
_pyproject_path = Path(__file__).parent.parent / "pyproject.toml"
|
||||
with open(_pyproject_path, "rb") as f:
|
||||
_pyproject = tomllib.load(f)
|
||||
__version__ = _pyproject["project"]["version"]
|
||||
except Exception:
|
||||
__version__ = "0.0.0" # Fallback if pyproject.toml not found
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
"""Application settings loaded from environment variables."""
|
||||
@@ -75,7 +86,7 @@ class Settings(BaseSettings):
|
||||
|
||||
# Application
|
||||
app_name: str = Field(default="Library Desk", description="Application name")
|
||||
app_version: str = Field(default="1.0.0", description="Application version")
|
||||
app_version: str = Field(default=__version__, description="Application version")
|
||||
debug: bool = Field(default=False, description="Debug mode")
|
||||
|
||||
@property
|
||||
|
||||
+2
-2
@@ -16,7 +16,7 @@ from typing import Dict, Any
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
from src.config import Settings, get_settings
|
||||
from src.config import Settings, get_settings, __version__
|
||||
from src.core.dependencies import verify_api_key
|
||||
|
||||
# Configure logging
|
||||
@@ -30,7 +30,7 @@ logger = logging.getLogger(__name__)
|
||||
app = FastAPI(
|
||||
title="Library Desk API",
|
||||
description="Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and mind map generation",
|
||||
version="1.0.0",
|
||||
version=__version__,
|
||||
docs_url="/docs",
|
||||
redoc_url="/redoc",
|
||||
)
|
||||
|
||||
+42
-1
@@ -8,7 +8,7 @@ Models for:
|
||||
"""
|
||||
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
from typing import Optional, List
|
||||
from typing import Optional, List, Dict, Any
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
@@ -190,3 +190,44 @@ class DossierOperationResponse(BaseModel):
|
||||
dossier_name: str = Field(..., description="Dossier name")
|
||||
index_page_id: Optional[int] = Field(None, description="Index page ID (if created)")
|
||||
index_page_path: Optional[str] = Field(None, description="Index page path (if created)")
|
||||
|
||||
|
||||
# Smart create models (HybridRAG-powered page creation)
|
||||
class WikiSmartCreateRequest(BaseModel):
|
||||
"""Request model for smart page creation with research."""
|
||||
topic: str = Field(..., min_length=1, max_length=500, description="Topic to research and create page about")
|
||||
path: Optional[str] = Field(None, description="Page path (auto-generated from topic if not provided)")
|
||||
tags: List[str] = Field(default_factory=list, description="Tags for the page")
|
||||
user: Optional[str] = Field(None, description="User identifier")
|
||||
include_web_research: bool = Field(default=True, description="Include web search results")
|
||||
include_wiki_search: bool = Field(default=True, description="Include existing wiki knowledge")
|
||||
|
||||
@field_validator("tags")
|
||||
@classmethod
|
||||
def validate_tags(cls, v: List[str]) -> List[str]:
|
||||
"""Validate and clean tags."""
|
||||
cleaned = [tag.strip() for tag in v if tag.strip()]
|
||||
return list(set(cleaned))
|
||||
|
||||
@field_validator("path")
|
||||
@classmethod
|
||||
def validate_path(cls, v: Optional[str]) -> Optional[str]:
|
||||
"""Validate page path if provided."""
|
||||
if v is None:
|
||||
return None
|
||||
# Ensure path starts with /
|
||||
if not v.startswith("/"):
|
||||
v = f"/{v}"
|
||||
# Remove trailing slash
|
||||
if v.endswith("/") and v != "/":
|
||||
v = v.rstrip("/")
|
||||
return v
|
||||
|
||||
|
||||
class WikiSmartCreateResponse(BaseModel):
|
||||
"""Response model for smart page creation."""
|
||||
page: WikiPage = Field(..., description="Created wiki page")
|
||||
research_summary: Dict[str, Any] = Field(..., description="Summary of research used")
|
||||
sources_used: int = Field(..., description="Number of sources incorporated")
|
||||
search_id: Optional[str] = Field(None, description="HybridRAG search ID for reference")
|
||||
entity_linking: Dict[str, int] = Field(default_factory=dict, description="Entity linking statistics")
|
||||
|
||||
+127
-2
@@ -13,7 +13,8 @@ import logging
|
||||
from src.models.wiki import (
|
||||
WikiPage, WikiPageList, WikiPageCreate, WikiPageUpdate, WikiPageMove,
|
||||
WikiOperationResponse, WikiSearchResponse,
|
||||
DossierList, WikiSearchResult
|
||||
DossierList, WikiSearchResult,
|
||||
WikiSmartCreateRequest, WikiSmartCreateResponse
|
||||
)
|
||||
from src.services.wiki_service import WikiService
|
||||
from src.services.graph_service import GraphService
|
||||
@@ -22,8 +23,15 @@ from src.clients.wikijs_client import WikiJSClient
|
||||
from src.clients.neo4j_client import Neo4jClient
|
||||
from src.clients.qdrant_client import QdrantClientWrapper
|
||||
from src.clients.ollama_client import OllamaClient
|
||||
from src.core.dependencies import WikiJSDep, Neo4jDep, QdrantDep, OllamaDep, verify_api_key
|
||||
from src.core.dependencies import (
|
||||
WikiJSDep, Neo4jDep, QdrantDep, OllamaDep, SearXNGDep,
|
||||
verify_api_key, get_settings, get_hybrid_rag_service, get_ingestion_service
|
||||
)
|
||||
from src.core.multi_tenancy import DEFAULT_USER
|
||||
from src.services.hybrid_rag_service import HybridRAGService
|
||||
from src.services.wiki_page_writer import WikiPageWriter
|
||||
from src.services.entity_linking_utils import apply_bidirectional_entity_linking
|
||||
from src.config import Settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -163,6 +171,123 @@ async def create_page(
|
||||
raise HTTPException(status_code=500, detail="Internal server error")
|
||||
|
||||
|
||||
@router.post("/pages/smart-create", response_model=WikiSmartCreateResponse, status_code=201)
|
||||
async def smart_create_page(
|
||||
request: WikiSmartCreateRequest,
|
||||
background_tasks: BackgroundTasks,
|
||||
wiki_client: WikiJSDep,
|
||||
neo4j_client: Neo4jDep,
|
||||
qdrant_client: QdrantDep,
|
||||
ollama_client: OllamaDep,
|
||||
searxng_client: SearXNGDep,
|
||||
settings: Settings = Depends(get_settings),
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Create wiki page with intelligent research.
|
||||
|
||||
Combines HybridRAG search with LLM content generation to create
|
||||
rich, well-researched wiki pages in a single API call.
|
||||
|
||||
**Process:**
|
||||
1. Runs HybridRAG search on the topic (wiki + graph + web)
|
||||
2. Uses LLM to synthesize findings into structured wiki content
|
||||
3. Creates the page with proper attribution/sources
|
||||
4. Indexes into vectors + knowledge graph (background)
|
||||
5. Applies bidirectional entity linking (background)
|
||||
|
||||
**Example Request:**
|
||||
```json
|
||||
{
|
||||
"topic": "Docker orchestration patterns",
|
||||
"path": "/technology/containers/docker-orchestration",
|
||||
"tags": ["technology", "devops", "containers"],
|
||||
"user": "jpmschweitzer",
|
||||
"include_web_research": true,
|
||||
"include_wiki_search": true
|
||||
}
|
||||
```
|
||||
|
||||
**Returns:**
|
||||
- Created page with ID, path, content
|
||||
- Research summary (wiki/web/graph result counts)
|
||||
- Entity linking statistics (forward/backward links)
|
||||
"""
|
||||
try:
|
||||
user = request.user or DEFAULT_USER
|
||||
|
||||
# Build services
|
||||
wiki_service = WikiService(wiki_client)
|
||||
vector_service = VectorService(qdrant_client, wiki_client, ollama_client)
|
||||
graph_service = GraphService(neo4j_client, wiki_client)
|
||||
hybrid_rag_service = HybridRAGService(
|
||||
vector_service=vector_service,
|
||||
graph_service=graph_service,
|
||||
searxng_client=searxng_client,
|
||||
ollama_client=ollama_client,
|
||||
settings=settings
|
||||
)
|
||||
wiki_page_writer = WikiPageWriter(ollama_client=ollama_client)
|
||||
|
||||
# Step 1-5: Research + Generate + Create page
|
||||
page, research_data = await wiki_service.smart_create_page(
|
||||
topic=request.topic,
|
||||
user=user,
|
||||
path=request.path,
|
||||
tags=request.tags,
|
||||
hybrid_rag_service=hybrid_rag_service,
|
||||
wiki_page_writer=wiki_page_writer,
|
||||
include_web=request.include_web_research,
|
||||
include_wiki=request.include_wiki_search
|
||||
)
|
||||
|
||||
# Schedule graph and vector updates in background
|
||||
background_tasks.add_task(
|
||||
graph_service.update_from_page,
|
||||
page_id=page.id,
|
||||
user=user
|
||||
)
|
||||
background_tasks.add_task(
|
||||
vector_service.update_from_page,
|
||||
page_id=page.id,
|
||||
user=user
|
||||
)
|
||||
|
||||
# Schedule bidirectional entity linking in background
|
||||
async def run_entity_linking():
|
||||
ingestion_service = get_ingestion_service()
|
||||
return await apply_bidirectional_entity_linking(
|
||||
page_id=page.id,
|
||||
page_title=page.title,
|
||||
user=user,
|
||||
neo4j_client=neo4j_client,
|
||||
wiki_service=wiki_service,
|
||||
ingestion_service=ingestion_service
|
||||
)
|
||||
|
||||
background_tasks.add_task(run_entity_linking)
|
||||
|
||||
logger.info(
|
||||
f"Smart page created: id={page.id}, path={page.path}, "
|
||||
f"sources={research_data['sources_used']}"
|
||||
)
|
||||
|
||||
return WikiSmartCreateResponse(
|
||||
page=page,
|
||||
research_summary=research_data["research_summary"],
|
||||
sources_used=research_data["sources_used"],
|
||||
search_id=research_data["search_id"],
|
||||
entity_linking={"forward_links": 0, "backward_links": 0, "pages_updated": 0}
|
||||
# Note: entity_linking stats are 0 here as it runs in background
|
||||
)
|
||||
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to smart create page: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail="Internal server error")
|
||||
|
||||
|
||||
@router.put("/pages/{page_id}", response_model=WikiPage)
|
||||
async def update_page(
|
||||
page_id: int,
|
||||
|
||||
@@ -0,0 +1,160 @@
|
||||
"""
|
||||
Shared entity linking utilities for Library Desk.
|
||||
|
||||
Provides bidirectional entity linking functionality that can be used by:
|
||||
- Consolidation service (knowledge consolidation)
|
||||
- Wiki router (smart page creation)
|
||||
- Any other service that creates wiki pages
|
||||
"""
|
||||
import logging
|
||||
from typing import Dict, Any, Optional
|
||||
|
||||
from src.core.multi_tenancy import get_neo4j_user_base_label
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def apply_bidirectional_entity_linking(
|
||||
page_id: int,
|
||||
page_title: str,
|
||||
user: str,
|
||||
neo4j_client: "Neo4jClient",
|
||||
wiki_service: "WikiService",
|
||||
ingestion_service: Optional["IngestionService"] = None
|
||||
) -> Dict[str, int]:
|
||||
"""
|
||||
Apply bidirectional entity linking after page creation/update.
|
||||
|
||||
This runs AFTER ingestion so entities are extracted and in the graph.
|
||||
|
||||
Steps:
|
||||
1. Link entities in the new page (forward links to existing entities)
|
||||
2. Find pages that mention the new entity (reverse references)
|
||||
3. Link entities in those pages (backward links to the new entity)
|
||||
|
||||
Args:
|
||||
page_id: Wiki page ID
|
||||
page_title: Page title (used to find reverse references)
|
||||
user: User identifier
|
||||
neo4j_client: Neo4j client for graph queries
|
||||
wiki_service: Wiki service for page operations
|
||||
ingestion_service: Optional ingestion service for re-indexing
|
||||
|
||||
Returns:
|
||||
Dict with link counts: {
|
||||
"forward_links": int, # Links added to the new page
|
||||
"backward_links": int, # Links added to other pages pointing to new page
|
||||
"pages_updated": int # Number of other pages updated
|
||||
}
|
||||
"""
|
||||
from src.routers.entity_linking import (
|
||||
link_entities_in_page,
|
||||
EntityLinkingRequest,
|
||||
get_entities_with_paths,
|
||||
add_entity_links_to_content
|
||||
)
|
||||
from src.core.dependencies import get_graph_service, get_wiki_service, get_ingestion_service
|
||||
from src.models.wiki import WikiPageUpdate
|
||||
|
||||
forward_links = 0
|
||||
backward_links = 0
|
||||
pages_updated = 0
|
||||
|
||||
try:
|
||||
graph_service = get_graph_service()
|
||||
|
||||
# Use provided services or get defaults
|
||||
wiki_svc = wiki_service
|
||||
ingestion_svc = ingestion_service or get_ingestion_service()
|
||||
|
||||
# STEP 1: Forward linking - link entities in the new page
|
||||
logger.info(f"Step 1/3: Linking entities in page {page_id} ('{page_title}')")
|
||||
try:
|
||||
forward_result = await link_entities_in_page(
|
||||
request=EntityLinkingRequest(
|
||||
user=user,
|
||||
page_id=page_id,
|
||||
create_relationships=True,
|
||||
re_index_if_changed=False # Already indexed, no need to re-index
|
||||
),
|
||||
wiki_service=wiki_svc,
|
||||
graph_service=graph_service,
|
||||
ingestion_service=ingestion_svc,
|
||||
api_key="" # Internal call, no auth needed
|
||||
)
|
||||
forward_links = forward_result.content_links_added
|
||||
logger.info(f"Added {forward_links} forward links in page {page_id}")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to add forward links: {e}")
|
||||
|
||||
# STEP 2: Find reverse references - which pages mention this new entity?
|
||||
logger.info(f"Step 2/3: Finding pages that mention '{page_title}'")
|
||||
user_base_label = get_neo4j_user_base_label(user)
|
||||
|
||||
# Query to find documents that mention entities with this page's title
|
||||
reverse_query = f"""
|
||||
// Find entities with the same name as the page title
|
||||
MATCH (e:{user_base_label})
|
||||
WHERE toLower(e.name) = toLower($title)
|
||||
AND NOT e:Document
|
||||
|
||||
// Find documents that mention those entities
|
||||
MATCH (d:Document)-[r:MENTIONS]->(e)
|
||||
WHERE d.page_id <> $page_id // Exclude the page itself
|
||||
|
||||
RETURN DISTINCT d.page_id as page_id, d.title as title
|
||||
LIMIT 50
|
||||
"""
|
||||
|
||||
try:
|
||||
reverse_refs = await neo4j_client.execute_query(
|
||||
reverse_query,
|
||||
{"title": page_title, "page_id": page_id}
|
||||
)
|
||||
logger.info(f"Found {len(reverse_refs)} pages that mention '{page_title}'")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to find reverse references: {e}")
|
||||
reverse_refs = []
|
||||
|
||||
# STEP 3: Backward linking - add links in those pages to the new entity
|
||||
if reverse_refs:
|
||||
logger.info(f"Step 3/3: Adding backward links in {len(reverse_refs)} pages")
|
||||
for ref in reverse_refs:
|
||||
try:
|
||||
backward_result = await link_entities_in_page(
|
||||
request=EntityLinkingRequest(
|
||||
user=user,
|
||||
page_id=ref['page_id'],
|
||||
create_relationships=False, # Relationships already exist
|
||||
re_index_if_changed=False # Don't re-index for link updates
|
||||
),
|
||||
wiki_service=wiki_svc,
|
||||
graph_service=graph_service,
|
||||
ingestion_service=ingestion_svc,
|
||||
api_key=""
|
||||
)
|
||||
if backward_result.content_links_added > 0:
|
||||
backward_links += backward_result.content_links_added
|
||||
pages_updated += 1
|
||||
logger.info(
|
||||
f"Added {backward_result.content_links_added} links "
|
||||
f"in page {ref['page_id']} ('{ref['title']}')"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to add backward links in page {ref['page_id']}: {e}")
|
||||
else:
|
||||
logger.info("Step 3/3: No reverse references found, skipping backward linking")
|
||||
|
||||
return {
|
||||
"forward_links": forward_links,
|
||||
"backward_links": backward_links,
|
||||
"pages_updated": pages_updated
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Bidirectional entity linking failed: {e}", exc_info=True)
|
||||
return {
|
||||
"forward_links": 0,
|
||||
"backward_links": 0,
|
||||
"pages_updated": 0
|
||||
}
|
||||
@@ -431,3 +431,166 @@ class WikiService:
|
||||
WikiPageList filtered by dossier tag
|
||||
"""
|
||||
return await self.list_pages(user, tag=dossier_name, limit=limit)
|
||||
|
||||
async def smart_create_page(
|
||||
self,
|
||||
topic: str,
|
||||
user: str,
|
||||
path: Optional[str],
|
||||
tags: List[str],
|
||||
hybrid_rag_service: "HybridRAGService",
|
||||
wiki_page_writer: "WikiPageWriter",
|
||||
include_web: bool = True,
|
||||
include_wiki: bool = True
|
||||
) -> tuple["WikiPage", Dict[str, Any]]:
|
||||
"""
|
||||
Create wiki page with research from HybridRAG.
|
||||
|
||||
This method combines research + content generation + page creation:
|
||||
1. Run HybridRAG search on topic
|
||||
2. Format results for WikiPageWriter
|
||||
3. Generate page content with LLM
|
||||
4. Create page in Wiki.js
|
||||
5. Return page + research summary
|
||||
|
||||
Args:
|
||||
topic: Topic to research and create page about
|
||||
user: User identifier
|
||||
path: Optional page path (auto-generated from topic if not provided)
|
||||
tags: Tags for the page
|
||||
hybrid_rag_service: HybridRAG service for multi-source search
|
||||
wiki_page_writer: WikiPageWriter for LLM content generation
|
||||
include_web: Include web search results
|
||||
include_wiki: Include existing wiki knowledge
|
||||
|
||||
Returns:
|
||||
Tuple of (created WikiPage, research summary dict)
|
||||
"""
|
||||
from src.models.hybrid_rag import HybridRAGConfig
|
||||
|
||||
logger.info(f"Smart create page: topic='{topic}', user='{user}'")
|
||||
|
||||
# Step 1: Run HybridRAG search on the topic
|
||||
config = HybridRAGConfig(
|
||||
enable_vector=include_wiki,
|
||||
enable_graph=include_wiki,
|
||||
enable_web=include_web,
|
||||
enable_reranking=True,
|
||||
enable_enrichment=True,
|
||||
final_result_count=15 # Get more results for rich content
|
||||
)
|
||||
|
||||
search_response = await hybrid_rag_service.search(
|
||||
query=topic,
|
||||
user=user,
|
||||
config=config
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"HybridRAG search completed: {search_response.total_results} results, "
|
||||
f"search_id={search_response.search_id}"
|
||||
)
|
||||
|
||||
# Step 2: Format results for WikiPageWriter
|
||||
source_information = []
|
||||
wiki_results_count = 0
|
||||
web_results_count = 0
|
||||
graph_entities_count = 0
|
||||
|
||||
for result in search_response.results:
|
||||
source_type = result.source_type
|
||||
|
||||
if "web" in source_type:
|
||||
web_results_count += 1
|
||||
source_information.append({
|
||||
"title": result.title,
|
||||
"url": result.url or "",
|
||||
"content": result.content[:500] if result.content else ""
|
||||
})
|
||||
elif "vector" in source_type or "graph" in source_type:
|
||||
wiki_results_count += 1
|
||||
# For wiki results, use page path as URL
|
||||
source_information.append({
|
||||
"title": result.title,
|
||||
"url": f"/{result.page_path}" if result.page_path else "",
|
||||
"content": result.content[:500] if result.content else ""
|
||||
})
|
||||
|
||||
# Count entities from related dossiers
|
||||
if result.related_dossiers:
|
||||
graph_entities_count += len(result.related_dossiers)
|
||||
|
||||
# Step 3: Generate page content with LLM
|
||||
# Use topic as summary and let WikiPageWriter create structured content
|
||||
topic_summary = f"Research findings about: {topic}"
|
||||
if search_response.keywords:
|
||||
topic_summary += f"\n\nKey concepts: {', '.join(search_response.keywords.core_keywords)}"
|
||||
|
||||
# Extract entities from search results for knowledge graph linking
|
||||
entities = []
|
||||
if search_response.keywords and search_response.keywords.core_keywords:
|
||||
entities = search_response.keywords.core_keywords[:10]
|
||||
|
||||
# Get related documents for cross-linking
|
||||
related_docs = []
|
||||
for result in search_response.results[:5]:
|
||||
if result.page_path:
|
||||
related_docs.append(f"[{result.title}](/{result.page_path})")
|
||||
|
||||
content = await wiki_page_writer.create_page(
|
||||
title=topic,
|
||||
topic_summary=topic_summary,
|
||||
source_information=source_information[:10], # Limit sources
|
||||
entities=entities,
|
||||
related_docs=related_docs
|
||||
)
|
||||
|
||||
logger.info(f"Generated page content: {len(content)} characters")
|
||||
|
||||
# Step 4: Auto-generate path from topic if not provided
|
||||
if not path:
|
||||
# Convert topic to kebab-case path
|
||||
import re
|
||||
path_slug = topic.lower()
|
||||
path_slug = re.sub(r'[^\w\s-]', '', path_slug) # Remove special chars
|
||||
path_slug = re.sub(r'\s+', '-', path_slug) # Spaces to hyphens
|
||||
path_slug = re.sub(r'-+', '-', path_slug) # Multiple hyphens to single
|
||||
path_slug = path_slug.strip('-')
|
||||
|
||||
# Infer category from tags or use reference
|
||||
category = "reference"
|
||||
if tags:
|
||||
category = tags[0].lower()
|
||||
|
||||
path = f"/{category}/{path_slug}"
|
||||
|
||||
# Step 5: Create page using existing create_page method
|
||||
from src.models.wiki import WikiPageCreate
|
||||
|
||||
page_data = WikiPageCreate(
|
||||
title=topic,
|
||||
path=path,
|
||||
content=content,
|
||||
description=f"Research summary about {topic}",
|
||||
tags=tags,
|
||||
user=user
|
||||
)
|
||||
|
||||
page = await self.create_page(page_data)
|
||||
|
||||
logger.info(f"Created page: id={page.id}, path={page.path}")
|
||||
|
||||
# Build research summary
|
||||
research_summary = {
|
||||
"wiki_results": wiki_results_count,
|
||||
"web_results": web_results_count,
|
||||
"graph_entities": graph_entities_count,
|
||||
"keywords_extracted": len(search_response.keywords.core_keywords) if search_response.keywords else 0,
|
||||
"timing_ms": search_response.timing.total_ms if search_response.timing else 0
|
||||
}
|
||||
|
||||
return page, {
|
||||
"research_summary": research_summary,
|
||||
"sources_used": len(source_information),
|
||||
"search_id": search_response.search_id
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user