Commit Graph
14 Commits
Author SHA1 Message Date
jpmschweitzer 15930a9600 feat(library-desk): implement HybridRAG query system
HybridRAG Service:
- Combine vector (Qdrant), graph (Neo4j), and web (SearXNG) search
- Reciprocal Rank Fusion (RRF) for result merging
- LLM re-ranking with mistral-nemo
- Graph enrichment with related dossiers
- Query enhancement with keyword/synonym extraction
- Search result persistence for offline processing

Router:
- POST /query/hybrid endpoint
- Configurable search limits per source
- Enable/disable individual sources
- Timing breakdown for performance monitoring

Models:
- HybridRAGRequest, HybridRAGResponse
- HybridRAGResult with source tracking
- KeywordExtraction for query analysis
- TimingBreakdown for performance metrics

Tests:
- End-to-end HybridRAG query tests
- RRF fusion algorithm validation
- Multi-source result merging
2025-12-10 01:28:13 +01:00
jpmschweitzer 0bd4f8056a feat(library-desk): add configuration for HybridRAG and enhance multi-tenancy
Configuration:
- Add HybridRAG settings (reranker model, search limits)
- Change Wiki.js auth from API key to username/password
- Configure vector, graph, and web search limits

Multi-tenancy:
- Add get_neo4j_user_base_label() for entity node labeling
- Support title-cased labels following Neo4j conventions
- Maintain namespace isolation for entities vs documents
2025-12-10 01:25:40 +01:00
jpmschweitzer 12fe7e55c0 refactor(library-desk): improve client implementations
Ollama Client:
- Improve model checking to handle :latest tag variants
- Match models with or without explicit tag

Qdrant Client:
- Add collection_exists() method for checking collection presence
- Refactor ensure_collection() to accept collection name directly
- Better separation of concerns

SearXNG Client:
- Add health_check() method for service monitoring
- Simple endpoint check without full search
- Used by health check endpoint
2025-12-10 01:25:07 +01:00
jpmschweitzer 67a124af91 feat(library-desk): add vector service for semantic search
Vector Service:
- Manage document embeddings in Qdrant
- Update vectors from wiki pages
- Handle chunking and embedding generation
- Support force refresh and incremental updates

Vector Models:
- VectorSearchResult for search responses
- VectorUpdateSummary for indexing metrics
- Track chunks created/deleted

Used by ingestion_service for page embedding
2025-12-10 01:21:56 +01:00
jpmschweitzer c2faddf54a feat(library-desk): add ingestion models for page processing
- Add IngestionResult model for single page ingestion
- Add BatchIngestionResult for batch operations
- Track vector chunks, graph entities, and relationships
- Include processing time metrics

Used by ingestion_service for page indexing
2025-12-10 01:21:21 +01:00
jpmschweitzer b777098957 feat(library-desk): add wiki models for page operations
- Add WikiPage model for page data
- Add WikiPageUpdate model for partial updates
- Add field validators for tags and descriptions
- Support optional fields for flexible updates

Used by wiki_service and entity_linking router
2025-12-10 01:20:15 +01:00
jpmschweitzer 7354a66a3d feat(library-desk): integrate entity linking into main app
Main App:
- Mount /static directory for serving Wiki.js integration scripts
- Register entity_linking router
- Refactor API key verification to dependencies module

Dependencies:
- Add service factory functions for all services
- Add get_wiki_service() for wiki operations
- Add get_graph_service() for entity operations
- Add get_ingestion_service() for auto entity linking
- Improve health check for SearXNG
2025-12-10 01:18:40 +01:00
jpmschweitzer d80f063186 feat(library-desk): add Wiki.js integration buttons for entity linking
- Add combined integration script with both re-index and entity linking
- Add standalone entity linking button
- Add standalone re-index button
- Auto-detect Library Desk URL from script tag
- Support both toolbar and floating button positions
- Show real-time status updates and notifications
- Auto-reload page after successful entity linking

Usage: Inject via Wiki.js Code Injection settings
<script src="http://IP:8089/static/wikijs-integration.js"></script>
2025-12-10 01:16:46 +01:00
jpmschweitzer 30b9825365 feat(library-desk): add entity management to graph and ingestion services
Graph Service:
- Add get_all_entities() to retrieve entities with wiki page paths
- Add create_entity_mentions() for MENTIONS relationship creation
- Support entity-to-document linking via title matching

Ingestion Service:
- Add _link_existing_entities() for automatic entity linking
- Auto-link entities during page ingestion
- Support skip_entity_linking parameter for granular control
2025-12-10 01:16:24 +01:00
jpmschweitzer 5a4a203beb fix(library-desk): preserve published status when updating wiki pages
- Add is_published parameter to WikiJS client update_page() method
- Update wiki_service to always pass is_published=True
- Prevents pages from being unpublished during entity linking updates
- Important for internal wikis where all pages should remain published
2025-12-10 01:16:11 +01:00
jpmschweitzer 8638aad2d4 feat(library-desk): add entity linking endpoint
- Add /entity-linking/link-page endpoint to find and link entity mentions
- Creates both MENTIONS relationships in Neo4j and hyperlinks in wiki content
- Supports automatic re-indexing after linking
- Returns detailed statistics on entities found and linked
- Protects existing markdown links from being nested
- Idempotent: safe to run multiple times

Implements dual entity linking:
1. Graph relationships (MENTIONS) for knowledge graph traversal
2. Wiki content hyperlinks for user navigation
2025-12-10 01:15:38 +01:00
jpmschweitzerandClaude Opus 4.5 5263523fcd fix(library-desk): update Qdrant client to use query_points API
The Qdrant client API changed from search() to query_points().
Updated both search() and find_similar_chunks() methods.

All integration tests now passing: 14/14 ✓

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-08 19:13:33 +01:00
jpmschweitzerandClaude Opus 4.5 1a41e5bb80 feat(library-desk): implement Phase 1 service clients and infrastructure
Implements comprehensive service client layer for Library Desk API to support
Librarian AI agent with multi-tenant knowledge management across Neo4j, Qdrant,
Wiki.js, SearXNG, and Ollama.

## Service Clients (src/clients/)
- Neo4j async client with connection pooling and user-scoped labels
- Qdrant vector store with collection-per-user multi-tenancy
- Wiki.js GraphQL API client for page/dossier management
- SearXNG client for web search integration
- Ollama client for text embeddings (nomic-embed-text)

## Core Infrastructure (src/core/)
- Multi-tenancy helpers for user namespace management
  - Wiki.js: path-based namespaces (/users/{user})
  - Neo4j: user-specific labels (User_{User}_Document)
  - Qdrant: collection per user (library_desk_{user})
- Dependency injection with FastAPI Depends and @lru_cache singletons
- Lifecycle management (startup/shutdown) for all service connections

## Background Jobs (src/jobs/)
- Redis-based job manager for long-running operations
- Job status tracking with 24-hour TTL
- Support for queued, processing, completed, failed states

## Configuration
- Updated config.py with Redis DB 4 for library-desk jobs
- Updated docker-compose.yml: REDIS_DB from 2 to 4
- Added pytest and pytest-asyncio to requirements.txt

## Testing
- Unit tests: 25/25 passed (multi-tenancy helpers)
- Integration tests: 12/12 passed (all services verified)
  - Neo4j connection and CRUD operations
  - Qdrant vector operations with 768-dim embeddings
  - Wiki.js GraphQL queries
  - SearXNG web search
  - Job Manager with Redis
  - Dependency injection lifecycle
- pytest.ini configuration with asyncio support

## Health Monitoring
- Real-time service health checks via /health endpoint
- Connection status for all 5 external services
- Graceful degradation for partial service availability

## Architecture
- Follows async/await pattern throughout
- Connection pooling for Neo4j (singleton driver)
- HTTP client lifecycle management (httpx)
- Multi-tenancy enforced at client layer
- Default user: jpmschweitzer

Files changed: 26 files
- 5 new service clients (~1500 lines)
- 2 core modules (~500 lines)
- 1 job manager (~350 lines)
- 3 test files with 37 test cases
- Updated main.py with lifecycle hooks

All services tested and operational. Ready for Phase 2 (routers/services).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-08 19:09:54 +01:00
jpmschweitzerandClaude Opus 4.5 e886a2f9ba feat(library): deploy Library infrastructure (Neo4j, Wiki.js, Library Desk API)
Implements The Library system - a knowledge management and HybridRAG platform.

**Stack Files:**
- neo4j.yml: Knowledge graph database with APOC plugin
- wiki.yml: Wiki.js for human-facing dossier management
- library-desk.yml: FastAPI coordination service

**Library Desk Service:**
- FastAPI application following best practices
- Pydantic Settings for configuration management
- Bearer token authentication
- Health monitoring endpoints
- Stub endpoints for future HybridRAG implementation

**Features:**
- All services on docker-dataplane network
- Proper healthchecks for all containers
- Neo4j password validation (alphanumeric only)
- Wiki.js healthcheck fixed for IPv4/IPv6 compatibility
- Python 3.12+ with CVE-checked dependencies
- Minor version locking for stability

**Endpoints:**
- Neo4j Browser: http://192.168.86.149:7474
- Wiki.js: http://192.168.86.149:8088
- Library Desk API: http://192.168.86.149:8089
- API Docs: http://192.168.86.149:8089/docs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-08 17:35:09 +01:00