- Add WikiChangeListener service for PostgreSQL notifications
- Add webhooks router for HTTP webhook fallback
- Add asyncpg dependency for PostgreSQL async support
- Include setup scripts and documentation for triggers
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
search_pages with empty query uses stale search index. list_all_pages
fetches actual page list from GraphQL for reliable bulk re-indexing.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Set d.tags = \$tags in MERGE query to prevent Neo4j warnings about
missing tags property in related documents query.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Pass ingestion_service to ConsolidationService constructor so pages
created/updated during consolidation are properly indexed in Qdrant
and Neo4j.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fetch existing wiki taxonomy structure before LLM analysis
- Include existing paths in prompt to prefer existing categories
- Add _format_taxonomy_for_prompt helper
- Mark searches as processed even when skipped/errored (prevents buildup)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- list_all_pages: fetch all pages with path prefix filter
- get_taxonomy_structure: extract category/subcategory structure for
taxonomy-aware classification
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Use pages.singleByPath(path, locale) GraphQL query to get page ID
during initialization instead of searching through page list.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Fix create_entity_mentions to correctly count only newly created relationships,
not all relationships processed by MERGE.
**Problem**: count(r) was returning ALL relationships touched by MERGE (both
created and matched), breaking idempotency tests.
**Solution**: Use temporary flag 'just_created' set only ON CREATE, filter to
those relationships, count them, then remove the flag. This ensures the count
only includes new relationships.
Now properly returns:
- N on first run (N new relationships created)
- 0 on subsequent runs (no new relationships, all already exist)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Two critical bug fixes for entity linking:
1. **Path cleaning**: Replace hardcoded "users/jpmschweitzer/" with regex pattern
to handle any user namespace. Now properly cleans paths for all users.
2. **Longest-first matching**: Move protected_ranges computation inside entity loop
to recompute after each entity is processed. Prevents nested links like
[[Machine](/machine) Learning](/ml) when processing multi-word entities.
These fixes ensure entity linking works correctly across all users and prevents
nested markdown links when entity names overlap.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Cleanup Scripts:
- cleanup_graph.py - Clean up duplicate entities and orphaned nodes
- cleanup_wiki.py - Remove orphaned pages and fix broken links
- Utility scripts for database maintenance
- Not part of main application, run manually
Ingestion Router:
- POST /ingest/page - Index single page
- POST /ingest/batch - Batch indexing
- POST /ingest/all - Full knowledge base refresh
- Support vector and graph ingestion
Vector Router:
- POST /vector/search - Semantic search via Qdrant
- GET /vector/stats - Collection statistics
- DELETE /vector/page - Remove page embeddings
Wiki Router:
- GET /wiki/pages - List wiki pages
- GET /wiki/pages/{id} - Get page details
- PUT /wiki/pages/{id} - Update page
- POST /wiki/search - Search wiki content
- Full Wiki.js GraphQL integration
Tools Router:
- Expose library-desk capabilities as AI tool endpoints
- Support function calling for LLM agents
- Query wiki pages, search knowledge base
- Access graph entities and relationships
Tools Models:
- ToolDefinition for function schemas
- ToolParameter specifications
- ToolResponse format
- OpenAI function calling compatible
Graph Router:
- GET /graph/entities - List all entities for user
- GET /graph/relationships - Query entity relationships
- GET /graph/search - Search entities by name/type
- GET /graph/stats - Knowledge graph statistics
Graph Models:
- Entity, Relationship models
- GraphStats for analytics
- SearchFilters for queries
- Support multi-tenancy with user isolation
Configuration:
- Add HybridRAG settings (reranker model, search limits)
- Change Wiki.js auth from API key to username/password
- Configure vector, graph, and web search limits
Multi-tenancy:
- Add get_neo4j_user_base_label() for entity node labeling
- Support title-cased labels following Neo4j conventions
- Maintain namespace isolation for entities vs documents
Ollama Client:
- Improve model checking to handle :latest tag variants
- Match models with or without explicit tag
Qdrant Client:
- Add collection_exists() method for checking collection presence
- Refactor ensure_collection() to accept collection name directly
- Better separation of concerns
SearXNG Client:
- Add health_check() method for service monitoring
- Simple endpoint check without full search
- Used by health check endpoint
Vector Service:
- Manage document embeddings in Qdrant
- Update vectors from wiki pages
- Handle chunking and embedding generation
- Support force refresh and incremental updates
Vector Models:
- VectorSearchResult for search responses
- VectorUpdateSummary for indexing metrics
- Track chunks created/deleted
Used by ingestion_service for page embedding
- Add IngestionResult model for single page ingestion
- Add BatchIngestionResult for batch operations
- Track vector chunks, graph entities, and relationships
- Include processing time metrics
Used by ingestion_service for page indexing
- Add WikiPage model for page data
- Add WikiPageUpdate model for partial updates
- Add field validators for tags and descriptions
- Support optional fields for flexible updates
Used by wiki_service and entity_linking router
Main App:
- Mount /static directory for serving Wiki.js integration scripts
- Register entity_linking router
- Refactor API key verification to dependencies module
Dependencies:
- Add service factory functions for all services
- Add get_wiki_service() for wiki operations
- Add get_graph_service() for entity operations
- Add get_ingestion_service() for auto entity linking
- Improve health check for SearXNG
- Add combined integration script with both re-index and entity linking
- Add standalone entity linking button
- Add standalone re-index button
- Auto-detect Library Desk URL from script tag
- Support both toolbar and floating button positions
- Show real-time status updates and notifications
- Auto-reload page after successful entity linking
Usage: Inject via Wiki.js Code Injection settings
<script src="http://IP:8089/static/wikijs-integration.js"></script>
Graph Service:
- Add get_all_entities() to retrieve entities with wiki page paths
- Add create_entity_mentions() for MENTIONS relationship creation
- Support entity-to-document linking via title matching
Ingestion Service:
- Add _link_existing_entities() for automatic entity linking
- Auto-link entities during page ingestion
- Support skip_entity_linking parameter for granular control
- Add is_published parameter to WikiJS client update_page() method
- Update wiki_service to always pass is_published=True
- Prevents pages from being unpublished during entity linking updates
- Important for internal wikis where all pages should remain published
- Add /entity-linking/link-page endpoint to find and link entity mentions
- Creates both MENTIONS relationships in Neo4j and hyperlinks in wiki content
- Supports automatic re-indexing after linking
- Returns detailed statistics on entities found and linked
- Protects existing markdown links from being nested
- Idempotent: safe to run multiple times
Implements dual entity linking:
1. Graph relationships (MENTIONS) for knowledge graph traversal
2. Wiki content hyperlinks for user navigation
Add rest_api_executor as a universal executor that can call any REST API
endpoint across the system. This provides a standard way to trigger HTTP
operations from scheduled tasks.
Features:
- All HTTP methods: GET, POST, PUT, DELETE, PATCH
- Authentication: Bearer token, Basic auth, API key
- Environment variable substitution: ${VAR_NAME}
- JSONPath response extraction
- Configurable timeouts and SSL verification
- Sensitive data redaction in logs
- Custom headers support
This executor enables scheduler to call any service endpoint (Library Desk,
Core API, external webhooks) without needing service-specific executors.
Example usage:
{
"executor": "rest_api_executor",
"config": {
"url": "http://library-desk:8089/consolidate/knowledge",
"method": "POST",
"payload": {"process_limit": 10},
"auth": {"type": "bearer", "token": "${API_KEY}"}
}
}
The Qdrant client API changed from search() to query_points().
Updated both search() and find_similar_chunks() methods.
All integration tests now passing: 14/14 ✓
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implements comprehensive service client layer for Library Desk API to support
Librarian AI agent with multi-tenant knowledge management across Neo4j, Qdrant,
Wiki.js, SearXNG, and Ollama.
## Service Clients (src/clients/)
- Neo4j async client with connection pooling and user-scoped labels
- Qdrant vector store with collection-per-user multi-tenancy
- Wiki.js GraphQL API client for page/dossier management
- SearXNG client for web search integration
- Ollama client for text embeddings (nomic-embed-text)
## Core Infrastructure (src/core/)
- Multi-tenancy helpers for user namespace management
- Wiki.js: path-based namespaces (/users/{user})
- Neo4j: user-specific labels (User_{User}_Document)
- Qdrant: collection per user (library_desk_{user})
- Dependency injection with FastAPI Depends and @lru_cache singletons
- Lifecycle management (startup/shutdown) for all service connections
## Background Jobs (src/jobs/)
- Redis-based job manager for long-running operations
- Job status tracking with 24-hour TTL
- Support for queued, processing, completed, failed states
## Configuration
- Updated config.py with Redis DB 4 for library-desk jobs
- Updated docker-compose.yml: REDIS_DB from 2 to 4
- Added pytest and pytest-asyncio to requirements.txt
## Testing
- Unit tests: 25/25 passed (multi-tenancy helpers)
- Integration tests: 12/12 passed (all services verified)
- Neo4j connection and CRUD operations
- Qdrant vector operations with 768-dim embeddings
- Wiki.js GraphQL queries
- SearXNG web search
- Job Manager with Redis
- Dependency injection lifecycle
- pytest.ini configuration with asyncio support
## Health Monitoring
- Real-time service health checks via /health endpoint
- Connection status for all 5 external services
- Graceful degradation for partial service availability
## Architecture
- Follows async/await pattern throughout
- Connection pooling for Neo4j (singleton driver)
- HTTP client lifecycle management (httpx)
- Multi-tenancy enforced at client layer
- Default user: jpmschweitzer
Files changed: 26 files
- 5 new service clients (~1500 lines)
- 2 core modules (~500 lines)
- 1 job manager (~350 lines)
- 3 test files with 37 test cases
- Updated main.py with lifecycle hooks
All services tested and operational. Ready for Phase 2 (routers/services).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add complete documentation for The Scheduler service.
README.md (500+ lines):
- Architecture overview with ASCII diagram
- Quick start guide
- Complete API reference with curl examples
- Task scheduling patterns and examples
- Priority system documentation
- Built-in executors documentation (example, doc_sync, config_backup)
- Custom executor development guide
- Current tasks table
- Database schema documentation
- Testing guide with coverage metrics
- Development and debugging information
- Monitoring and troubleshooting
- Security and performance notes
- API reference with response codes and filtering
CHANGELOG.md:
- Initial v1.0.0 release documentation
- Core features and architecture
- REST API endpoints
- Task executors and pre-configured tasks
- Testing infrastructure and metrics
- Technical details and dependencies
- Coverage metrics breakdown
- Planned features for future releases
Documentation covers:
- All API endpoints and authentication
- Scheduling examples (every minute, daily, monthly, etc.)
- Priority ranges and usage
- Executor configuration
- Test database setup
- Docker stack configuration
- Common issues and solutions
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add hybrid APScheduler + PostgreSQL-based task scheduling system with minute-based execution and priority queue.
Core features:
- Minute-based scheduling with cron-like patterns (-1 = wildcard)
- Priority queue system (1-100, lower = higher priority)
- Concurrent execution (max 5 tasks simultaneously)
- Full REST API for task management (CRUD operations)
- Task execution tracking with audit trail
- API key authentication (Bearer token)
- Health checks and system statistics
Architecture:
- APScheduler runs every minute
- Queries PostgreSQL for tasks scheduled for current minute
- Executes tasks concurrently by priority
- Records execution history in database
Database schema:
- scheduled_tasks: Task definitions/templates
- task_executions: Individual execution records
Technical stack:
- FastAPI for REST API
- APScheduler for scheduling
- PostgreSQL for persistence
- Pydantic for configuration
Endpoints:
- POST/GET/PUT/DELETE /tasks - Task management
- POST /tasks/{name}/trigger - Manual execution
- GET /executions - Execution history
- GET /health - Health check
- GET /stats - System statistics
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Fixed: Change `result.data` to `result.output` (correct PydanticAI API)
- Increased analysis_timeout from 3s to 10s (mistral-nemo needs more time)
**Status:** Steward now initializes correctly but there's a remaining issue
with the async generator merging logic in two_stage_agent.py causing
requests to hang. This needs further investigation.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Removes the "simple" fallback model and renames "pydantic" to "Tatlock"
to match the agent's British butler persona.
Changes:
- /models endpoint now returns only "Tatlock" model
- Removed "simple" model from advertised models
- Updated default model name from "pydantic" to "Tatlock"
- Updated health endpoint to show "Tatlock" agent status
- Added description: "PydanticAI agent with full tool support - your British butler assistant"
Benefits:
- Clearer model naming that matches agent persona
- Simplified model selection in Open WebUI
- Eliminates confusion between pydantic/simple models
- Consistent branding with Tatlock character
Open WebUI will now show only "Tatlock" as an available model, which uses
the full PydanticAI agent with tool calling capabilities.
Tested:
✅ /models endpoint returns only Tatlock
✅ Health check shows Tatlock as default agent
✅ Chat completions work with model="Tatlock"
✅ Tatlock persona responds correctly
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
After PydanticAI migration (Dec 3), AI agent functionality was moved to
separate core-ai service. Health check was still trying to validate agent
in core-api, causing persistent unhealthy status (503 errors).
Changes:
- Remove ADK agent import attempts (no longer exists in core-api)
- Update /health/full to only check Ollama connectivity
- Update diagnostics endpoint with service separation notes
- Clarify that core-api is infrastructure/tools API only
Result: Container now reports healthy status consistently (200 OK).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add Scenario 6 to validate OpenAPI tool discovery and infrastructure integration.
Changes:
- Add test_scenario6_dns_lookup test
- Tests DNS lookup via core-api discovered tool
- Verifies OpenAPI discovery mechanism works
- Query: "What are the A records for github.com?"
- Performance target: < 15s
- Update test scenario list in run_full_quality_check()
- Document new scenario in QUALITY_TESTS.md
Purpose:
Validates that core-ai can discover and use infrastructure tools
from core-api via OpenAPI spec. DNS tool serves as example of
dynamic tool integration without manual registration.
Tool Discovery Chain:
1. core-api exposes /tools/dns/lookup endpoint (dnspython)
2. core-api publishes endpoint in /openapi.json
3. core-ai discovers tool via OpenAPI discovery
4. Agent can use tool as core-api__dns_lookup_tools_dns_lookup_post
Note: Agent currently prefers web_search for DNS queries, but
explicit instruction to use the DNS tool works. Tool naming
optimization can be addressed in future improvements.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implement automatic tool discovery from OpenAPI specifications, enabling
core-ai to dynamically use infrastructure management endpoints without
manual tool definitions.
Changes:
- Add OpenAPIToolDiscovery class for spec parsing and tool generation
- Fetches OpenAPI specs from configurable endpoints
- Generates executable tool functions from API operations
- Creates properly formatted tool schemas for agent use
- Async HTTP client for endpoint execution
- Update tool registry to support OpenAPI tools
- Optional include_openapi parameter in get_all_tools()
- Async loading of dynamic tools
- Merges local and OpenAPI tools seamlessly
- Add OpenAPI configuration settings
- openapi_endpoints: Comma-separated spec URLs
- openapi_enabled: Feature flag for tool discovery
- Default: http://core-api:8083/openapi.json
Architecture:
- Core tools (local.py): Always available essentials (web_search, calculate)
- OpenAPI tools: Infrastructure/automation from core-api dynamically discovered
Benefits:
- Auto-discovers new endpoints as core-api evolves
- No manual tool definition needed for REST APIs
- Maintains single source of truth (OpenAPI spec)
- Enables agent to manage infrastructure via discovered tools
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
jpmschweitzerandClaude Opus 4.5