From 1f0a894291b4dedc4373faa92a3d84f5298f7db0 Mon Sep 17 00:00:00 2001 From: Jeroen Schweitzer Date: Wed, 10 Dec 2025 01:40:23 +0100 Subject: [PATCH] docs(library-desk): add architecture and API documentation Documentation: - Architecture overview and design decisions - API endpoint reference - Service interaction diagrams - Configuration guide - Multi-tenancy patterns - Knowledge graph schema --- .../docs/KNOWLEDGE_CONSOLIDATION.md | 469 ++++++++++++++++++ .../library-desk/docs/WIKI_PAGE_TEMPLATE.md | 107 ++++ 2 files changed, 576 insertions(+) create mode 100644 services/library-desk/docs/KNOWLEDGE_CONSOLIDATION.md create mode 100644 services/library-desk/docs/WIKI_PAGE_TEMPLATE.md diff --git a/services/library-desk/docs/KNOWLEDGE_CONSOLIDATION.md b/services/library-desk/docs/KNOWLEDGE_CONSOLIDATION.md new file mode 100644 index 0000000..aeb2437 --- /dev/null +++ b/services/library-desk/docs/KNOWLEDGE_CONSOLIDATION.md @@ -0,0 +1,469 @@ +# Knowledge Consolidation (Librarian Task) + +Automated processing of HybridRAG search results to consolidate knowledge into wiki pages. + +## Overview + +The **Knowledge Consolidation** system is a Librarian task that: +1. Processes unprocessed `SearchQuery` nodes from Neo4j (created by HybridRAG searches) +2. Analyzes web results with Ollama (mistral-nemo) for novel information +3. Creates new wiki pages for new concepts/technologies +4. Updates existing pages with new facts and citations +5. Adds new entities to the knowledge graph +6. Marks `SearchQuery` nodes as processed + +**Architecture:** +- **Library Desk**: Exposes `/consolidate/knowledge` REST endpoint +- **Scheduler**: Calls endpoint periodically via generic `rest_api_executor` +- **Separation of concerns**: Scheduler triggers, Library Desk executes + +--- + +## Endpoint: POST /consolidate/knowledge + +**URL:** `http://library-desk:8089/consolidate/knowledge` + +**Authentication:** Bearer token (LIBRARY_DESK_API_KEY) + +**Request Body:** +```json +{ + "process_limit": 10, // Max searches to process per run (1-100) + "lookback_days": 7, // Only process searches from last N days (1-90) + "min_web_results": 2, // Minimum web results required (1-20) + "dry_run": false // If true, analyze but don't create pages +} +``` + +**Response:** +```json +{ + "total_found": 5, + "processed_count": 4, + "pages_created": 2, + "pages_updated": 3, + "entities_added": 7, + "errors": ["Search abc123: Failed to parse response"], + "results": [ + { + "search_id": "uuid-1", + "query": "docker orchestration kubernetes", + "pages_created": 1, + "pages_updated": 1, + "entities_added": 3, + "error": null + } + ], + "dry_run": false +} +``` + +--- + +## Manual Testing + +### Test the endpoint directly: + +```bash +# Set API key +export LIBRARY_DESK_API_KEY="your-api-key-here" + +# Test with dry run (no changes) +curl -X POST http://192.168.86.149:8089/consolidate/knowledge \ + -H "Authorization: Bearer $LIBRARY_DESK_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "process_limit": 5, + "lookback_days": 7, + "min_web_results": 2, + "dry_run": true + }' | python3 -m json.tool + +# Real run (creates pages) +curl -X POST http://192.168.86.149:8089/consolidate/knowledge \ + -H "Authorization: Bearer $LIBRARY_DESK_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "process_limit": 10, + "lookback_days": 7, + "min_web_results": 2, + "dry_run": false + }' | python3 -m json.tool +``` + +--- + +## Scheduler Integration + +### Step 1: Set Environment Variable + +The Scheduler needs the Library Desk API key to authenticate: + +```bash +# Add to Scheduler's .env or docker-compose +LIBRARY_DESK_API_KEY=your-library-desk-api-key-here +``` + +### Step 2: Register Task via Scheduler API + +Create the scheduled task using the Scheduler's REST API: + +```bash +# Set Scheduler API key +export SCHEDULER_API_KEY="your-scheduler-api-key" + +# Create knowledge consolidation task (runs hourly) +curl -X POST http://192.168.86.149:8090/tasks \ + -H "Authorization: Bearer $SCHEDULER_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "task_name": "knowledge_consolidation", + "service": "library-desk", + "executor": "rest_api_executor", + "priority": 50, + "minute": 0, + "hour": -1, + "day_of_month": -1, + "month": -1, + "day_of_week": -1, + "enabled": true, + "description": "Hourly Librarian knowledge consolidation from HybridRAG search results", + "config": { + "url": "http://library-desk:8089/consolidate/knowledge", + "method": "POST", + "payload": { + "process_limit": 10, + "lookback_days": 7, + "min_web_results": 2, + "dry_run": false + }, + "auth": { + "type": "bearer", + "token": "${LIBRARY_DESK_API_KEY}" + }, + "timeout": 300, + "response_path": "processed_count" + } +}' | python3 -m json.tool +``` + +**Schedule Patterns:** + +```bash +# Every hour at minute 0 +"minute": 0, "hour": -1, ... + +# Every 4 hours at minute 15 +"minute": 15, "hour": [0, 4, 8, 12, 16, 20], ... + +# Daily at 3:00 AM +"minute": 0, "hour": 3, "day_of_month": -1, ... + +# Every Monday at 9:00 AM +"minute": 0, "hour": 9, "day_of_week": 0, ... +``` + +--- + +## Generic REST API Executor + +The Scheduler's `rest_api_executor` is a universal executor for calling any REST API across the system. + +### Features: +- **HTTP Methods**: GET, POST, PUT, DELETE, PATCH +- **Authentication**: Bearer token, Basic auth, API key +- **Environment Variables**: Use `${VAR_NAME}` for secrets +- **Configurable timeouts and SSL verification** +- **JSON payload support** +- **Response extraction via JSONPath** + +### Example Configurations: + +**1. Library Desk Knowledge Consolidation:** +```json +{ + "executor": "rest_api_executor", + "config": { + "url": "http://library-desk:8089/consolidate/knowledge", + "method": "POST", + "payload": {"process_limit": 10, "lookback_days": 7}, + "auth": {"type": "bearer", "token": "${LIBRARY_DESK_API_KEY}"} + } +} +``` + +**2. Core API Container Restart:** +```json +{ + "executor": "rest_api_executor", + "config": { + "url": "http://core-api:8088/v1/infrastructure/containers/nginx/restart", + "method": "POST", + "auth": {"type": "bearer", "token": "${CORE_API_KEY}"} + } +} +``` + +**3. External Webhook (Slack notification):** +```json +{ + "executor": "rest_api_executor", + "config": { + "url": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL", + "method": "POST", + "payload": {"text": "Daily backup completed"}, + "verify_ssl": true + } +} +``` + +**4. Basic Auth Example:** +```json +{ + "executor": "rest_api_executor", + "config": { + "url": "http://internal-service:8080/api/sync", + "method": "GET", + "auth": { + "type": "basic", + "username": "${SERVICE_USER}", + "password": "${SERVICE_PASSWORD}" + } + } +} +``` + +--- + +## Managing the Task + +### View Task Status: +```bash +curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \ + http://192.168.86.149:8090/tasks/knowledge_consolidation | python3 -m json.tool +``` + +### View Execution History: +```bash +curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \ + "http://192.168.86.149:8090/executions?task_name=knowledge_consolidation&limit=10" \ + | python3 -m json.tool +``` + +### Manually Trigger Task: +```bash +curl -X POST \ + -H "Authorization: Bearer $SCHEDULER_API_KEY" \ + http://192.168.86.149:8090/tasks/knowledge_consolidation/trigger +``` + +### Update Task Configuration: +```bash +curl -X PUT \ + -H "Authorization: Bearer $SCHEDULER_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "enabled": false, + "config": { + "payload": { + "process_limit": 20, + "dry_run": true + } + } + }' \ + http://192.168.86.149:8090/tasks/knowledge_consolidation +``` + +### Delete Task: +```bash +curl -X DELETE \ + -H "Authorization: Bearer $SCHEDULER_API_KEY" \ + http://192.168.86.149:8090/tasks/knowledge_consolidation +``` + +--- + +## How It Works + +### 1. HybridRAG Search Creates SearchQuery Nodes + +When a user performs a HybridRAG search (e.g., "docker orchestration kubernetes"): +- Library Desk stores search in Neo4j as `SearchQuery` node +- Web results stored as `WebResult` nodes with `FOUND` relationships +- `processed: false` flag indicates it needs consolidation + +### 2. Scheduler Triggers Consolidation (Hourly) + +The Scheduler runs the task at configured intervals: +- Calls Library Desk `/consolidate/knowledge` endpoint via HTTP +- Passes configuration (process_limit, lookback_days, etc.) +- Authenticates with Bearer token from environment + +### 3. Library Desk Processes Searches + +For each unprocessed `SearchQuery`: +1. **Query Neo4j** for searches with `processed: false` from last N days +2. **Retrieve web results** via `FOUND` relationships +3. **Analyze with Ollama** (mistral-nemo) for novel information: + - Extract new concepts/technologies worth documenting + - Identify facts to add to existing pages + - Find new entities for knowledge graph +4. **Create/update wiki pages** (TODO: implement wiki API) +5. **Update knowledge graph** with new entities +6. **Mark SearchQuery as processed** (`processed: true, processed_at: timestamp`) + +### 4. Response Returned to Scheduler + +Library Desk returns summary: +- Total searches found and processed +- Pages created/updated +- Entities added +- Errors encountered + +Scheduler logs the result and marks execution as complete. + +--- + +## Future Enhancements + +### 1. Wiki Page Creation +Currently logs what would be created. Need to implement: +- Wiki.js API integration for page creation +- Template-based page generation +- Citation formatting from web sources + +### 2. Enhanced Entity Extraction +- Use spaCy or similar NLP for entity recognition +- Link entities between documents +- Build entity relationship graph + +### 3. Smart Consolidation +- Detect duplicate/similar pages +- Merge related content +- Suggest tag improvements + +### 4. User Feedback Loop +- Flag low-confidence consolidations for review +- Allow users to approve/reject suggestions +- Learn from user feedback + +### 5. Metrics and Monitoring +- Track consolidation success rates +- Measure wiki growth over time +- Identify knowledge gaps + +--- + +## Troubleshooting + +### Task not running? + +1. **Check task is enabled:** + ```bash + curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \ + http://192.168.86.149:8090/tasks/knowledge_consolidation + ``` + +2. **Check environment variable:** + ```bash + docker exec scheduler env | grep LIBRARY_DESK_API_KEY + ``` + +3. **Check Scheduler logs:** + ```bash + docker logs scheduler --tail 50 + ``` + +### Task failing? + +1. **Check execution history:** + ```bash + curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \ + "http://192.168.86.149:8090/executions?task_name=knowledge_consolidation&limit=5" + ``` + +2. **Test endpoint directly:** + ```bash + curl -X POST http://192.168.86.149:8089/consolidate/knowledge \ + -H "Authorization: Bearer $LIBRARY_DESK_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"process_limit": 1, "dry_run": true}' + ``` + +3. **Check Library Desk logs:** + ```bash + docker logs library-desk --tail 100 | grep -i consolidation + ``` + +### No searches being processed? + +Searches may already be processed or too old: +- Increase `lookback_days` to process older searches +- Lower `min_web_results` threshold +- Check Neo4j for unprocessed searches: + ```cypher + MATCH (sq:SearchQuery {processed: false}) + RETURN sq.query, sq.timestamp, sq.web_count + ORDER BY sq.timestamp DESC + LIMIT 10 + ``` + +--- + +## Example Workflow + +**1. User searches:** "What is Docker Swarm orchestration?" + +**2. HybridRAG creates SearchQuery:** +```cypher +CREATE (sq:SearchQuery { + id: "uuid-123", + query: "What is Docker Swarm orchestration?", + user: "jpmschweitzer", + timestamp: datetime(), + processed: false, + web_count: 5 +}) +// + WebResult nodes for each web search result +``` + +**3. Scheduler triggers consolidation (hourly):** +``` +[2025-01-09 14:00:00] Scheduler: Running knowledge_consolidation task +[2025-01-09 14:00:01] REST API Executor: POST http://library-desk:8089/consolidate/knowledge +``` + +**4. Library Desk processes:** +``` +[2025-01-09 14:00:02] Found 3 unprocessed searches +[2025-01-09 14:00:03] Processing: "What is Docker Swarm orchestration?" +[2025-01-09 14:00:04] Retrieved 5 web results +[2025-01-09 14:00:08] LLM analysis: Novel info found +[2025-01-09 14:00:08] Would create page: "Docker Swarm Orchestration" +[2025-01-09 14:00:08] Marked search uuid-123 as processed +[2025-01-09 14:00:10] Consolidation complete: 3/3 searches, 2 pages created +``` + +**5. Scheduler logs result:** +``` +[2025-01-09 14:00:10] Task knowledge_consolidation succeeded: Processed 3/3 searches. Created 2 pages, updated 1 pages, added 5 entities. +``` + +--- + +## Summary + +The Knowledge Consolidation system provides: +- ✅ **Automated knowledge capture** from HybridRAG searches +- ✅ **Separation of concerns** via REST API +- ✅ **Generic REST executor** reusable across system +- ✅ **Flexible scheduling** via Scheduler +- ✅ **Comprehensive logging** and error handling +- ✅ **Dry run mode** for testing +- ⏳ **Wiki integration** (TODO) +- ⏳ **Advanced entity extraction** (TODO) + +For more information: +- Scheduler: `/services/scheduler/README.md` +- Library Desk: `/services/library-desk/README.md` +- HybridRAG: `/services/library-desk/docs/HYBRID_RAG.md` diff --git a/services/library-desk/docs/WIKI_PAGE_TEMPLATE.md b/services/library-desk/docs/WIKI_PAGE_TEMPLATE.md new file mode 100644 index 0000000..7c0923d --- /dev/null +++ b/services/library-desk/docs/WIKI_PAGE_TEMPLATE.md @@ -0,0 +1,107 @@ +# [Topic Title] + +> **Last Updated:** YYYY-MM-DD | **Status:** Draft/Active/Archived +> **Tags:** #tag1 #tag2 #tag3 + +## Executive Summary + +A concise 2-3 sentence overview of the topic. This should capture the essence and primary purpose for both AI and human readers. + +**Key Facts:** +- Most important fact #1 +- Most important fact #2 +- Most important fact #3 + +--- + +## Overview + +Detailed introduction to the topic. Explain what it is, why it matters, and its context within the broader ecosystem. + +## Core Concepts + +### Concept 1: [Name] + +Explanation of the first core concept. + +### Concept 2: [Name] + +Explanation of the second core concept. + +## Technical Details + +### Architecture + +Description of how the system/concept is structured. + +``` +[Diagram or code block if applicable] +``` + +### Specifications + +| Property | Value | Notes | +|----------|-------|-------| +| Property 1 | Value 1 | Additional context | +| Property 2 | Value 2 | Additional context | + +## Use Cases + +### Primary Use Case + +Description and example. + +### Secondary Use Cases + +- Use case 1 +- Use case 2 + +## Best Practices + +1. **Practice 1**: Description +2. **Practice 2**: Description + +## Common Issues & Solutions + +| Issue | Solution | Reference | +|-------|----------|-----------| +| Problem description | How to resolve | [Link] | + +## Related Topics + +- [Related Topic 1](link) - Brief description +- [Related Topic 2](link) - Brief description + +## Changes & Updates + +### 2025-01-09 +- Updated section X with new information from [Source] +- Corrected fact Y (previously stated Z) + +--- + +## Sources + +1. [Source Title](URL) - Publication Name, Date +2. [Source Title](URL) - Publication Name, Date +3. [Source Title](URL) - Publication Name, Date + +## Knowledge Graph + +**Related Entities:** +- [Entity 1](graph-link) - Relationship description +- [Entity 2](graph-link) - Relationship description + +**Mentioned In:** +- [Document 1](wiki-link) - Context +- [Document 2](wiki-link) - Context + +## Mind Map + +View this topic in context: [Mind Map Link](mindmap-url) + +--- + +*This page follows the Library Desk Wiki Standard v1.0* +*Generated/Updated by: [Human/Librarian Agent]* +*Quality Score: [0-100] | Completeness: [0-100%]*