Files

T

jpmschweitzer 1f0a894291 docs(library-desk): add architecture and API documentation

Documentation:
- Architecture overview and design decisions
- API endpoint reference
- Service interaction diagrams
- Configuration guide
- Multi-tenancy patterns
- Knowledge graph schema

2025-12-10 01:40:23 +01:00

12 KiB

Raw Blame History

Knowledge Consolidation (Librarian Task)

Automated processing of HybridRAG search results to consolidate knowledge into wiki pages.

Overview

The Knowledge Consolidation system is a Librarian task that:

Processes unprocessed SearchQuery nodes from Neo4j (created by HybridRAG searches)
Analyzes web results with Ollama (mistral-nemo) for novel information
Creates new wiki pages for new concepts/technologies
Updates existing pages with new facts and citations
Adds new entities to the knowledge graph
Marks SearchQuery nodes as processed

Architecture:

Library Desk: Exposes /consolidate/knowledge REST endpoint
Scheduler: Calls endpoint periodically via generic rest_api_executor
Separation of concerns: Scheduler triggers, Library Desk executes

Endpoint: POST /consolidate/knowledge

URL: http://library-desk:8089/consolidate/knowledge

Authentication: Bearer token (LIBRARY_DESK_API_KEY)

Request Body:

{
    "process_limit": 10,        // Max searches to process per run (1-100)
    "lookback_days": 7,          // Only process searches from last N days (1-90)
    "min_web_results": 2,        // Minimum web results required (1-20)
    "dry_run": false             // If true, analyze but don't create pages
}

Response:

{
    "total_found": 5,
    "processed_count": 4,
    "pages_created": 2,
    "pages_updated": 3,
    "entities_added": 7,
    "errors": ["Search abc123: Failed to parse response"],
    "results": [
        {
            "search_id": "uuid-1",
            "query": "docker orchestration kubernetes",
            "pages_created": 1,
            "pages_updated": 1,
            "entities_added": 3,
            "error": null
        }
    ],
    "dry_run": false
}

Manual Testing

Test the endpoint directly:

# Set API key
export LIBRARY_DESK_API_KEY="your-api-key-here"

# Test with dry run (no changes)
curl -X POST http://192.168.86.149:8089/consolidate/knowledge \
  -H "Authorization: Bearer $LIBRARY_DESK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "process_limit": 5,
    "lookback_days": 7,
    "min_web_results": 2,
    "dry_run": true
  }' | python3 -m json.tool

# Real run (creates pages)
curl -X POST http://192.168.86.149:8089/consolidate/knowledge \
  -H "Authorization: Bearer $LIBRARY_DESK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "process_limit": 10,
    "lookback_days": 7,
    "min_web_results": 2,
    "dry_run": false
  }' | python3 -m json.tool

Scheduler Integration

Step 1: Set Environment Variable

The Scheduler needs the Library Desk API key to authenticate:

# Add to Scheduler's .env or docker-compose
LIBRARY_DESK_API_KEY=your-library-desk-api-key-here

Step 2: Register Task via Scheduler API

Create the scheduled task using the Scheduler's REST API:

# Set Scheduler API key
export SCHEDULER_API_KEY="your-scheduler-api-key"

# Create knowledge consolidation task (runs hourly)
curl -X POST http://192.168.86.149:8090/tasks \
  -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task_name": "knowledge_consolidation",
    "service": "library-desk",
    "executor": "rest_api_executor",
    "priority": 50,
    "minute": 0,
    "hour": -1,
    "day_of_month": -1,
    "month": -1,
    "day_of_week": -1,
    "enabled": true,
    "description": "Hourly Librarian knowledge consolidation from HybridRAG search results",
    "config": {
        "url": "http://library-desk:8089/consolidate/knowledge",
        "method": "POST",
        "payload": {
            "process_limit": 10,
            "lookback_days": 7,
            "min_web_results": 2,
            "dry_run": false
        },
        "auth": {
            "type": "bearer",
            "token": "${LIBRARY_DESK_API_KEY}"
        },
        "timeout": 300,
        "response_path": "processed_count"
    }
}' | python3 -m json.tool

Schedule Patterns:

# Every hour at minute 0
"minute": 0, "hour": -1, ...

# Every 4 hours at minute 15
"minute": 15, "hour": [0, 4, 8, 12, 16, 20], ...

# Daily at 3:00 AM
"minute": 0, "hour": 3, "day_of_month": -1, ...

# Every Monday at 9:00 AM
"minute": 0, "hour": 9, "day_of_week": 0, ...

Generic REST API Executor

The Scheduler's rest_api_executor is a universal executor for calling any REST API across the system.

Features:

HTTP Methods: GET, POST, PUT, DELETE, PATCH
Authentication: Bearer token, Basic auth, API key
Environment Variables: Use ${VAR_NAME} for secrets
Configurable timeouts and SSL verification
JSON payload support
Response extraction via JSONPath

Example Configurations:

1. Library Desk Knowledge Consolidation:

{
    "executor": "rest_api_executor",
    "config": {
        "url": "http://library-desk:8089/consolidate/knowledge",
        "method": "POST",
        "payload": {"process_limit": 10, "lookback_days": 7},
        "auth": {"type": "bearer", "token": "${LIBRARY_DESK_API_KEY}"}
    }
}

2. Core API Container Restart:

{
    "executor": "rest_api_executor",
    "config": {
        "url": "http://core-api:8088/v1/infrastructure/containers/nginx/restart",
        "method": "POST",
        "auth": {"type": "bearer", "token": "${CORE_API_KEY}"}
    }
}

3. External Webhook (Slack notification):

{
    "executor": "rest_api_executor",
    "config": {
        "url": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL",
        "method": "POST",
        "payload": {"text": "Daily backup completed"},
        "verify_ssl": true
    }
}

4. Basic Auth Example:

{
    "executor": "rest_api_executor",
    "config": {
        "url": "http://internal-service:8080/api/sync",
        "method": "GET",
        "auth": {
            "type": "basic",
            "username": "${SERVICE_USER}",
            "password": "${SERVICE_PASSWORD}"
        }
    }
}

Managing the Task

View Task Status:

curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  http://192.168.86.149:8090/tasks/knowledge_consolidation | python3 -m json.tool

View Execution History:

curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  "http://192.168.86.149:8090/executions?task_name=knowledge_consolidation&limit=10" \
  | python3 -m json.tool

Manually Trigger Task:

curl -X POST \
  -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  http://192.168.86.149:8090/tasks/knowledge_consolidation/trigger

Update Task Configuration:

curl -X PUT \
  -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": false,
    "config": {
        "payload": {
            "process_limit": 20,
            "dry_run": true
        }
    }
  }' \
  http://192.168.86.149:8090/tasks/knowledge_consolidation

Delete Task:

curl -X DELETE \
  -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  http://192.168.86.149:8090/tasks/knowledge_consolidation

How It Works

1. HybridRAG Search Creates SearchQuery Nodes

When a user performs a HybridRAG search (e.g., "docker orchestration kubernetes"):

Library Desk stores search in Neo4j as SearchQuery node
Web results stored as WebResult nodes with FOUND relationships
processed: false flag indicates it needs consolidation

2. Scheduler Triggers Consolidation (Hourly)

The Scheduler runs the task at configured intervals:

Calls Library Desk /consolidate/knowledge endpoint via HTTP
Passes configuration (process_limit, lookback_days, etc.)
Authenticates with Bearer token from environment

3. Library Desk Processes Searches

For each unprocessed SearchQuery:

Query Neo4j for searches with processed: false from last N days
Retrieve web results via FOUND relationships
Analyze with Ollama (mistral-nemo) for novel information:
- Extract new concepts/technologies worth documenting
- Identify facts to add to existing pages
- Find new entities for knowledge graph
Create/update wiki pages (TODO: implement wiki API)
Update knowledge graph with new entities
Mark SearchQuery as processed (processed: true, processed_at: timestamp)

4. Response Returned to Scheduler

Library Desk returns summary:

Total searches found and processed
Pages created/updated
Entities added
Errors encountered

Scheduler logs the result and marks execution as complete.

Future Enhancements

1. Wiki Page Creation

Currently logs what would be created. Need to implement:

Wiki.js API integration for page creation
Template-based page generation
Citation formatting from web sources

2. Enhanced Entity Extraction

Use spaCy or similar NLP for entity recognition
Link entities between documents
Build entity relationship graph

3. Smart Consolidation

Detect duplicate/similar pages
Merge related content
Suggest tag improvements

4. User Feedback Loop

Flag low-confidence consolidations for review
Allow users to approve/reject suggestions
Learn from user feedback

5. Metrics and Monitoring

Track consolidation success rates
Measure wiki growth over time
Identify knowledge gaps

Troubleshooting

Task not running?

Check task is enabled:

curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  http://192.168.86.149:8090/tasks/knowledge_consolidation

Check environment variable:

docker exec scheduler env | grep LIBRARY_DESK_API_KEY

Check Scheduler logs:
```
docker logs scheduler --tail 50
```

Task failing?

Check execution history:

curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  "http://192.168.86.149:8090/executions?task_name=knowledge_consolidation&limit=5"

Test endpoint directly:

curl -X POST http://192.168.86.149:8089/consolidate/knowledge \
  -H "Authorization: Bearer $LIBRARY_DESK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"process_limit": 1, "dry_run": true}'

Check Library Desk logs:

docker logs library-desk --tail 100 | grep -i consolidation

No searches being processed?

Searches may already be processed or too old:

Increase lookback_days to process older searches
Lower min_web_results threshold

Check Neo4j for unprocessed searches:

MATCH (sq:SearchQuery {processed: false})
RETURN sq.query, sq.timestamp, sq.web_count
ORDER BY sq.timestamp DESC
LIMIT 10

Example Workflow

1. User searches: "What is Docker Swarm orchestration?"

2. HybridRAG creates SearchQuery:

CREATE (sq:SearchQuery {
  id: "uuid-123",
  query: "What is Docker Swarm orchestration?",
  user: "jpmschweitzer",
  timestamp: datetime(),
  processed: false,
  web_count: 5
})
// + WebResult nodes for each web search result

3. Scheduler triggers consolidation (hourly):

[2025-01-09 14:00:00] Scheduler: Running knowledge_consolidation task
[2025-01-09 14:00:01] REST API Executor: POST http://library-desk:8089/consolidate/knowledge

4. Library Desk processes:

[2025-01-09 14:00:02] Found 3 unprocessed searches
[2025-01-09 14:00:03] Processing: "What is Docker Swarm orchestration?"
[2025-01-09 14:00:04] Retrieved 5 web results
[2025-01-09 14:00:08] LLM analysis: Novel info found
[2025-01-09 14:00:08] Would create page: "Docker Swarm Orchestration"
[2025-01-09 14:00:08] Marked search uuid-123 as processed
[2025-01-09 14:00:10] Consolidation complete: 3/3 searches, 2 pages created

5. Scheduler logs result:

[2025-01-09 14:00:10] Task knowledge_consolidation succeeded: Processed 3/3 searches. Created 2 pages, updated 1 pages, added 5 entities.

Summary

The Knowledge Consolidation system provides:

✅ Automated knowledge capture from HybridRAG searches
✅ Separation of concerns via REST API
✅ Generic REST executor reusable across system
✅ Flexible scheduling via Scheduler
✅ Comprehensive logging and error handling
✅ Dry run mode for testing
⏳ Wiki integration (TODO)
⏳ Advanced entity extraction (TODO)

For more information:

Scheduler: /services/scheduler/README.md
Library Desk: /services/library-desk/README.md
HybridRAG: /services/library-desk/docs/HYBRID_RAG.md

12 KiB Raw Blame History