jpmschweitzer/portainer-core
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(library-desk): add architecture and API documentation

Browse Source 
Documentation:
- Architecture overview and design decisions
- API endpoint reference
- Service interaction diagrams
- Configuration guide
- Multi-tenancy patterns
- Knowledge graph schema
This commit is contained in:
jpmschweitzer
2025-12-10 01:40:23 +01:00
parent ae8edf6111
commit 1f0a894291
2 changed files with 576 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
services/library-desk/docs/KNOWLEDGE_CONSOLIDATION.md
+469
View File
@@ -0,0 +1,469 @@
# Knowledge Consolidation (Librarian Task)
Automated processing of HybridRAG search results to consolidate knowledge into wiki pages.
## Overview
The **Knowledge Consolidation** system is a Librarian task that:
1. Processes unprocessed `SearchQuery` nodes from Neo4j (created by HybridRAG searches)
2. Analyzes web results with Ollama (mistral-nemo) for novel information
3. Creates new wiki pages for new concepts/technologies
4. Updates existing pages with new facts and citations
5. Adds new entities to the knowledge graph
6. Marks `SearchQuery` nodes as processed
**Architecture:**
- **Library Desk**: Exposes `/consolidate/knowledge` REST endpoint
- **Scheduler**: Calls endpoint periodically via generic `rest_api_executor`
- **Separation of concerns**: Scheduler triggers, Library Desk executes
---
## Endpoint: POST /consolidate/knowledge
**URL:** `http://library-desk:8089/consolidate/knowledge`
**Authentication:** Bearer token (LIBRARY_DESK_API_KEY)
**Request Body:**
```json
{
"process_limit": 10, // Max searches to process per run (1-100)
"lookback_days": 7, // Only process searches from last N days (1-90)
"min_web_results": 2, // Minimum web results required (1-20)
"dry_run": false // If true, analyze but don't create pages
}
```
**Response:**
```json
{
"total_found": 5,
"processed_count": 4,
"pages_created": 2,
"pages_updated": 3,
"entities_added": 7,
"errors": ["Search abc123: Failed to parse response"],
"results": [
{
"search_id": "uuid-1",
"query": "docker orchestration kubernetes",
"pages_created": 1,
"pages_updated": 1,
"entities_added": 3,
"error": null
}
],
"dry_run": false
}
```
---
## Manual Testing
### Test the endpoint directly:
```bash
# Set API key
export LIBRARY_DESK_API_KEY="your-api-key-here"
# Test with dry run (no changes)
curl -X POST http://192.168.86.149:8089/consolidate/knowledge \
-H "Authorization: Bearer $LIBRARY_DESK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"process_limit": 5,
"lookback_days": 7,
"min_web_results": 2,
"dry_run": true
}' | python3 -m json.tool
# Real run (creates pages)
curl -X POST http://192.168.86.149:8089/consolidate/knowledge \
-H "Authorization: Bearer $LIBRARY_DESK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"process_limit": 10,
"lookback_days": 7,
"min_web_results": 2,
"dry_run": false
}' | python3 -m json.tool
```
---
## Scheduler Integration
### Step 1: Set Environment Variable
The Scheduler needs the Library Desk API key to authenticate:
```bash
# Add to Scheduler's .env or docker-compose
LIBRARY_DESK_API_KEY=your-library-desk-api-key-here
```
### Step 2: Register Task via Scheduler API
Create the scheduled task using the Scheduler's REST API:
```bash
# Set Scheduler API key
export SCHEDULER_API_KEY="your-scheduler-api-key"
# Create knowledge consolidation task (runs hourly)
curl -X POST http://192.168.86.149:8090/tasks \
-H "Authorization: Bearer $SCHEDULER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"task_name": "knowledge_consolidation",
"service": "library-desk",
"executor": "rest_api_executor",
"priority": 50,
"minute": 0,
"hour": -1,
"day_of_month": -1,
"month": -1,
"day_of_week": -1,
"enabled": true,
"description": "Hourly Librarian knowledge consolidation from HybridRAG search results",
"config": {
"url": "http://library-desk:8089/consolidate/knowledge",
"method": "POST",
"payload": {
"process_limit": 10,
"lookback_days": 7,
"min_web_results": 2,
"dry_run": false
},
"auth": {
"type": "bearer",
"token": "${LIBRARY_DESK_API_KEY}"
},
"timeout": 300,
"response_path": "processed_count"
}
}' | python3 -m json.tool
```
**Schedule Patterns:**
```bash
# Every hour at minute 0
"minute": 0, "hour": -1, ...
# Every 4 hours at minute 15
"minute": 15, "hour": [0, 4, 8, 12, 16, 20], ...
# Daily at 3:00 AM
"minute": 0, "hour": 3, "day_of_month": -1, ...
# Every Monday at 9:00 AM
"minute": 0, "hour": 9, "day_of_week": 0, ...
```
---
## Generic REST API Executor
The Scheduler's `rest_api_executor` is a universal executor for calling any REST API across the system.
### Features:
- **HTTP Methods**: GET, POST, PUT, DELETE, PATCH
- **Authentication**: Bearer token, Basic auth, API key
- **Environment Variables**: Use `${VAR_NAME}` for secrets
- **Configurable timeouts and SSL verification**
- **JSON payload support**
- **Response extraction via JSONPath**
### Example Configurations:
**1. Library Desk Knowledge Consolidation:**
```json
{
"executor": "rest_api_executor",
"config": {
"url": "http://library-desk:8089/consolidate/knowledge",
"method": "POST",
"payload": {"process_limit": 10, "lookback_days": 7},
"auth": {"type": "bearer", "token": "${LIBRARY_DESK_API_KEY}"}
}
}
```
**2. Core API Container Restart:**
```json
{
"executor": "rest_api_executor",
"config": {
"url": "http://core-api:8088/v1/infrastructure/containers/nginx/restart",
"method": "POST",
"auth": {"type": "bearer", "token": "${CORE_API_KEY}"}
}
}
```
**3. External Webhook (Slack notification):**
```json
{
"executor": "rest_api_executor",
"config": {
"url": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL",
"method": "POST",
"payload": {"text": "Daily backup completed"},
"verify_ssl": true
}
}
```
**4. Basic Auth Example:**
```json
{
"executor": "rest_api_executor",
"config": {
"url": "http://internal-service:8080/api/sync",
"method": "GET",
"auth": {
"type": "basic",
"username": "${SERVICE_USER}",
"password": "${SERVICE_PASSWORD}"
}
}
}
```
---
## Managing the Task
### View Task Status:
```bash
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
http://192.168.86.149:8090/tasks/knowledge_consolidation | python3 -m json.tool
```
### View Execution History:
```bash
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
"http://192.168.86.149:8090/executions?task_name=knowledge_consolidation&limit=10" \
| python3 -m json.tool
```
### Manually Trigger Task:
```bash
curl -X POST \
-H "Authorization: Bearer $SCHEDULER_API_KEY" \
http://192.168.86.149:8090/tasks/knowledge_consolidation/trigger
```
### Update Task Configuration:
```bash
curl -X PUT \
-H "Authorization: Bearer $SCHEDULER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"enabled": false,
"config": {
"payload": {
"process_limit": 20,
"dry_run": true
}
}
}' \
http://192.168.86.149:8090/tasks/knowledge_consolidation
```
### Delete Task:
```bash
curl -X DELETE \
-H "Authorization: Bearer $SCHEDULER_API_KEY" \
http://192.168.86.149:8090/tasks/knowledge_consolidation
```
---
## How It Works
### 1. HybridRAG Search Creates SearchQuery Nodes
When a user performs a HybridRAG search (e.g., "docker orchestration kubernetes"):
- Library Desk stores search in Neo4j as `SearchQuery` node
- Web results stored as `WebResult` nodes with `FOUND` relationships
- `processed: false` flag indicates it needs consolidation
### 2. Scheduler Triggers Consolidation (Hourly)
The Scheduler runs the task at configured intervals:
- Calls Library Desk `/consolidate/knowledge` endpoint via HTTP
- Passes configuration (process_limit, lookback_days, etc.)
- Authenticates with Bearer token from environment
### 3. Library Desk Processes Searches
For each unprocessed `SearchQuery`:
1. **Query Neo4j** for searches with `processed: false` from last N days
2. **Retrieve web results** via `FOUND` relationships
3. **Analyze with Ollama** (mistral-nemo) for novel information:
- Extract new concepts/technologies worth documenting
- Identify facts to add to existing pages
- Find new entities for knowledge graph
4. **Create/update wiki pages** (TODO: implement wiki API)
5. **Update knowledge graph** with new entities
6. **Mark SearchQuery as processed** (`processed: true, processed_at: timestamp`)
### 4. Response Returned to Scheduler
Library Desk returns summary:
- Total searches found and processed
- Pages created/updated
- Entities added
- Errors encountered
Scheduler logs the result and marks execution as complete.
---
## Future Enhancements
### 1. Wiki Page Creation
Currently logs what would be created. Need to implement:
- Wiki.js API integration for page creation
- Template-based page generation
- Citation formatting from web sources
### 2. Enhanced Entity Extraction
- Use spaCy or similar NLP for entity recognition
- Link entities between documents
- Build entity relationship graph
### 3. Smart Consolidation
- Detect duplicate/similar pages
- Merge related content
- Suggest tag improvements
### 4. User Feedback Loop
- Flag low-confidence consolidations for review
- Allow users to approve/reject suggestions
- Learn from user feedback
### 5. Metrics and Monitoring
- Track consolidation success rates
- Measure wiki growth over time
- Identify knowledge gaps
---
## Troubleshooting
### Task not running?
1. **Check task is enabled:**
```bash
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
http://192.168.86.149:8090/tasks/knowledge_consolidation
```
2. **Check environment variable:**
```bash
docker exec scheduler env | grep LIBRARY_DESK_API_KEY
```
3. **Check Scheduler logs:**
```bash
docker logs scheduler --tail 50
```
### Task failing?
1. **Check execution history:**
```bash
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
"http://192.168.86.149:8090/executions?task_name=knowledge_consolidation&limit=5"
```
2. **Test endpoint directly:**
```bash
curl -X POST http://192.168.86.149:8089/consolidate/knowledge \
-H "Authorization: Bearer $LIBRARY_DESK_API_KEY" \
-H "Content-Type: application/json" \
-d '{"process_limit": 1, "dry_run": true}'
```
3. **Check Library Desk logs:**
```bash
docker logs library-desk --tail 100 | grep -i consolidation
```
### No searches being processed?
Searches may already be processed or too old:
- Increase `lookback_days` to process older searches
- Lower `min_web_results` threshold
- Check Neo4j for unprocessed searches:
```cypher
MATCH (sq:SearchQuery {processed: false})
RETURN sq.query, sq.timestamp, sq.web_count
ORDER BY sq.timestamp DESC
LIMIT 10
```
---
## Example Workflow
**1. User searches:** "What is Docker Swarm orchestration?"
**2. HybridRAG creates SearchQuery:**
```cypher
CREATE (sq:SearchQuery {
id: "uuid-123",
query: "What is Docker Swarm orchestration?",
user: "jpmschweitzer",
timestamp: datetime(),
processed: false,
web_count: 5
})
// + WebResult nodes for each web search result
```
**3. Scheduler triggers consolidation (hourly):**
```
[2025-01-09 14:00:00] Scheduler: Running knowledge_consolidation task
[2025-01-09 14:00:01] REST API Executor: POST http://library-desk:8089/consolidate/knowledge
```
**4. Library Desk processes:**
```
[2025-01-09 14:00:02] Found 3 unprocessed searches
[2025-01-09 14:00:03] Processing: "What is Docker Swarm orchestration?"
[2025-01-09 14:00:04] Retrieved 5 web results
[2025-01-09 14:00:08] LLM analysis: Novel info found
[2025-01-09 14:00:08] Would create page: "Docker Swarm Orchestration"
[2025-01-09 14:00:08] Marked search uuid-123 as processed
[2025-01-09 14:00:10] Consolidation complete: 3/3 searches, 2 pages created
```
**5. Scheduler logs result:**
```
[2025-01-09 14:00:10] Task knowledge_consolidation succeeded: Processed 3/3 searches. Created 2 pages, updated 1 pages, added 5 entities.
```
---
## Summary
The Knowledge Consolidation system provides:
- ✅ **Automated knowledge capture** from HybridRAG searches
- ✅ **Separation of concerns** via REST API
- ✅ **Generic REST executor** reusable across system
- ✅ **Flexible scheduling** via Scheduler
- ✅ **Comprehensive logging** and error handling
- ✅ **Dry run mode** for testing
- ⏳ **Wiki integration** (TODO)
- ⏳ **Advanced entity extraction** (TODO)
For more information:
- Scheduler: `/services/scheduler/README.md`
- Library Desk: `/services/library-desk/README.md`
- HybridRAG: `/services/library-desk/docs/HYBRID_RAG.md`
services/library-desk/docs/WIKI_PAGE_TEMPLATE.md
+107
View File
@@ -0,0 +1,107 @@
# [Topic Title]
> **Last Updated:** YYYY-MM-DD | **Status:** Draft/Active/Archived
> **Tags:** #tag1 #tag2 #tag3
## Executive Summary
A concise 2-3 sentence overview of the topic. This should capture the essence and primary purpose for both AI and human readers.
**Key Facts:**
- Most important fact #1
- Most important fact #2
- Most important fact #3
---
## Overview
Detailed introduction to the topic. Explain what it is, why it matters, and its context within the broader ecosystem.
## Core Concepts
### Concept 1: [Name]
Explanation of the first core concept.
### Concept 2: [Name]
Explanation of the second core concept.
## Technical Details
### Architecture
Description of how the system/concept is structured.
```
[Diagram or code block if applicable]
```
### Specifications
| Property | Value | Notes |
|----------|-------|-------|
| Property 1 | Value 1 | Additional context |
| Property 2 | Value 2 | Additional context |
## Use Cases
### Primary Use Case
Description and example.
### Secondary Use Cases
- Use case 1
- Use case 2
## Best Practices
1. **Practice 1**: Description
2. **Practice 2**: Description
## Common Issues & Solutions
| Issue | Solution | Reference |
|-------|----------|-----------|
| Problem description | How to resolve | [Link] |
## Related Topics
- [Related Topic 1](link) - Brief description
- [Related Topic 2](link) - Brief description
## Changes & Updates
### 2025-01-09
- Updated section X with new information from [Source]
- Corrected fact Y (previously stated Z)
---
## Sources
1. [Source Title](URL) - Publication Name, Date
2. [Source Title](URL) - Publication Name, Date
3. [Source Title](URL) - Publication Name, Date
## Knowledge Graph
**Related Entities:**
- [Entity 1](graph-link) - Relationship description
- [Entity 2](graph-link) - Relationship description
**Mentioned In:**
- [Document 1](wiki-link) - Context
- [Document 2](wiki-link) - Context
## Mind Map
View this topic in context: [Mind Map Link](mindmap-url)
---
*This page follows the Library Desk Wiki Standard v1.0*
*Generated/Updated by: [Human/Librarian Agent]*
*Quality Score: [0-100] | Completeness: [0-100%]*