# Knowledge Consolidation (Librarian Task)

Automated processing of HybridRAG search results to consolidate knowledge into wiki pages.

## Overview

The **Knowledge Consolidation** system is a Librarian task that:
1. Processes unprocessed `SearchQuery` nodes from Neo4j (created by HybridRAG searches)
2. Analyzes web results with Ollama (mistral-nemo) for novel information
3. Creates new wiki pages for new concepts/technologies
4. Updates existing pages with new facts and citations
5. Adds new entities to the knowledge graph
6. Marks `SearchQuery` nodes as processed

**Architecture:**
- **Library Desk**: Exposes `/consolidate/knowledge` REST endpoint
- **Scheduler**: Calls endpoint periodically via generic `rest_api_executor`
- **Separation of concerns**: Scheduler triggers, Library Desk executes

---

## Endpoint: POST /consolidate/knowledge

**URL:** `http://library-desk:8089/consolidate/knowledge`

**Authentication:** Bearer token (LIBRARY_DESK_API_KEY)

**Request Body:**
```json
{
    "process_limit": 10,        // Max searches to process per run (1-100)
    "lookback_days": 7,          // Only process searches from last N days (1-90)
    "min_web_results": 2,        // Minimum web results required (1-20)
    "dry_run": false             // If true, analyze but don't create pages
}
```

**Response:**
```json
{
    "total_found": 5,
    "processed_count": 4,
    "pages_created": 2,
    "pages_updated": 3,
    "entities_added": 7,
    "errors": ["Search abc123: Failed to parse response"],
    "results": [
        {
            "search_id": "uuid-1",
            "query": "docker orchestration kubernetes",
            "pages_created": 1,
            "pages_updated": 1,
            "entities_added": 3,
            "error": null
        }
    ],
    "dry_run": false
}
```

---

## Manual Testing

### Test the endpoint directly:

```bash
# Set API key
export LIBRARY_DESK_API_KEY="your-api-key-here"

# Test with dry run (no changes)
curl -X POST http://192.168.86.149:8089/consolidate/knowledge \
  -H "Authorization: Bearer $LIBRARY_DESK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "process_limit": 5,
    "lookback_days": 7,
    "min_web_results": 2,
    "dry_run": true
  }' | python3 -m json.tool

# Real run (creates pages)
curl -X POST http://192.168.86.149:8089/consolidate/knowledge \
  -H "Authorization: Bearer $LIBRARY_DESK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "process_limit": 10,
    "lookback_days": 7,
    "min_web_results": 2,
    "dry_run": false
  }' | python3 -m json.tool
```

---

## Scheduler Integration

### Step 1: Set Environment Variable

The Scheduler needs the Library Desk API key to authenticate:

```bash
# Add to Scheduler's .env or docker-compose
LIBRARY_DESK_API_KEY=your-library-desk-api-key-here
```

### Step 2: Register Task via Scheduler API

Create the scheduled task using the Scheduler's REST API:

```bash
# Set Scheduler API key
export SCHEDULER_API_KEY="your-scheduler-api-key"

# Create knowledge consolidation task (runs hourly)
curl -X POST http://192.168.86.149:8090/tasks \
  -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task_name": "knowledge_consolidation",
    "service": "library-desk",
    "executor": "rest_api_executor",
    "priority": 50,
    "minute": 0,
    "hour": -1,
    "day_of_month": -1,
    "month": -1,
    "day_of_week": -1,
    "enabled": true,
    "description": "Hourly Librarian knowledge consolidation from HybridRAG search results",
    "config": {
        "url": "http://library-desk:8089/consolidate/knowledge",
        "method": "POST",
        "payload": {
            "process_limit": 10,
            "lookback_days": 7,
            "min_web_results": 2,
            "dry_run": false
        },
        "auth": {
            "type": "bearer",
            "token": "${LIBRARY_DESK_API_KEY}"
        },
        "timeout": 300,
        "response_path": "processed_count"
    }
}' | python3 -m json.tool
```

**Schedule Patterns:**

```bash
# Every hour at minute 0
"minute": 0, "hour": -1, ...

# Every 4 hours at minute 15
"minute": 15, "hour": [0, 4, 8, 12, 16, 20], ...

# Daily at 3:00 AM
"minute": 0, "hour": 3, "day_of_month": -1, ...

# Every Monday at 9:00 AM
"minute": 0, "hour": 9, "day_of_week": 0, ...
```

---

## Generic REST API Executor

The Scheduler's `rest_api_executor` is a universal executor for calling any REST API across the system.

### Features:
- **HTTP Methods**: GET, POST, PUT, DELETE, PATCH
- **Authentication**: Bearer token, Basic auth, API key
- **Environment Variables**: Use `${VAR_NAME}` for secrets
- **Configurable timeouts and SSL verification**
- **JSON payload support**
- **Response extraction via JSONPath**

### Example Configurations:

**1. Library Desk Knowledge Consolidation:**
```json
{
    "executor": "rest_api_executor",
    "config": {
        "url": "http://library-desk:8089/consolidate/knowledge",
        "method": "POST",
        "payload": {"process_limit": 10, "lookback_days": 7},
        "auth": {"type": "bearer", "token": "${LIBRARY_DESK_API_KEY}"}
    }
}
```

**2. Core API Container Restart:**
```json
{
    "executor": "rest_api_executor",
    "config": {
        "url": "http://core-api:8088/v1/infrastructure/containers/nginx/restart",
        "method": "POST",
        "auth": {"type": "bearer", "token": "${CORE_API_KEY}"}
    }
}
```

**3. External Webhook (Slack notification):**
```json
{
    "executor": "rest_api_executor",
    "config": {
        "url": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL",
        "method": "POST",
        "payload": {"text": "Daily backup completed"},
        "verify_ssl": true
    }
}
```

**4. Basic Auth Example:**
```json
{
    "executor": "rest_api_executor",
    "config": {
        "url": "http://internal-service:8080/api/sync",
        "method": "GET",
        "auth": {
            "type": "basic",
            "username": "${SERVICE_USER}",
            "password": "${SERVICE_PASSWORD}"
        }
    }
}
```

---

## Managing the Task

### View Task Status:
```bash
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  http://192.168.86.149:8090/tasks/knowledge_consolidation | python3 -m json.tool
```

### View Execution History:
```bash
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  "http://192.168.86.149:8090/executions?task_name=knowledge_consolidation&limit=10" \
  | python3 -m json.tool
```

### Manually Trigger Task:
```bash
curl -X POST \
  -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  http://192.168.86.149:8090/tasks/knowledge_consolidation/trigger
```

### Update Task Configuration:
```bash
curl -X PUT \
  -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": false,
    "config": {
        "payload": {
            "process_limit": 20,
            "dry_run": true
        }
    }
  }' \
  http://192.168.86.149:8090/tasks/knowledge_consolidation
```

### Delete Task:
```bash
curl -X DELETE \
  -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  http://192.168.86.149:8090/tasks/knowledge_consolidation
```

---

## How It Works

### 1. HybridRAG Search Creates SearchQuery Nodes

When a user performs a HybridRAG search (e.g., "docker orchestration kubernetes"):
- Library Desk stores search in Neo4j as `SearchQuery` node
- Web results stored as `WebResult` nodes with `FOUND` relationships
- `processed: false` flag indicates it needs consolidation

### 2. Scheduler Triggers Consolidation (Hourly)

The Scheduler runs the task at configured intervals:
- Calls Library Desk `/consolidate/knowledge` endpoint via HTTP
- Passes configuration (process_limit, lookback_days, etc.)
- Authenticates with Bearer token from environment

### 3. Library Desk Processes Searches

For each unprocessed `SearchQuery`:
1. **Query Neo4j** for searches with `processed: false` from last N days
2. **Retrieve web results** via `FOUND` relationships
3. **Analyze with Ollama** (mistral-nemo) for novel information:
   - Extract new concepts/technologies worth documenting
   - Identify facts to add to existing pages
   - Find new entities for knowledge graph
4. **Create/update wiki pages** (TODO: implement wiki API)
5. **Update knowledge graph** with new entities
6. **Mark SearchQuery as processed** (`processed: true, processed_at: timestamp`)

### 4. Response Returned to Scheduler

Library Desk returns summary:
- Total searches found and processed
- Pages created/updated
- Entities added
- Errors encountered

Scheduler logs the result and marks execution as complete.

---

## Future Enhancements

### 1. Wiki Page Creation
Currently logs what would be created. Need to implement:
- Wiki.js API integration for page creation
- Template-based page generation
- Citation formatting from web sources

### 2. Enhanced Entity Extraction
- Use spaCy or similar NLP for entity recognition
- Link entities between documents
- Build entity relationship graph

### 3. Smart Consolidation
- Detect duplicate/similar pages
- Merge related content
- Suggest tag improvements

### 4. User Feedback Loop
- Flag low-confidence consolidations for review
- Allow users to approve/reject suggestions
- Learn from user feedback

### 5. Metrics and Monitoring
- Track consolidation success rates
- Measure wiki growth over time
- Identify knowledge gaps

---

## Troubleshooting

### Task not running?

1. **Check task is enabled:**
   ```bash
   curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
     http://192.168.86.149:8090/tasks/knowledge_consolidation
   ```

2. **Check environment variable:**
   ```bash
   docker exec scheduler env | grep LIBRARY_DESK_API_KEY
   ```

3. **Check Scheduler logs:**
   ```bash
   docker logs scheduler --tail 50
   ```

### Task failing?

1. **Check execution history:**
   ```bash
   curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
     "http://192.168.86.149:8090/executions?task_name=knowledge_consolidation&limit=5"
   ```

2. **Test endpoint directly:**
   ```bash
   curl -X POST http://192.168.86.149:8089/consolidate/knowledge \
     -H "Authorization: Bearer $LIBRARY_DESK_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"process_limit": 1, "dry_run": true}'
   ```

3. **Check Library Desk logs:**
   ```bash
   docker logs library-desk --tail 100 | grep -i consolidation
   ```

### No searches being processed?

Searches may already be processed or too old:
- Increase `lookback_days` to process older searches
- Lower `min_web_results` threshold
- Check Neo4j for unprocessed searches:
  ```cypher
  MATCH (sq:SearchQuery {processed: false})
  RETURN sq.query, sq.timestamp, sq.web_count
  ORDER BY sq.timestamp DESC
  LIMIT 10
  ```

---

## Example Workflow

**1. User searches:** "What is Docker Swarm orchestration?"

**2. HybridRAG creates SearchQuery:**
```cypher
CREATE (sq:SearchQuery {
  id: "uuid-123",
  query: "What is Docker Swarm orchestration?",
  user: "jpmschweitzer",
  timestamp: datetime(),
  processed: false,
  web_count: 5
})
// + WebResult nodes for each web search result
```

**3. Scheduler triggers consolidation (hourly):**
```
[2025-01-09 14:00:00] Scheduler: Running knowledge_consolidation task
[2025-01-09 14:00:01] REST API Executor: POST http://library-desk:8089/consolidate/knowledge
```

**4. Library Desk processes:**
```
[2025-01-09 14:00:02] Found 3 unprocessed searches
[2025-01-09 14:00:03] Processing: "What is Docker Swarm orchestration?"
[2025-01-09 14:00:04] Retrieved 5 web results
[2025-01-09 14:00:08] LLM analysis: Novel info found
[2025-01-09 14:00:08] Would create page: "Docker Swarm Orchestration"
[2025-01-09 14:00:08] Marked search uuid-123 as processed
[2025-01-09 14:00:10] Consolidation complete: 3/3 searches, 2 pages created
```

**5. Scheduler logs result:**
```
[2025-01-09 14:00:10] Task knowledge_consolidation succeeded: Processed 3/3 searches. Created 2 pages, updated 1 pages, added 5 entities.
```

---

## Summary

The Knowledge Consolidation system provides:
- ✅ **Automated knowledge capture** from HybridRAG searches
- ✅ **Separation of concerns** via REST API
- ✅ **Generic REST executor** reusable across system
- ✅ **Flexible scheduling** via Scheduler
- ✅ **Comprehensive logging** and error handling
- ✅ **Dry run mode** for testing
- ⏳ **Wiki integration** (TODO)
- ⏳ **Advanced entity extraction** (TODO)

For more information:
- Scheduler: `/services/scheduler/README.md`
- Library Desk: `/services/library-desk/README.md`
- HybridRAG: `/services/library-desk/docs/HYBRID_RAG.md`