Initial commit: scheduler service extraction from portainer-core

Extracted standalone scheduler service with: - FastAPI REST API for task management - APScheduler-based task execution - PostgreSQL persistence - Docker container support - Gitea Actions CI/CD workflow 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-11 11:59:32 +01:00
commit 64574bcc39
31 changed files with 6284 additions and 0 deletions
@@ -0,0 +1,289 @@
+# Scheduler Task Registration Guide
+
+## Overview
+
+The Scheduler is **DB-driven** (not YAML-based). Tasks are registered via REST API and stored in PostgreSQL.
+
+## Quick Reference
+
+### Register a New Task
+
+```bash
+curl -X POST http://192.168.86.149:8090/tasks \
+  -H "Authorization: Bearer $SCHEDULER_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "task_name": "my_task",
+    "service": "my-service",
+    "executor": "rest_api",
+    "priority": 25,
+    "description": "Task description",
+    "minute": 0,
+    "hour": -1,
+    "config": {
+      "method": "POST",
+      "url": "http://service:port/endpoint",
+      "headers": {
+        "Authorization": "Bearer ${ENV_VAR}"
+      },
+      "body": {}
+    }
+  }'
+```
+
+### List All Tasks
+
+```bash
+curl http://192.168.86.149:8090/tasks \
+  -H "Authorization: Bearer $SCHEDULER_API_KEY"
+```
+
+### Trigger Task Manually
+
+```bash
+curl -X POST http://192.168.86.149:8090/tasks/{task_name}/trigger \
+  -H "Authorization: Bearer $SCHEDULER_API_KEY"
+```
+
+### View Task Execution History
+
+```bash
+curl http://192.168.86.149:8090/executions?task_name={task_name} \
+  -H "Authorization: Bearer $SCHEDULER_API_KEY"
+```
+
+## Schedule Patterns
+
+Use cron-style fields where **`-1` means "every"**:
+
+| Pattern | minute | hour | day_of_month | month | day_of_week | Description |
+|---------|--------|------|--------------|-------|-------------|-------------|
+| Every minute | -1 | -1 | -1 | -1 | -1 | Runs every minute |
+| Hourly | 0 | -1 | -1 | -1 | -1 | Runs at minute 0 of every hour |
+| Daily at 2 AM | 0 | 2 | -1 | -1 | -1 | Runs at 2:00 AM every day |
+| Weekly (Monday 2 AM) | 0 | 2 | -1 | -1 | 0 | Runs at 2:00 AM every Monday |
+| Monthly (1st at 2 AM) | 0 | 2 | 1 | -1 | -1 | Runs at 2:00 AM on 1st of month |
+
+**Note:** `day_of_week` is 0-6 (Monday-Sunday)
+
+## Priority Levels
+
+| Priority Range | Level | Use Cases |
+|----------------|-------|-----------|
+| 1-5 | Emergency/System | Critical system tasks |
+| 10-30 | User | User-initiated tasks |
+| 40-70 | Maintenance | Background maintenance |
+| 70+ | Low | Low priority cleanup |
+
+## Executor Types
+
+### `rest_api` Executor
+
+Calls HTTP endpoints. Supports environment variable substitution in headers/body.
+
+**Config Format:**
+```json
+{
+  "method": "POST|GET|PUT|DELETE|PATCH",
+  "url": "http://service:port/endpoint",
+  "headers": {
+    "Authorization": "Bearer ${ENV_VAR}",
+    "Content-Type": "application/json"
+  },
+  "body": {
+    "key": "value"
+  },
+  "timeout": 30,
+  "auth": {
+    "type": "bearer|basic|api_key",
+    "value": "${TOKEN_ENV_VAR}"
+  }
+}
+```
+
+**Environment Variables:** Use `${VAR_NAME}` syntax to reference env vars from the scheduler container.
+
+### Other Executors
+
+- `shell`: Execute shell commands
+- `python`: Execute Python scripts
+- `docker`: Docker operations
+- `backup`: Backup operations
+- `doc_sync`: Documentation sync
+
+## Complete Task Schema
+
+### Required Fields
+
+- `task_name` (string): Unique task identifier
+- `service` (string): Service that owns this task (e.g., "library-desk", "core-api")
+- `executor` (string): Executor type ("rest_api", "shell", etc.)
+- `priority` (int): Priority level (1-100)
+
+### Optional Fields
+
+- `description` (string): Human-readable description
+- `minute` (int): -1 to 59, default: -1 (every minute)
+- `hour` (int): -1 to 23, default: -1 (every hour)
+- `day_of_month` (int): -1 to 31, default: -1 (every day)
+- `month` (int): -1 to 12, default: -1 (every month)
+- `day_of_week` (int): -1 to 6, default: -1 (every day)
+- `enabled` (bool): Whether task is enabled, default: true
+- `max_retries` (int): Max retry attempts, default: 3
+- `timeout_seconds` (int): Execution timeout, default: 3600
+- `config` (object): Executor-specific configuration
+- `created_by` (string): Creator identifier, default: "api"
+
+## Example: Librarian Knowledge Consolidation
+
+This task processes unprocessed search queries and consolidates knowledge into wiki pages.
+
+**Registration:**
+```bash
+curl -X POST http://192.168.86.149:8090/tasks \
+  -H "Authorization: Bearer $SCHEDULER_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "task_name": "librarian_consolidation",
+    "service": "library-desk",
+    "executor": "rest_api",
+    "priority": 25,
+    "description": "Processes unprocessed search queries and consolidates knowledge",
+    "minute": 0,
+    "hour": -1,
+    "config": {
+      "method": "POST",
+      "url": "http://library-desk:8089/consolidation/run",
+      "headers": {
+        "Authorization": "Bearer ${LIBRARY_API_KEY}",
+        "Content-Type": "application/json"
+      },
+      "body": {
+        "dry_run": false,
+        "process_limit": 10,
+        "lookback_days": 7,
+        "min_web_results": 2
+      }
+    }
+  }'
+```
+
+**Schedule:** Runs hourly (at minute 0 of every hour)
+
+**What it does:**
+1. Queries Neo4j for unprocessed `SearchQuery` nodes
+2. Analyzes web results with Ollama
+3. Creates/updates wiki pages via WikiPageWriter
+4. Extracts and adds new entities to knowledge graph
+5. Marks searches as processed
+
+## API Endpoints
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| GET | `/tasks` | List all tasks |
+| GET | `/tasks/{task_name}` | Get task details |
+| POST | `/tasks` | Create new task |
+| PUT | `/tasks/{task_name}` | Update task |
+| DELETE | `/tasks/{task_name}` | Delete task |
+| POST | `/tasks/{task_name}/trigger` | Manually trigger task |
+| GET | `/executions` | View execution history |
+| GET | `/stats` | Get scheduler statistics |
+| GET | `/health` | Health check |
+
+## Interactive Documentation
+
+Visit **http://192.168.86.149:8090/docs** for interactive Swagger UI with:
+- Full API documentation
+- Request/response schemas
+- Try-it-out functionality
+- Example payloads
+
+## Environment Variables
+
+Get API keys from containers:
+```bash
+# Scheduler API key
+docker exec scheduler printenv SCHEDULER_API_KEY
+
+# Library Desk API key
+docker exec library-desk printenv LIBRARY_API_KEY
+```
+
+## Task Management Tips
+
+1. **Test with dry_run**: Set `enabled: false` initially or use dry_run in task config
+2. **Monitor executions**: Check `/executions` endpoint regularly
+3. **Start conservative**: Use longer intervals (hourly, daily) before going to frequent runs
+4. **Use priorities wisely**: Reserve 1-10 for critical tasks, use 20-30 for user tasks
+5. **Set timeouts appropriately**: Long-running tasks need higher timeout_seconds
+
+## Troubleshooting
+
+### Task not running
+
+1. Check if task is enabled: `GET /tasks/{task_name}`
+2. Check execution history: `GET /executions?task_name={task_name}`
+3. Verify schedule matches current time
+4. Check scheduler logs: `docker logs scheduler`
+
+### Task failing
+
+1. Check execution history for error messages
+2. Verify endpoint is accessible from scheduler container
+3. Test endpoint manually with curl
+4. Check API keys and authentication
+5. Verify executor configuration (especially for rest_api)
+
+### Debug a task
+
+```bash
+# Get task details
+curl http://192.168.86.149:8090/tasks/librarian_consolidation \
+  -H "Authorization: Bearer $SCHEDULER_API_KEY"
+
+# View recent executions
+curl http://192.168.86.149:8090/executions?task_name=librarian_consolidation&limit=5 \
+  -H "Authorization: Bearer $SCHEDULER_API_KEY"
+
+# Trigger manually
+curl -X POST http://192.168.86.149:8090/tasks/librarian_consolidation/trigger \
+  -H "Authorization: Bearer $SCHEDULER_API_KEY"
+```
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────┐
+│  The Scheduler (AsyncIOScheduler)                       │
+│                                                          │
+│  ┌────────────────────────────────────────────┐        │
+│  │  Cron Job (every minute)                   │        │
+│  │  ↓                                          │        │
+│  │  Query PostgreSQL for tasks scheduled      │        │
+│  │  for current minute/hour/day               │        │
+│  │  ↓                                          │        │
+│  │  Execute up to 5 tasks concurrently        │        │
+│  │  (based on priority)                       │        │
+│  │  ↓                                          │        │
+│  │  Call appropriate executor:                │        │
+│  │  - rest_api → HTTP request                │        │
+│  │  - shell → subprocess                      │        │
+│  │  - python → exec                           │        │
+│  └────────────────────────────────────────────┘        │
+│                                                          │
+│  PostgreSQL:                                            │
+│  - scheduled_tasks (task definitions)                  │
+│  - task_executions (execution history)                 │
+└─────────────────────────────────────────────────────────┘
+```
+
+## Current Registered Tasks
+
+As of 2025-12-09:
+
+1. **backup_docker_configs_daily** - Daily at 3:05 AM
+2. **librarian_consolidation** - Hourly (minute 0)
+3. **test_example_task** - Every minute (testing)
+4. **sync_fastapi_docs_monthly** - 11th of month at 4:00 AM
+5. **sync_ollama_docs_monthly** - 12th of month at 4:00 AM