scheduler/TASK_REGISTRATION.md

Scheduler Task Registration Guide

Overview

The Scheduler is DB-driven (not YAML-based). Tasks are registered via REST API and stored in PostgreSQL.

Quick Reference

Register a New Task

curl -X POST http://192.168.86.149:8090/tasks \
  -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task_name": "my_task",
    "service": "my-service",
    "executor": "rest_api",
    "priority": 25,
    "description": "Task description",
    "minute": 0,
    "hour": -1,
    "config": {
      "method": "POST",
      "url": "http://service:port/endpoint",
      "headers": {
        "Authorization": "Bearer ${ENV_VAR}"
      },
      "body": {}
    }
  }'

List All Tasks

curl http://192.168.86.149:8090/tasks \
  -H "Authorization: Bearer $SCHEDULER_API_KEY"

Trigger Task Manually

curl -X POST http://192.168.86.149:8090/tasks/{task_name}/trigger \
  -H "Authorization: Bearer $SCHEDULER_API_KEY"

View Task Execution History

curl http://192.168.86.149:8090/executions?task_name={task_name} \
  -H "Authorization: Bearer $SCHEDULER_API_KEY"

Schedule Patterns

Use cron-style fields where -1 means "every":

Pattern	minute	hour	day_of_month	month	day_of_week	Description
Every minute	-1	-1	-1	-1	-1	Runs every minute
Hourly	0	-1	-1	-1	-1	Runs at minute 0 of every hour
Daily at 2 AM	0	2	-1	-1	-1	Runs at 2:00 AM every day
Weekly (Monday 2 AM)	0	2	-1	-1	0	Runs at 2:00 AM every Monday
Monthly (1st at 2 AM)	0	2	1	-1	-1	Runs at 2:00 AM on 1st of month

Note: day_of_week is 0-6 (Monday-Sunday)

Priority Levels

Priority Range	Level	Use Cases
1-5	Emergency/System	Critical system tasks
10-30	User	User-initiated tasks
40-70	Maintenance	Background maintenance
70+	Low	Low priority cleanup

Executor Types

rest_api Executor

Calls HTTP endpoints. Supports environment variable substitution in headers/body.

Config Format:

{
  "method": "POST|GET|PUT|DELETE|PATCH",
  "url": "http://service:port/endpoint",
  "headers": {
    "Authorization": "Bearer ${ENV_VAR}",
    "Content-Type": "application/json"
  },
  "body": {
    "key": "value"
  },
  "timeout": 30,
  "auth": {
    "type": "bearer|basic|api_key",
    "value": "${TOKEN_ENV_VAR}"
  }
}

Environment Variables: Use ${VAR_NAME} syntax to reference env vars from the scheduler container.

Other Executors

• shell: Execute shell commands
• python: Execute Python scripts
• docker: Docker operations
• backup: Backup operations
• doc_sync: Documentation sync

Complete Task Schema

Required Fields

• task_name (string): Unique task identifier
• service (string): Service that owns this task (e.g., "library-desk", "core-api")
• executor (string): Executor type ("rest_api", "shell", etc.)
• priority (int): Priority level (1-100)

Optional Fields

• description (string): Human-readable description
• minute (int): -1 to 59, default: -1 (every minute)
• hour (int): -1 to 23, default: -1 (every hour)
• day_of_month (int): -1 to 31, default: -1 (every day)
• month (int): -1 to 12, default: -1 (every month)
• day_of_week (int): -1 to 6, default: -1 (every day)
• enabled (bool): Whether task is enabled, default: true
• max_retries (int): Max retry attempts, default: 3
• timeout_seconds (int): Execution timeout, default: 3600
• config (object): Executor-specific configuration
• created_by (string): Creator identifier, default: "api"

Example: Librarian Knowledge Consolidation

This task processes unprocessed search queries and consolidates knowledge into wiki pages.

Registration:

curl -X POST http://192.168.86.149:8090/tasks \
  -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task_name": "librarian_consolidation",
    "service": "library-desk",
    "executor": "rest_api",
    "priority": 25,
    "description": "Processes unprocessed search queries and consolidates knowledge",
    "minute": 0,
    "hour": -1,
    "config": {
      "method": "POST",
      "url": "http://library-desk:8089/consolidation/run",
      "headers": {
        "Authorization": "Bearer ${LIBRARY_API_KEY}",
        "Content-Type": "application/json"
      },
      "body": {
        "dry_run": false,
        "process_limit": 10,
        "lookback_days": 7,
        "min_web_results": 2
      }
    }
  }'

Schedule: Runs hourly (at minute 0 of every hour)

What it does:

1. Queries Neo4j for unprocessed SearchQuery nodes
2. Analyzes web results with Ollama
3. Creates/updates wiki pages via WikiPageWriter
4. Extracts and adds new entities to knowledge graph
5. Marks searches as processed

API Endpoints

Method	Endpoint	Description
GET	/tasks	List all tasks
GET	/tasks/{task_name}	Get task details
POST	/tasks	Create new task
PUT	/tasks/{task_name}	Update task
DELETE	/tasks/{task_name}	Delete task
POST	/tasks/{task_name}/trigger	Manually trigger task
GET	/executions	View execution history
GET	/stats	Get scheduler statistics
GET	/health	Health check

Interactive Documentation

Visit http://192.168.86.149:8090/docs for interactive Swagger UI with:

• Full API documentation
• Request/response schemas
• Try-it-out functionality
• Example payloads

Environment Variables

Get API keys from containers:

# Scheduler API key
docker exec scheduler printenv SCHEDULER_API_KEY

# Library Desk API key
docker exec library-desk printenv LIBRARY_API_KEY

Task Management Tips

1. Test with dry_run: Set enabled: false initially or use dry_run in task config
2. Monitor executions: Check /executions endpoint regularly
3. Start conservative: Use longer intervals (hourly, daily) before going to frequent runs
4. Use priorities wisely: Reserve 1-10 for critical tasks, use 20-30 for user tasks
5. Set timeouts appropriately: Long-running tasks need higher timeout_seconds

Troubleshooting

Task not running

1. Check if task is enabled: GET /tasks/{task_name}
2. Check execution history: GET /executions?task_name={task_name}
3. Verify schedule matches current time
4. Check scheduler logs: docker logs scheduler

Task failing

1. Check execution history for error messages
2. Verify endpoint is accessible from scheduler container
3. Test endpoint manually with curl
4. Check API keys and authentication
5. Verify executor configuration (especially for rest_api)

Debug a task

# Get task details
curl http://192.168.86.149:8090/tasks/librarian_consolidation \
  -H "Authorization: Bearer $SCHEDULER_API_KEY"

# View recent executions
curl http://192.168.86.149:8090/executions?task_name=librarian_consolidation&limit=5 \
  -H "Authorization: Bearer $SCHEDULER_API_KEY"

# Trigger manually
curl -X POST http://192.168.86.149:8090/tasks/librarian_consolidation/trigger \
  -H "Authorization: Bearer $SCHEDULER_API_KEY"

Architecture

┌─────────────────────────────────────────────────────────┐
│  The Scheduler (AsyncIOScheduler)                       │
│                                                          │
│  ┌────────────────────────────────────────────┐        │
│  │  Cron Job (every minute)                   │        │
│  │  ↓                                          │        │
│  │  Query PostgreSQL for tasks scheduled      │        │
│  │  for current minute/hour/day               │        │
│  │  ↓                                          │        │
│  │  Execute up to 5 tasks concurrently        │        │
│  │  (based on priority)                       │        │
│  │  ↓                                          │        │
│  │  Call appropriate executor:                │        │
│  │  - rest_api → HTTP request                │        │
│  │  - shell → subprocess                      │        │
│  │  - python → exec                           │        │
│  └────────────────────────────────────────────┘        │
│                                                          │
│  PostgreSQL:                                            │
│  - scheduled_tasks (task definitions)                  │
│  - task_executions (execution history)                 │
└─────────────────────────────────────────────────────────┘

Current Registered Tasks

As of 2025-12-09:

1. backup_docker_configs_daily - Daily at 3:05 AM
2. librarian_consolidation - Hourly (minute 0)
3. test_example_task - Every minute (testing)
4. sync_fastapi_docs_monthly - 11th of month at 4:00 AM
5. sync_ollama_docs_monthly - 12th of month at 4:00 AM