The Scheduler

System-wide maintenance orchestration for automated task scheduling.

The Scheduler is a hybrid APScheduler + PostgreSQL-based task scheduling system that handles backups, documentation mirroring, cleanup, and automated maintenance tasks across the Portainer Core infrastructure.

Architecture

Hybrid Design: APScheduler + Database-driven Priority System

┌─────────────────────────────────────────────────────────┐
│  APScheduler (runs every minute)                        │
│  └─> Query PostgreSQL for tasks scheduled this minute   │
│      └─> Execute up to 5 tasks concurrently by priority │
└─────────────────────────────────────────────────────────┘

Key Features:

• Minute-based scheduling with cron-like patterns (-1 = wildcard)
• Priority queue (1-100, lower = higher priority)
• Concurrent execution (max 5 tasks simultaneously)
• Execution tracking (full audit trail in database)
• REST API for task management
• Flexible executors (modular task implementations)

Quick Start

Starting the Scheduler

# Deploy via Portainer
# Uses stack: /stacks/scheduler.yml

# Check health
curl http://localhost:8090/health

# View API docs
open http://localhost:8090/docs

API Authentication

All protected endpoints require an API key:

# Set in environment or use default
export SCHEDULER_API_KEY="your-api-key-here"

# Make authenticated request
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  http://localhost:8090/tasks

Scheduling Tasks

Task Configuration

Tasks are defined with:

• task_name: Unique identifier
• service: Which service owns this task
• executor: Python module to execute
• priority: 1-100 (1=emergency, 10-30=user, 40-70=maintenance)
• schedule: Minute, hour, day, month, day_of_week (-1 = any)
• config: JSON configuration for executor

Scheduling Examples

# Every minute (wildcard)
minute=-1, hour=-1, day_of_month=-1, month=-1, day_of_week=-1

# Daily at 3:05 AM
minute=5, hour=3, day_of_month=-1, month=-1, day_of_week=-1

# 11th of every month at 4:00 AM
minute=0, hour=4, day_of_month=11, month=-1, day_of_week=-1

# Every Monday at 9:00 AM
minute=0, hour=9, day_of_month=-1, month=-1, day_of_week=0

# First day of January at midnight
minute=0, hour=0, day_of_month=1, month=1, day_of_week=-1

Priority System

Priority Range  Purpose           Examples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1-5             Emergency/System  Critical system tasks
10-30           User Tasks        User-initiated operations
40-70           Maintenance       Backups, cleanup, doc sync
71-100          Low Priority      Optional background tasks

REST API

Task Management

# List all tasks
GET /tasks
GET /tasks?enabled=true&service=scheduler

# Get task details
GET /tasks/{task_name}

# Create new task
POST /tasks
{
  "task_name": "my_task",
  "service": "scheduler",
  "executor": "example_executor",
  "priority": 50,
  "minute": 0,
  "hour": 4,
  "description": "Daily task at 4 AM",
  "config": {"key": "value"}
}

# Update task
PUT /tasks/{task_name}
{
  "priority": 60,
  "enabled": false
}

# Delete task
DELETE /tasks/{task_name}

# Manually trigger task
POST /tasks/{task_name}/trigger

Execution History

# View recent executions
GET /executions?limit=20

# Filter by task
GET /executions?task_name=my_task&limit=10

# Filter by status
GET /executions?status=success

# Filter by service
GET /executions?service=scheduler

System Stats

# Get system statistics
GET /stats
{
  "scheduler_running": true,
  "tasks_enabled": 3,
  "tasks_currently_running": 0,
  "concurrent_limit": 5,
  "execution_stats_24h": {
    "success": 15,
    "failed": 1
  }
}

Task Executors

Executors are Python modules that implement the actual task logic.

Built-in Executors

1. Example Executor

File: src/executors/example_executor.py Purpose: Test/example implementation

config = {
    "message": "Hello World",
    "delay_seconds": 2
}

2. Doc Sync Executor

File: src/executors/doc_sync_executor.py Purpose: Mirror documentation from GitHub to Gitea

config = {
    "project": "fastapi",
    "upstream_repo": "https://github.com/tiangolo/fastapi.git",
    "docs_paths": ["/docs"],  # Empty = entire repo
    "gitea_repo": "library/docs-fastapi",
    "branch": "main"
}

Features:

• Clones upstream repository
• Filters to specific paths or mirrors entire repo
• Pushes to Gitea with authentication
• Creates date-tagged snapshots
• Generates .SYNC_INFO.md with metadata

3. Config Backup Executor

File: src/executors/config_backup_executor.py Purpose: Backup Docker configs and data

config = {
    "sources": [
        {
            "path": "/data/docker-data",
            "name": "docker-data",
            "excludes": ["*/cache/*", "*.log"]
        }
    ],
    "backup_dir": "/backups/configs",
    "compress": true,
    "retention_days": 30
}

Creating Custom Executors

1. Create file in src/executors/your_executor.py
2. Implement async execute(config: dict, settings: Settings) -> str function
3. Return success message or raise exception on failure

# src/executors/my_executor.py
async def execute(config: dict, settings: Settings) -> str:
    """
    Execute custom task.

    Args:
        config: Task configuration from database
        settings: Global scheduler settings

    Returns:
        Success message

    Raises:
        Exception: On task failure
    """
    # Your task logic here
    result = do_something(config.get('param'))

    return f"Task completed: {result}"

4. Create task using API:

curl -X POST http://localhost:8090/tasks \
  -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task_name": "my_custom_task",
    "service": "scheduler",
    "executor": "my_executor",
    "priority": 50,
    "minute": 0,
    "hour": 2,
    "config": {"param": "value"}
  }'

Current Tasks

Active Schedules

Task	Priority	Schedule	Description
test_example_task	50	Every minute	Test task (can be disabled)
backup_docker_configs_daily	20	Daily 03:05	Backup Docker configs
sync_fastapi_docs_monthly	60	11th @ 04:00	Sync FastAPI docs to Gitea
sync_ollama_docs_monthly	60	12th @ 04:00	Sync Ollama docs to Gitea

Managing Tasks

# Disable test task
curl -X PUT http://localhost:8090/tasks/test_example_task \
  -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"enabled": false}'

# View execution history
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  http://localhost:8090/executions?limit=10

Database Schema

scheduled_tasks

Stores task definitions (templates for execution).

- id (serial)
- task_name (varchar, unique)
- service (varchar) - owner service
- executor (varchar) - executor module name
- priority (integer) - 1-100
- minute, hour, day_of_month, month, day_of_week (integer) - schedule
- enabled (boolean)
- description (text)
- config (jsonb) - executor configuration
- last_run, last_status, last_duration_seconds - tracking
- retry_count, max_retries, timeout_seconds - execution control
- created_at, updated_at, created_by - metadata

task_executions

Stores individual execution records (audit trail).

- id (serial)
- task_id (integer) - references scheduled_tasks
- task_name, service, executor, priority - snapshot
- status (varchar) - pending, running, success, failed, timeout
- triggered_by (varchar) - scheduler, manual, retry
- triggered_at, started_at, completed_at - timestamps
- duration_seconds (integer)
- output (text) - success message
- error (text) - error details
- retry_count (integer)
- metadata (jsonb)

Testing

Running Tests

# All tests with coverage
docker exec -w /app scheduler /venv/bin/pytest --cov=src --cov-report=term

# Only unit tests (fast, no database)
docker exec -w /app scheduler /venv/bin/pytest -m unit

# Only integration tests (with database)
docker exec -w /app scheduler /venv/bin/pytest -m integration

# Only API tests
docker exec -w /app scheduler /venv/bin/pytest -m api

# Only executor tests
docker exec -w /app scheduler /venv/bin/pytest -m executor

# Specific test file
docker exec -w /app scheduler /venv/bin/pytest tests/test_api.py -v

# With detailed output
docker exec -w /app scheduler /venv/bin/pytest -vv --tb=long

Test Coverage

Current Coverage: 80%

Module                          Coverage
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
config.py                       100% ✅
example_executor.py             100% ✅
main.py (API endpoints)          95% ✅
doc_sync_executor.py             78% ✅
executor.py (core logic)         71% ✅
config_backup_executor.py        50% 📈
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
TOTAL                            80% 🎯

Test Database

Tests use a dedicated PostgreSQL database:

• Database: test_scheduler
• User: test_scheduler_user
• Location: postgres-shared container

Tests automatically:

• Clean database before each test
• Insert test fixtures
• Verify database state
• Rollback changes after test

Development

Project Structure

scheduler/
├── src/
│   ├── config.py              # Pydantic settings
│   ├── main.py                # FastAPI app & endpoints
│   ├── executors/             # Task executors
│   │   ├── example_executor.py
│   │   ├── doc_sync_executor.py
│   │   └── config_backup_executor.py
│   └── tasks/
│       └── executor.py        # Task execution engine
├── tests/
│   ├── conftest.py            # Pytest fixtures
│   ├── test_api.py            # API tests
│   ├── test_*_executor.py     # Executor tests
│   └── test_database_integration.py
├── requirements.txt           # Python dependencies
├── pytest.ini                 # Test configuration
└── README.md                  # This file

Adding Dependencies

# Add to requirements.txt with version pinning
echo "new-package~=1.0.0" >> requirements.txt

# Rebuild venv in container
docker restart scheduler

Debugging

# View logs
docker logs scheduler --tail 100 -f

# Check scheduler status
curl http://localhost:8090/health

# View database tasks
docker exec postgres-shared psql -U scheduler_user -d scheduler \
  -c "SELECT task_name, enabled, priority, last_status FROM scheduled_tasks;"

# View execution history
docker exec postgres-shared psql -U scheduler_user -d scheduler \
  -c "SELECT task_name, status, duration_seconds, completed_at FROM task_executions ORDER BY id DESC LIMIT 10;"

Environment Variables

Required environment variables (set in stacks/scheduler.yml):

# Database
POSTGRES_HOST=postgres-shared
POSTGRES_PORT=5432
POSTGRES_DB=scheduler
POSTGRES_USER=scheduler_user
POSTGRES_PASSWORD=${SCHEDULER_DB_PASSWORD}

# API Security
SCHEDULER_API_KEY=${SCHEDULER_API_KEY}

# Gitea (for doc sync)
GITEA_URL=http://gitea:3000
GITEA_USER=librarian
GITEA_PASSWORD=${GITEA_PASSWORD}

# Redis (future use)
REDIS_HOST=redis-shared
REDIS_PORT=6379
REDIS_DB=3

Monitoring

Health Checks

# Container health
docker ps | grep scheduler

# API health
curl http://localhost:8090/health

# Scheduler running
curl http://localhost:8090/stats | jq '.scheduler_running'

Metrics

# System statistics
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  http://localhost:8090/stats | jq

# Recent executions
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  "http://localhost:8090/executions?limit=20" | jq

# Failed tasks in last 24h
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  "http://localhost:8090/executions?status=failed" | jq

Common Issues

Task not running?

1. Check if task is enabled: GET /tasks/{name}
2. Verify schedule matches current time
3. Check execution history for errors: GET /executions?task_name={name}
4. View scheduler logs: docker logs scheduler

Task timing out?

• Increase timeout_seconds in task configuration
• Check executor implementation for long-running operations
• Consider breaking into smaller tasks

Database connection errors?

• Verify postgres-shared container is running
• Check database credentials in environment
• Test connection: docker exec scheduler psql -h postgres-shared -U scheduler_user -d scheduler

API Reference

Full OpenAPI documentation available at: http://localhost:8090/docs

Response Codes

• 200 - Success
• 400 - Bad request (invalid data)
• 401 - Missing API key
• 403 - Invalid API key
• 404 - Resource not found
• 500 - Internal server error

Pagination

Use limit parameter for controlling result count:

GET /executions?limit=50  # Default: 20, Max: 100

Filtering

Most list endpoints support filtering:

GET /tasks?enabled=true&service=scheduler
GET /executions?task_name=my_task&status=success&limit=10

Security

• API Key Authentication: Required for all protected endpoints
• Network Isolation: Runs on docker-dataplane network
• Database Credentials: Stored in environment variables
• Gitea Tokens: Used instead of passwords for Git operations
• Resource Limits: CPU and memory limits in docker-compose

Performance

• Concurrent Execution: Max 5 tasks simultaneously
• Minute-based Processing: Lightweight scheduler tick every minute
• Priority Queue: Higher priority tasks execute first
• Database Indexes: Optimized queries for task selection
• Connection Pooling: Reuses database connections

License

Part of Portainer Core infrastructure.

Support

For issues or questions:

1. Check logs: docker logs scheduler
2. View health: curl http://localhost:8090/health
3. Review execution history: GET /executions
4. Check this README for common solutions