scheduler/README.md

# The Scheduler

**System-wide maintenance orchestration for automated task scheduling.**

The Scheduler is a hybrid APScheduler + PostgreSQL-based task scheduling system that handles backups, documentation mirroring, cleanup, and automated maintenance tasks across the Portainer Core infrastructure.

## Architecture

**Hybrid Design**: APScheduler + Database-driven Priority System

```
┌─────────────────────────────────────────────────────────┐
│  APScheduler (runs every minute)                        │
│  └─> Query PostgreSQL for tasks scheduled this minute   │
│      └─> Execute up to 5 tasks concurrently by priority │
└─────────────────────────────────────────────────────────┘
```

**Key Features**:
- **Minute-based scheduling** with cron-like patterns (`-1` = wildcard)
- **Priority queue** (1-100, lower = higher priority)
- **Concurrent execution** (max 5 tasks simultaneously)
- **Execution tracking** (full audit trail in database)
- **REST API** for task management
- **Flexible executors** (modular task implementations)

## Quick Start

### Starting the Scheduler

```bash
# Deploy via Portainer
# Uses stack: /stacks/scheduler.yml

# Check health
curl http://localhost:8090/health

# View API docs
open http://localhost:8090/docs
```

### API Authentication

All protected endpoints require an API key:

```bash
# Set in environment or use default
export SCHEDULER_API_KEY="your-api-key-here"

# Make authenticated request
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  http://localhost:8090/tasks
```

## Scheduling Tasks

### Task Configuration

Tasks are defined with:
- **task_name**: Unique identifier
- **service**: Which service owns this task
- **executor**: Python module to execute
- **priority**: 1-100 (1=emergency, 10-30=user, 40-70=maintenance)
- **schedule**: Minute, hour, day, month, day_of_week (-1 = any)
- **config**: JSON configuration for executor

### Scheduling Examples

```python
# Every minute (wildcard)
minute=-1, hour=-1, day_of_month=-1, month=-1, day_of_week=-1

# Daily at 3:05 AM
minute=5, hour=3, day_of_month=-1, month=-1, day_of_week=-1

# 11th of every month at 4:00 AM
minute=0, hour=4, day_of_month=11, month=-1, day_of_week=-1

# Every Monday at 9:00 AM
minute=0, hour=9, day_of_month=-1, month=-1, day_of_week=0

# First day of January at midnight
minute=0, hour=0, day_of_month=1, month=1, day_of_week=-1
```

### Priority System

```
Priority Range  Purpose           Examples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1-5             Emergency/System  Critical system tasks
10-30           User Tasks        User-initiated operations
40-70           Maintenance       Backups, cleanup, doc sync
71-100          Low Priority      Optional background tasks
```

## REST API

### Task Management

```bash
# List all tasks
GET /tasks
GET /tasks?enabled=true&service=scheduler

# Get task details
GET /tasks/{task_name}

# Create new task
POST /tasks
{
  "task_name": "my_task",
  "service": "scheduler",
  "executor": "example_executor",
  "priority": 50,
  "minute": 0,
  "hour": 4,
  "description": "Daily task at 4 AM",
  "config": {"key": "value"}
}

# Update task
PUT /tasks/{task_name}
{
  "priority": 60,
  "enabled": false
}

# Delete task
DELETE /tasks/{task_name}

# Manually trigger task
POST /tasks/{task_name}/trigger
```

### Execution History

```bash
# View recent executions
GET /executions?limit=20

# Filter by task
GET /executions?task_name=my_task&limit=10

# Filter by status
GET /executions?status=success

# Filter by service
GET /executions?service=scheduler
```

### System Stats

```bash
# Get system statistics
GET /stats
{
  "scheduler_running": true,
  "tasks_enabled": 3,
  "tasks_currently_running": 0,
  "concurrent_limit": 5,
  "execution_stats_24h": {
    "success": 15,
    "failed": 1
  }
}
```

## Task Executors

Executors are Python modules that implement the actual task logic.

### Built-in Executors

#### 1. Example Executor
**File**: `src/executors/example_executor.py`
**Purpose**: Test/example implementation

```python
config = {
    "message": "Hello World",
    "delay_seconds": 2
}
```

#### 2. Doc Sync Executor
**File**: `src/executors/doc_sync_executor.py`
**Purpose**: Mirror documentation from GitHub to Gitea

```python
config = {
    "project": "fastapi",
    "upstream_repo": "https://github.com/tiangolo/fastapi.git",
    "docs_paths": ["/docs"],  # Empty = entire repo
    "gitea_repo": "library/docs-fastapi",
    "branch": "main"
}
```

**Features**:
- Clones upstream repository
- Filters to specific paths or mirrors entire repo
- Pushes to Gitea with authentication
- Creates date-tagged snapshots
- Generates `.SYNC_INFO.md` with metadata

#### 3. Config Backup Executor
**File**: `src/executors/config_backup_executor.py`
**Purpose**: Backup Docker configs and data

```python
config = {
    "sources": [
        {
            "path": "/data/docker-data",
            "name": "docker-data",
            "excludes": ["*/cache/*", "*.log"]
        }
    ],
    "backup_dir": "/backups/configs",
    "compress": true,
    "retention_days": 30
}
```

### Creating Custom Executors

1. Create file in `src/executors/your_executor.py`
2. Implement async `execute(config: dict, settings: Settings) -> str` function
3. Return success message or raise exception on failure

```python
# src/executors/my_executor.py
async def execute(config: dict, settings: Settings) -> str:
    """
    Execute custom task.

    Args:
        config: Task configuration from database
        settings: Global scheduler settings

    Returns:
        Success message

    Raises:
        Exception: On task failure
    """
    # Your task logic here
    result = do_something(config.get('param'))

    return f"Task completed: {result}"
```

4. Create task using API:

```bash
curl -X POST http://localhost:8090/tasks \
  -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task_name": "my_custom_task",
    "service": "scheduler",
    "executor": "my_executor",
    "priority": 50,
    "minute": 0,
    "hour": 2,
    "config": {"param": "value"}
  }'
```

## Current Tasks

### Active Schedules

| Task | Priority | Schedule | Description |
|------|----------|----------|-------------|
| `test_example_task` | 50 | Every minute | Test task (can be disabled) |
| `backup_docker_configs_daily` | 20 | Daily 03:05 | Backup Docker configs |
| `sync_fastapi_docs_monthly` | 60 | 11th @ 04:00 | Sync FastAPI docs to Gitea |
| `sync_ollama_docs_monthly` | 60 | 12th @ 04:00 | Sync Ollama docs to Gitea |

### Managing Tasks

```bash
# Disable test task
curl -X PUT http://localhost:8090/tasks/test_example_task \
  -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"enabled": false}'

# View execution history
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  http://localhost:8090/executions?limit=10
```

## Database Schema

### scheduled_tasks

Stores task definitions (templates for execution).

```sql
- id (serial)
- task_name (varchar, unique)
- service (varchar) - owner service
- executor (varchar) - executor module name
- priority (integer) - 1-100
- minute, hour, day_of_month, month, day_of_week (integer) - schedule
- enabled (boolean)
- description (text)
- config (jsonb) - executor configuration
- last_run, last_status, last_duration_seconds - tracking
- retry_count, max_retries, timeout_seconds - execution control
- created_at, updated_at, created_by - metadata
```

### task_executions

Stores individual execution records (audit trail).

```sql
- id (serial)
- task_id (integer) - references scheduled_tasks
- task_name, service, executor, priority - snapshot
- status (varchar) - pending, running, success, failed, timeout
- triggered_by (varchar) - scheduler, manual, retry
- triggered_at, started_at, completed_at - timestamps
- duration_seconds (integer)
- output (text) - success message
- error (text) - error details
- retry_count (integer)
- metadata (jsonb)
```

## Testing

### Running Tests

```bash
# All tests with coverage
docker exec -w /app scheduler /venv/bin/pytest --cov=src --cov-report=term

# Only unit tests (fast, no database)
docker exec -w /app scheduler /venv/bin/pytest -m unit

# Only integration tests (with database)
docker exec -w /app scheduler /venv/bin/pytest -m integration

# Only API tests
docker exec -w /app scheduler /venv/bin/pytest -m api

# Only executor tests
docker exec -w /app scheduler /venv/bin/pytest -m executor

# Specific test file
docker exec -w /app scheduler /venv/bin/pytest tests/test_api.py -v

# With detailed output
docker exec -w /app scheduler /venv/bin/pytest -vv --tb=long
```

### Test Coverage

**Current Coverage: 80%**

```
Module                          Coverage
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
config.py                       100% ✅
example_executor.py             100% ✅
main.py (API endpoints)          95% ✅
doc_sync_executor.py             78% ✅
executor.py (core logic)         71% ✅
config_backup_executor.py        50% 📈
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
TOTAL                            80% 🎯
```

### Test Database

Tests use a dedicated PostgreSQL database:
- **Database**: `test_scheduler`
- **User**: `test_scheduler_user`
- **Location**: `postgres-shared` container

Tests automatically:
- Clean database before each test
- Insert test fixtures
- Verify database state
- Rollback changes after test

## Development

### Project Structure

```
scheduler/
├── src/
│   ├── config.py              # Pydantic settings
│   ├── main.py                # FastAPI app & endpoints
│   ├── executors/             # Task executors
│   │   ├── example_executor.py
│   │   ├── doc_sync_executor.py
│   │   └── config_backup_executor.py
│   └── tasks/
│       └── executor.py        # Task execution engine
├── tests/
│   ├── conftest.py            # Pytest fixtures
│   ├── test_api.py            # API tests
│   ├── test_*_executor.py     # Executor tests
│   └── test_database_integration.py
├── requirements.txt           # Python dependencies
├── pytest.ini                 # Test configuration
└── README.md                  # This file
```

### Adding Dependencies

```bash
# Add to requirements.txt with version pinning
echo "new-package~=1.0.0" >> requirements.txt

# Rebuild venv in container
docker restart scheduler
```

### Debugging

```bash
# View logs
docker logs scheduler --tail 100 -f

# Check scheduler status
curl http://localhost:8090/health

# View database tasks
docker exec postgres-shared psql -U scheduler_user -d scheduler \
  -c "SELECT task_name, enabled, priority, last_status FROM scheduled_tasks;"

# View execution history
docker exec postgres-shared psql -U scheduler_user -d scheduler \
  -c "SELECT task_name, status, duration_seconds, completed_at FROM task_executions ORDER BY id DESC LIMIT 10;"
```

### Environment Variables

Required environment variables (set in `stacks/scheduler.yml`):

```bash
# Database
POSTGRES_HOST=postgres-shared
POSTGRES_PORT=5432
POSTGRES_DB=scheduler
POSTGRES_USER=scheduler_user
POSTGRES_PASSWORD=${SCHEDULER_DB_PASSWORD}

# API Security
SCHEDULER_API_KEY=${SCHEDULER_API_KEY}

# Gitea (for doc sync)
GITEA_URL=http://gitea:3000
GITEA_USER=librarian
GITEA_PASSWORD=${GITEA_PASSWORD}

# Redis (future use)
REDIS_HOST=redis-shared
REDIS_PORT=6379
REDIS_DB=3
```

## Monitoring

### Health Checks

```bash
# Container health
docker ps | grep scheduler

# API health
curl http://localhost:8090/health

# Scheduler running
curl http://localhost:8090/stats | jq '.scheduler_running'
```

### Metrics

```bash
# System statistics
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  http://localhost:8090/stats | jq

# Recent executions
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  "http://localhost:8090/executions?limit=20" | jq

# Failed tasks in last 24h
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
  "http://localhost:8090/executions?status=failed" | jq
```

### Common Issues

**Task not running?**
1. Check if task is enabled: `GET /tasks/{name}`
2. Verify schedule matches current time
3. Check execution history for errors: `GET /executions?task_name={name}`
4. View scheduler logs: `docker logs scheduler`

**Task timing out?**
- Increase `timeout_seconds` in task configuration
- Check executor implementation for long-running operations
- Consider breaking into smaller tasks

**Database connection errors?**
- Verify postgres-shared container is running
- Check database credentials in environment
- Test connection: `docker exec scheduler psql -h postgres-shared -U scheduler_user -d scheduler`

## API Reference

Full OpenAPI documentation available at: `http://localhost:8090/docs`

### Response Codes

- `200` - Success
- `400` - Bad request (invalid data)
- `401` - Missing API key
- `403` - Invalid API key
- `404` - Resource not found
- `500` - Internal server error

### Pagination

Use `limit` parameter for controlling result count:

```bash
GET /executions?limit=50  # Default: 20, Max: 100
```

### Filtering

Most list endpoints support filtering:

```bash
GET /tasks?enabled=true&service=scheduler
GET /executions?task_name=my_task&status=success&limit=10
```

## Security

- **API Key Authentication**: Required for all protected endpoints
- **Network Isolation**: Runs on `docker-dataplane` network
- **Database Credentials**: Stored in environment variables
- **Gitea Tokens**: Used instead of passwords for Git operations
- **Resource Limits**: CPU and memory limits in docker-compose

## Performance

- **Concurrent Execution**: Max 5 tasks simultaneously
- **Minute-based Processing**: Lightweight scheduler tick every minute
- **Priority Queue**: Higher priority tasks execute first
- **Database Indexes**: Optimized queries for task selection
- **Connection Pooling**: Reuses database connections

## License

Part of Portainer Core infrastructure.

## Support

For issues or questions:
1. Check logs: `docker logs scheduler`
2. View health: `curl http://localhost:8090/health`
3. Review execution history: `GET /executions`
4. Check this README for common solutions