jpmschweitzer/portainer-core
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(scheduler): add comprehensive README and CHANGELOG

Browse Source 
Add complete documentation for The Scheduler service.

README.md (500+ lines):
- Architecture overview with ASCII diagram
- Quick start guide
- Complete API reference with curl examples
- Task scheduling patterns and examples
- Priority system documentation
- Built-in executors documentation (example, doc_sync, config_backup)
- Custom executor development guide
- Current tasks table
- Database schema documentation
- Testing guide with coverage metrics
- Development and debugging information
- Monitoring and troubleshooting
- Security and performance notes
- API reference with response codes and filtering

CHANGELOG.md:
- Initial v1.0.0 release documentation
- Core features and architecture
- REST API endpoints
- Task executors and pre-configured tasks
- Testing infrastructure and metrics
- Technical details and dependencies
- Coverage metrics breakdown
- Planned features for future releases

Documentation covers:
- All API endpoints and authentication
- Scheduling examples (every minute, daily, monthly, etc.)
- Priority ranges and usage
- Executor configuration
- Test database setup
- Docker stack configuration
- Common issues and solutions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
jpmschweitzer
2025-12-07 23:19:38 +01:00
co-authored by Claude Sonnet 4.5
parent 4e4ce38db5
commit be0ff6780c
3 changed files with 768 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
services/scheduler/CHANGELOG.md
+191
View File
@@ -0,0 +1,191 @@
# Changelog
All notable changes to The Scheduler will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
## [1.0.0] - 2025-12-07
### Added
#### Core Scheduling System
- **Hybrid APScheduler + PostgreSQL architecture** for minute-based task scheduling
- Single scheduler job runs every minute
- Queries database for tasks scheduled for current minute
- Executes up to 5 tasks concurrently by priority
- Priority queue system (1-100, lower = higher priority)
- **Cron-like scheduling** with wildcard support (`-1` = any)
- Supports minute, hour, day_of_month, month, day_of_week patterns
- Flexible scheduling from every-minute to specific dates
- **Task execution tracking** with full audit trail
- Database tables: `scheduled_tasks` (definitions) and `task_executions` (history)
- Tracks status, duration, output, errors, and retry attempts
- Execution metadata stored as JSONB
#### REST API
- **Full CRUD API** for task management with FastAPI
- `POST /tasks` - Create new scheduled task
- `GET /tasks` - List all tasks with filtering (service, enabled)
- `GET /tasks/{name}` - Get task details
- `PUT /tasks/{name}` - Update task configuration
- `DELETE /tasks/{name}` - Remove task
- `POST /tasks/{name}/trigger` - Manually trigger task execution
- **Execution history endpoints**
- `GET /executions` - Query execution history
- Filter by task_name, status, service
- Pagination support (limit parameter)
- **System monitoring endpoints**
- `GET /health` - Health check
- `GET /stats` - System statistics (enabled tasks, running tasks, 24h execution counts)
- **API Key authentication** (Bearer token) for all protected endpoints
- **OpenAPI documentation** at `/docs`
#### Task Executors
- **Example Executor** (`example_executor.py`)
- Simple test implementation with configurable message and delay
- Demonstrates executor pattern
- **Documentation Sync Executor** (`doc_sync_executor.py`)
- Mirrors documentation from upstream Git repositories to Gitea
- Supports full repository mirroring or selective path syncing
- Creates date-tagged snapshots (YYYY-MM-DD format)
- Generates `.SYNC_INFO.md` with sync metadata
- Configurable upstream repo, paths, branch, and Gitea destination
- **Config Backup Executor** (`config_backup_executor.py`)
- Backs up Docker configurations and data directories
- Supports multiple source paths with exclusion patterns
- Optional compression (tar.gz)
- Retention policy (days-based cleanup)
- Creates timestamped backups
#### Pre-configured Tasks
- **FastAPI Documentation Sync** (monthly on 11th at 04:00)
- Syncs entire FastAPI repository to `library/docs-fastapi`
- Priority: 60 (maintenance)
- **Ollama Documentation Sync** (monthly on 12th at 04:00)
- Syncs only `/docs` folder from Ollama repository to `library/docs-ollama`
- Priority: 60 (maintenance)
- **Docker Config Backup** (daily at 03:05)
- Backs up Docker data and configurations
- Priority: 20 (user task)
- 30-day retention
- **Example Test Task** (every minute, can be disabled)
- Test task for validation
- Priority: 50 (maintenance)
#### Testing Infrastructure
- **Comprehensive test suite** with 80% code coverage
- 85 total tests across multiple test files
- Pytest configuration with markers (unit, integration, api, executor)
- Coverage reporting with pytest-cov
- **Test categories**:
- Unit tests: Fast tests with mocked dependencies
- API tests: Comprehensive endpoint testing (24 tests)
- Executor tests: Task executor validation
- Integration tests: Real database operations
- **Test database setup**
- Dedicated `test_scheduler` database on postgres-shared
- Automatic schema creation and cleanup
- Database fixtures for clean test state
- Test user: `test_scheduler_user`
- **Test fixtures** (conftest.py)
- Mock database connections
- Mock scheduler and executor
- Sample task data
- Authentication headers
- Clean database state management
#### Documentation
- **Comprehensive README.md** (500+ lines)
- Architecture overview with ASCII diagram
- Quick start guide
- Complete API reference with curl examples
- Task scheduling patterns and examples
- Executor development guide
- Testing guide with coverage metrics
- Development and debugging information
- Security and performance notes
- **Test database setup guide** (test_database_setup.sql)
- SQL script for creating test environment
- Schema matching production
- Test data fixtures
#### Configuration
- **Pydantic Settings** for environment-based configuration
- PostgreSQL connection settings
- Redis connection (for future use)
- Gitea authentication
- API key configuration
- Computed properties (database_url, redis_url)
- **Docker stack configuration** (stacks/scheduler.yml)
- Virtual environment setup on startup
- Git configuration for librarian user
- Health checks
- Network isolation (docker-dataplane)
- Resource limits
### Technical Details
#### Database Schema
```sql
scheduled_tasks:
- Task definitions/templates
- Scheduling configuration (minute/hour/day patterns)
- Priority, enabled status, retry settings
- Task configuration as JSONB
- Execution tracking fields
task_executions:
- Individual execution records
- Status tracking (pending, running, success, failed, timeout)
- Duration and timestamp tracking
- Output and error details
- Metadata as JSONB
```
#### Performance
- Minute-based processing with lightweight scheduler ticks
- Connection pooling for database efficiency
- Database indexes for optimized task queries
- Concurrent execution with configurable limit (default: 5)
- Priority-based execution order
#### Security
- API key authentication required for protected endpoints
- Network isolation on docker-dataplane
- Environment variable-based secrets
- Gitea token authentication for git operations
- Database credentials in environment
### Dependencies
- FastAPI (web framework)
- APScheduler (task scheduling)
- psycopg2-binary (PostgreSQL driver)
- Pydantic (configuration management)
- pytest + pytest-asyncio + pytest-cov (testing)
- GitPython (git operations)
### Coverage Metrics
```
Module Coverage
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
config.py 100% ✅
example_executor.py 100% ✅
main.py (API endpoints) 95% ✅
doc_sync_executor.py 78% ✅
executor.py (core logic) 71% ✅
config_backup_executor.py 50% 📈
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
TOTAL 80% 🎯
```
## [Unreleased]
### Planned
- Redis integration for distributed locking
- Webhook notifications for task completion
- Task dependencies (run task B after task A succeeds)
- Task groups and tags
- More executors (database backup, log rotation, etc.)
- Web UI for task management
- Metrics export (Prometheus)
- Advanced scheduling (last business day of month, etc.)
services/scheduler/README.md
+576
View File
@@ -0,0 +1,576 @@
# The Scheduler
**System-wide maintenance orchestration for automated task scheduling.**
The Scheduler is a hybrid APScheduler + PostgreSQL-based task scheduling system that handles backups, documentation mirroring, cleanup, and automated maintenance tasks across the Portainer Core infrastructure.
## Architecture
**Hybrid Design**: APScheduler + Database-driven Priority System
```
┌─────────────────────────────────────────────────────────┐
│ APScheduler (runs every minute) │
│ └─> Query PostgreSQL for tasks scheduled this minute │
│ └─> Execute up to 5 tasks concurrently by priority │
└─────────────────────────────────────────────────────────┘
```
**Key Features**:
- **Minute-based scheduling** with cron-like patterns (`-1` = wildcard)
- **Priority queue** (1-100, lower = higher priority)
- **Concurrent execution** (max 5 tasks simultaneously)
- **Execution tracking** (full audit trail in database)
- **REST API** for task management
- **Flexible executors** (modular task implementations)
## Quick Start
### Starting the Scheduler
```bash
# Deploy via Portainer
# Uses stack: /stacks/scheduler.yml
# Check health
curl http://localhost:8090/health
# View API docs
open http://localhost:8090/docs
```
### API Authentication
All protected endpoints require an API key:
```bash
# Set in environment or use default
export SCHEDULER_API_KEY="your-api-key-here"
# Make authenticated request
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
http://localhost:8090/tasks
```
## Scheduling Tasks
### Task Configuration
Tasks are defined with:
- **task_name**: Unique identifier
- **service**: Which service owns this task
- **executor**: Python module to execute
- **priority**: 1-100 (1=emergency, 10-30=user, 40-70=maintenance)
- **schedule**: Minute, hour, day, month, day_of_week (-1 = any)
- **config**: JSON configuration for executor
### Scheduling Examples
```python
# Every minute (wildcard)
minute=-1, hour=-1, day_of_month=-1, month=-1, day_of_week=-1
# Daily at 3:05 AM
minute=5, hour=3, day_of_month=-1, month=-1, day_of_week=-1
# 11th of every month at 4:00 AM
minute=0, hour=4, day_of_month=11, month=-1, day_of_week=-1
# Every Monday at 9:00 AM
minute=0, hour=9, day_of_month=-1, month=-1, day_of_week=0
# First day of January at midnight
minute=0, hour=0, day_of_month=1, month=1, day_of_week=-1
```
### Priority System
```
Priority Range Purpose Examples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1-5 Emergency/System Critical system tasks
10-30 User Tasks User-initiated operations
40-70 Maintenance Backups, cleanup, doc sync
71-100 Low Priority Optional background tasks
```
## REST API
### Task Management
```bash
# List all tasks
GET /tasks
GET /tasks?enabled=true&service=scheduler
# Get task details
GET /tasks/{task_name}
# Create new task
POST /tasks
{
"task_name": "my_task",
"service": "scheduler",
"executor": "example_executor",
"priority": 50,
"minute": 0,
"hour": 4,
"description": "Daily task at 4 AM",
"config": {"key": "value"}
}
# Update task
PUT /tasks/{task_name}
{
"priority": 60,
"enabled": false
}
# Delete task
DELETE /tasks/{task_name}
# Manually trigger task
POST /tasks/{task_name}/trigger
```
### Execution History
```bash
# View recent executions
GET /executions?limit=20
# Filter by task
GET /executions?task_name=my_task&limit=10
# Filter by status
GET /executions?status=success
# Filter by service
GET /executions?service=scheduler
```
### System Stats
```bash
# Get system statistics
GET /stats
{
"scheduler_running": true,
"tasks_enabled": 3,
"tasks_currently_running": 0,
"concurrent_limit": 5,
"execution_stats_24h": {
"success": 15,
"failed": 1
}
}
```
## Task Executors
Executors are Python modules that implement the actual task logic.
### Built-in Executors
#### 1. Example Executor
**File**: `src/executors/example_executor.py`
**Purpose**: Test/example implementation
```python
config = {
"message": "Hello World",
"delay_seconds": 2
}
```
#### 2. Doc Sync Executor
**File**: `src/executors/doc_sync_executor.py`
**Purpose**: Mirror documentation from GitHub to Gitea
```python
config = {
"project": "fastapi",
"upstream_repo": "https://github.com/tiangolo/fastapi.git",
"docs_paths": ["/docs"], # Empty = entire repo
"gitea_repo": "library/docs-fastapi",
"branch": "main"
}
```
**Features**:
- Clones upstream repository
- Filters to specific paths or mirrors entire repo
- Pushes to Gitea with authentication
- Creates date-tagged snapshots
- Generates `.SYNC_INFO.md` with metadata
#### 3. Config Backup Executor
**File**: `src/executors/config_backup_executor.py`
**Purpose**: Backup Docker configs and data
```python
config = {
"sources": [
{
"path": "/data/docker-data",
"name": "docker-data",
"excludes": ["*/cache/*", "*.log"]
}
],
"backup_dir": "/backups/configs",
"compress": true,
"retention_days": 30
}
```
### Creating Custom Executors
1. Create file in `src/executors/your_executor.py`
2. Implement async `execute(config: dict, settings: Settings) -> str` function
3. Return success message or raise exception on failure
```python
# src/executors/my_executor.py
async def execute(config: dict, settings: Settings) -> str:
"""
Execute custom task.
Args:
config: Task configuration from database
settings: Global scheduler settings
Returns:
Success message
Raises:
Exception: On task failure
"""
# Your task logic here
result = do_something(config.get('param'))
return f"Task completed: {result}"
```
4. Create task using API:
```bash
curl -X POST http://localhost:8090/tasks \
-H "Authorization: Bearer $SCHEDULER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"task_name": "my_custom_task",
"service": "scheduler",
"executor": "my_executor",
"priority": 50,
"minute": 0,
"hour": 2,
"config": {"param": "value"}
}'
```
## Current Tasks
### Active Schedules
| Task | Priority | Schedule | Description |
|------|----------|----------|-------------|
| `test_example_task` | 50 | Every minute | Test task (can be disabled) |
| `backup_docker_configs_daily` | 20 | Daily 03:05 | Backup Docker configs |
| `sync_fastapi_docs_monthly` | 60 | 11th @ 04:00 | Sync FastAPI docs to Gitea |
| `sync_ollama_docs_monthly` | 60 | 12th @ 04:00 | Sync Ollama docs to Gitea |
### Managing Tasks
```bash
# Disable test task
curl -X PUT http://localhost:8090/tasks/test_example_task \
-H "Authorization: Bearer $SCHEDULER_API_KEY" \
-H "Content-Type: application/json" \
-d '{"enabled": false}'
# View execution history
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
http://localhost:8090/executions?limit=10
```
## Database Schema
### scheduled_tasks
Stores task definitions (templates for execution).
```sql
- id (serial)
- task_name (varchar, unique)
- service (varchar) - owner service
- executor (varchar) - executor module name
- priority (integer) - 1-100
- minute, hour, day_of_month, month, day_of_week (integer) - schedule
- enabled (boolean)
- description (text)
- config (jsonb) - executor configuration
- last_run, last_status, last_duration_seconds - tracking
- retry_count, max_retries, timeout_seconds - execution control
- created_at, updated_at, created_by - metadata
```
### task_executions
Stores individual execution records (audit trail).
```sql
- id (serial)
- task_id (integer) - references scheduled_tasks
- task_name, service, executor, priority - snapshot
- status (varchar) - pending, running, success, failed, timeout
- triggered_by (varchar) - scheduler, manual, retry
- triggered_at, started_at, completed_at - timestamps
- duration_seconds (integer)
- output (text) - success message
- error (text) - error details
- retry_count (integer)
- metadata (jsonb)
```
## Testing
### Running Tests
```bash
# All tests with coverage
docker exec -w /app scheduler /venv/bin/pytest --cov=src --cov-report=term
# Only unit tests (fast, no database)
docker exec -w /app scheduler /venv/bin/pytest -m unit
# Only integration tests (with database)
docker exec -w /app scheduler /venv/bin/pytest -m integration
# Only API tests
docker exec -w /app scheduler /venv/bin/pytest -m api
# Only executor tests
docker exec -w /app scheduler /venv/bin/pytest -m executor
# Specific test file
docker exec -w /app scheduler /venv/bin/pytest tests/test_api.py -v
# With detailed output
docker exec -w /app scheduler /venv/bin/pytest -vv --tb=long
```
### Test Coverage
**Current Coverage: 80%**
```
Module Coverage
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
config.py 100% ✅
example_executor.py 100% ✅
main.py (API endpoints) 95% ✅
doc_sync_executor.py 78% ✅
executor.py (core logic) 71% ✅
config_backup_executor.py 50% 📈
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
TOTAL 80% 🎯
```
### Test Database
Tests use a dedicated PostgreSQL database:
- **Database**: `test_scheduler`
- **User**: `test_scheduler_user`
- **Location**: `postgres-shared` container
Tests automatically:
- Clean database before each test
- Insert test fixtures
- Verify database state
- Rollback changes after test
## Development
### Project Structure
```
scheduler/
├── src/
│ ├── config.py # Pydantic settings
│ ├── main.py # FastAPI app & endpoints
│ ├── executors/ # Task executors
│ │ ├── example_executor.py
│ │ ├── doc_sync_executor.py
│ │ └── config_backup_executor.py
│ └── tasks/
│ └── executor.py # Task execution engine
├── tests/
│ ├── conftest.py # Pytest fixtures
│ ├── test_api.py # API tests
│ ├── test_*_executor.py # Executor tests
│ └── test_database_integration.py
├── requirements.txt # Python dependencies
├── pytest.ini # Test configuration
└── README.md # This file
```
### Adding Dependencies
```bash
# Add to requirements.txt with version pinning
echo "new-package~=1.0.0" >> requirements.txt
# Rebuild venv in container
docker restart scheduler
```
### Debugging
```bash
# View logs
docker logs scheduler --tail 100 -f
# Check scheduler status
curl http://localhost:8090/health
# View database tasks
docker exec postgres-shared psql -U scheduler_user -d scheduler \
-c "SELECT task_name, enabled, priority, last_status FROM scheduled_tasks;"
# View execution history
docker exec postgres-shared psql -U scheduler_user -d scheduler \
-c "SELECT task_name, status, duration_seconds, completed_at FROM task_executions ORDER BY id DESC LIMIT 10;"
```
### Environment Variables
Required environment variables (set in `stacks/scheduler.yml`):
```bash
# Database
POSTGRES_HOST=postgres-shared
POSTGRES_PORT=5432
POSTGRES_DB=scheduler
POSTGRES_USER=scheduler_user
POSTGRES_PASSWORD=${SCHEDULER_DB_PASSWORD}
# API Security
SCHEDULER_API_KEY=${SCHEDULER_API_KEY}
# Gitea (for doc sync)
GITEA_URL=http://gitea:3000
GITEA_USER=librarian
GITEA_PASSWORD=${GITEA_PASSWORD}
# Redis (future use)
REDIS_HOST=redis-shared
REDIS_PORT=6379
REDIS_DB=3
```
## Monitoring
### Health Checks
```bash
# Container health
docker ps | grep scheduler
# API health
curl http://localhost:8090/health
# Scheduler running
curl http://localhost:8090/stats | jq '.scheduler_running'
```
### Metrics
```bash
# System statistics
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
http://localhost:8090/stats | jq
# Recent executions
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
"http://localhost:8090/executions?limit=20" | jq
# Failed tasks in last 24h
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
"http://localhost:8090/executions?status=failed" | jq
```
### Common Issues
**Task not running?**
1. Check if task is enabled: `GET /tasks/{name}`
2. Verify schedule matches current time
3. Check execution history for errors: `GET /executions?task_name={name}`
4. View scheduler logs: `docker logs scheduler`
**Task timing out?**
- Increase `timeout_seconds` in task configuration
- Check executor implementation for long-running operations
- Consider breaking into smaller tasks
**Database connection errors?**
- Verify postgres-shared container is running
- Check database credentials in environment
- Test connection: `docker exec scheduler psql -h postgres-shared -U scheduler_user -d scheduler`
## API Reference
Full OpenAPI documentation available at: `http://localhost:8090/docs`
### Response Codes
- `200` - Success
- `400` - Bad request (invalid data)
- `401` - Missing API key
- `403` - Invalid API key
- `404` - Resource not found
- `500` - Internal server error
### Pagination
Use `limit` parameter for controlling result count:
```bash
GET /executions?limit=50 # Default: 20, Max: 100
```
### Filtering
Most list endpoints support filtering:
```bash
GET /tasks?enabled=true&service=scheduler
GET /executions?task_name=my_task&status=success&limit=10
```
## Security
- **API Key Authentication**: Required for all protected endpoints
- **Network Isolation**: Runs on `docker-dataplane` network
- **Database Credentials**: Stored in environment variables
- **Gitea Tokens**: Used instead of passwords for Git operations
- **Resource Limits**: CPU and memory limits in docker-compose
## Performance
- **Concurrent Execution**: Max 5 tasks simultaneously
- **Minute-based Processing**: Lightweight scheduler tick every minute
- **Priority Queue**: Higher priority tasks execute first
- **Database Indexes**: Optimized queries for task selection
- **Connection Pooling**: Reuses database connections
## License
Part of Portainer Core infrastructure.
## Support
For issues or questions:
1. Check logs: `docker logs scheduler`
2. View health: `curl http://localhost:8090/health`
3. Review execution history: `GET /executions`
4. Check this README for common solutions
services/scheduler/src/tasks/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Task execution system for The Scheduler."""