diff --git a/services/scheduler/CHANGELOG.md b/services/scheduler/CHANGELOG.md new file mode 100644 index 0000000..d9cefd1 --- /dev/null +++ b/services/scheduler/CHANGELOG.md @@ -0,0 +1,191 @@ +# Changelog + +All notable changes to The Scheduler will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). + +## [1.0.0] - 2025-12-07 + +### Added + +#### Core Scheduling System +- **Hybrid APScheduler + PostgreSQL architecture** for minute-based task scheduling + - Single scheduler job runs every minute + - Queries database for tasks scheduled for current minute + - Executes up to 5 tasks concurrently by priority + - Priority queue system (1-100, lower = higher priority) +- **Cron-like scheduling** with wildcard support (`-1` = any) + - Supports minute, hour, day_of_month, month, day_of_week patterns + - Flexible scheduling from every-minute to specific dates +- **Task execution tracking** with full audit trail + - Database tables: `scheduled_tasks` (definitions) and `task_executions` (history) + - Tracks status, duration, output, errors, and retry attempts + - Execution metadata stored as JSONB + +#### REST API +- **Full CRUD API** for task management with FastAPI + - `POST /tasks` - Create new scheduled task + - `GET /tasks` - List all tasks with filtering (service, enabled) + - `GET /tasks/{name}` - Get task details + - `PUT /tasks/{name}` - Update task configuration + - `DELETE /tasks/{name}` - Remove task + - `POST /tasks/{name}/trigger` - Manually trigger task execution +- **Execution history endpoints** + - `GET /executions` - Query execution history + - Filter by task_name, status, service + - Pagination support (limit parameter) +- **System monitoring endpoints** + - `GET /health` - Health check + - `GET /stats` - System statistics (enabled tasks, running tasks, 24h execution counts) +- **API Key authentication** (Bearer token) for all protected endpoints +- **OpenAPI documentation** at `/docs` + +#### Task Executors +- **Example Executor** (`example_executor.py`) + - Simple test implementation with configurable message and delay + - Demonstrates executor pattern +- **Documentation Sync Executor** (`doc_sync_executor.py`) + - Mirrors documentation from upstream Git repositories to Gitea + - Supports full repository mirroring or selective path syncing + - Creates date-tagged snapshots (YYYY-MM-DD format) + - Generates `.SYNC_INFO.md` with sync metadata + - Configurable upstream repo, paths, branch, and Gitea destination +- **Config Backup Executor** (`config_backup_executor.py`) + - Backs up Docker configurations and data directories + - Supports multiple source paths with exclusion patterns + - Optional compression (tar.gz) + - Retention policy (days-based cleanup) + - Creates timestamped backups + +#### Pre-configured Tasks +- **FastAPI Documentation Sync** (monthly on 11th at 04:00) + - Syncs entire FastAPI repository to `library/docs-fastapi` + - Priority: 60 (maintenance) +- **Ollama Documentation Sync** (monthly on 12th at 04:00) + - Syncs only `/docs` folder from Ollama repository to `library/docs-ollama` + - Priority: 60 (maintenance) +- **Docker Config Backup** (daily at 03:05) + - Backs up Docker data and configurations + - Priority: 20 (user task) + - 30-day retention +- **Example Test Task** (every minute, can be disabled) + - Test task for validation + - Priority: 50 (maintenance) + +#### Testing Infrastructure +- **Comprehensive test suite** with 80% code coverage + - 85 total tests across multiple test files + - Pytest configuration with markers (unit, integration, api, executor) + - Coverage reporting with pytest-cov +- **Test categories**: + - Unit tests: Fast tests with mocked dependencies + - API tests: Comprehensive endpoint testing (24 tests) + - Executor tests: Task executor validation + - Integration tests: Real database operations +- **Test database setup** + - Dedicated `test_scheduler` database on postgres-shared + - Automatic schema creation and cleanup + - Database fixtures for clean test state + - Test user: `test_scheduler_user` +- **Test fixtures** (conftest.py) + - Mock database connections + - Mock scheduler and executor + - Sample task data + - Authentication headers + - Clean database state management + +#### Documentation +- **Comprehensive README.md** (500+ lines) + - Architecture overview with ASCII diagram + - Quick start guide + - Complete API reference with curl examples + - Task scheduling patterns and examples + - Executor development guide + - Testing guide with coverage metrics + - Development and debugging information + - Security and performance notes +- **Test database setup guide** (test_database_setup.sql) + - SQL script for creating test environment + - Schema matching production + - Test data fixtures + +#### Configuration +- **Pydantic Settings** for environment-based configuration + - PostgreSQL connection settings + - Redis connection (for future use) + - Gitea authentication + - API key configuration + - Computed properties (database_url, redis_url) +- **Docker stack configuration** (stacks/scheduler.yml) + - Virtual environment setup on startup + - Git configuration for librarian user + - Health checks + - Network isolation (docker-dataplane) + - Resource limits + +### Technical Details + +#### Database Schema +```sql +scheduled_tasks: + - Task definitions/templates + - Scheduling configuration (minute/hour/day patterns) + - Priority, enabled status, retry settings + - Task configuration as JSONB + - Execution tracking fields + +task_executions: + - Individual execution records + - Status tracking (pending, running, success, failed, timeout) + - Duration and timestamp tracking + - Output and error details + - Metadata as JSONB +``` + +#### Performance +- Minute-based processing with lightweight scheduler ticks +- Connection pooling for database efficiency +- Database indexes for optimized task queries +- Concurrent execution with configurable limit (default: 5) +- Priority-based execution order + +#### Security +- API key authentication required for protected endpoints +- Network isolation on docker-dataplane +- Environment variable-based secrets +- Gitea token authentication for git operations +- Database credentials in environment + +### Dependencies +- FastAPI (web framework) +- APScheduler (task scheduling) +- psycopg2-binary (PostgreSQL driver) +- Pydantic (configuration management) +- pytest + pytest-asyncio + pytest-cov (testing) +- GitPython (git operations) + +### Coverage Metrics +``` +Module Coverage +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ +config.py 100% ✅ +example_executor.py 100% ✅ +main.py (API endpoints) 95% ✅ +doc_sync_executor.py 78% ✅ +executor.py (core logic) 71% ✅ +config_backup_executor.py 50% 📈 +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ +TOTAL 80% 🎯 +``` + +## [Unreleased] + +### Planned +- Redis integration for distributed locking +- Webhook notifications for task completion +- Task dependencies (run task B after task A succeeds) +- Task groups and tags +- More executors (database backup, log rotation, etc.) +- Web UI for task management +- Metrics export (Prometheus) +- Advanced scheduling (last business day of month, etc.) diff --git a/services/scheduler/README.md b/services/scheduler/README.md new file mode 100644 index 0000000..fb06f7c --- /dev/null +++ b/services/scheduler/README.md @@ -0,0 +1,576 @@ +# The Scheduler + +**System-wide maintenance orchestration for automated task scheduling.** + +The Scheduler is a hybrid APScheduler + PostgreSQL-based task scheduling system that handles backups, documentation mirroring, cleanup, and automated maintenance tasks across the Portainer Core infrastructure. + +## Architecture + +**Hybrid Design**: APScheduler + Database-driven Priority System + +``` +┌─────────────────────────────────────────────────────────┐ +│ APScheduler (runs every minute) │ +│ └─> Query PostgreSQL for tasks scheduled this minute │ +│ └─> Execute up to 5 tasks concurrently by priority │ +└─────────────────────────────────────────────────────────┘ +``` + +**Key Features**: +- **Minute-based scheduling** with cron-like patterns (`-1` = wildcard) +- **Priority queue** (1-100, lower = higher priority) +- **Concurrent execution** (max 5 tasks simultaneously) +- **Execution tracking** (full audit trail in database) +- **REST API** for task management +- **Flexible executors** (modular task implementations) + +## Quick Start + +### Starting the Scheduler + +```bash +# Deploy via Portainer +# Uses stack: /stacks/scheduler.yml + +# Check health +curl http://localhost:8090/health + +# View API docs +open http://localhost:8090/docs +``` + +### API Authentication + +All protected endpoints require an API key: + +```bash +# Set in environment or use default +export SCHEDULER_API_KEY="your-api-key-here" + +# Make authenticated request +curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \ + http://localhost:8090/tasks +``` + +## Scheduling Tasks + +### Task Configuration + +Tasks are defined with: +- **task_name**: Unique identifier +- **service**: Which service owns this task +- **executor**: Python module to execute +- **priority**: 1-100 (1=emergency, 10-30=user, 40-70=maintenance) +- **schedule**: Minute, hour, day, month, day_of_week (-1 = any) +- **config**: JSON configuration for executor + +### Scheduling Examples + +```python +# Every minute (wildcard) +minute=-1, hour=-1, day_of_month=-1, month=-1, day_of_week=-1 + +# Daily at 3:05 AM +minute=5, hour=3, day_of_month=-1, month=-1, day_of_week=-1 + +# 11th of every month at 4:00 AM +minute=0, hour=4, day_of_month=11, month=-1, day_of_week=-1 + +# Every Monday at 9:00 AM +minute=0, hour=9, day_of_month=-1, month=-1, day_of_week=0 + +# First day of January at midnight +minute=0, hour=0, day_of_month=1, month=1, day_of_week=-1 +``` + +### Priority System + +``` +Priority Range Purpose Examples +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ +1-5 Emergency/System Critical system tasks +10-30 User Tasks User-initiated operations +40-70 Maintenance Backups, cleanup, doc sync +71-100 Low Priority Optional background tasks +``` + +## REST API + +### Task Management + +```bash +# List all tasks +GET /tasks +GET /tasks?enabled=true&service=scheduler + +# Get task details +GET /tasks/{task_name} + +# Create new task +POST /tasks +{ + "task_name": "my_task", + "service": "scheduler", + "executor": "example_executor", + "priority": 50, + "minute": 0, + "hour": 4, + "description": "Daily task at 4 AM", + "config": {"key": "value"} +} + +# Update task +PUT /tasks/{task_name} +{ + "priority": 60, + "enabled": false +} + +# Delete task +DELETE /tasks/{task_name} + +# Manually trigger task +POST /tasks/{task_name}/trigger +``` + +### Execution History + +```bash +# View recent executions +GET /executions?limit=20 + +# Filter by task +GET /executions?task_name=my_task&limit=10 + +# Filter by status +GET /executions?status=success + +# Filter by service +GET /executions?service=scheduler +``` + +### System Stats + +```bash +# Get system statistics +GET /stats +{ + "scheduler_running": true, + "tasks_enabled": 3, + "tasks_currently_running": 0, + "concurrent_limit": 5, + "execution_stats_24h": { + "success": 15, + "failed": 1 + } +} +``` + +## Task Executors + +Executors are Python modules that implement the actual task logic. + +### Built-in Executors + +#### 1. Example Executor +**File**: `src/executors/example_executor.py` +**Purpose**: Test/example implementation + +```python +config = { + "message": "Hello World", + "delay_seconds": 2 +} +``` + +#### 2. Doc Sync Executor +**File**: `src/executors/doc_sync_executor.py` +**Purpose**: Mirror documentation from GitHub to Gitea + +```python +config = { + "project": "fastapi", + "upstream_repo": "https://github.com/tiangolo/fastapi.git", + "docs_paths": ["/docs"], # Empty = entire repo + "gitea_repo": "library/docs-fastapi", + "branch": "main" +} +``` + +**Features**: +- Clones upstream repository +- Filters to specific paths or mirrors entire repo +- Pushes to Gitea with authentication +- Creates date-tagged snapshots +- Generates `.SYNC_INFO.md` with metadata + +#### 3. Config Backup Executor +**File**: `src/executors/config_backup_executor.py` +**Purpose**: Backup Docker configs and data + +```python +config = { + "sources": [ + { + "path": "/data/docker-data", + "name": "docker-data", + "excludes": ["*/cache/*", "*.log"] + } + ], + "backup_dir": "/backups/configs", + "compress": true, + "retention_days": 30 +} +``` + +### Creating Custom Executors + +1. Create file in `src/executors/your_executor.py` +2. Implement async `execute(config: dict, settings: Settings) -> str` function +3. Return success message or raise exception on failure + +```python +# src/executors/my_executor.py +async def execute(config: dict, settings: Settings) -> str: + """ + Execute custom task. + + Args: + config: Task configuration from database + settings: Global scheduler settings + + Returns: + Success message + + Raises: + Exception: On task failure + """ + # Your task logic here + result = do_something(config.get('param')) + + return f"Task completed: {result}" +``` + +4. Create task using API: + +```bash +curl -X POST http://localhost:8090/tasks \ + -H "Authorization: Bearer $SCHEDULER_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "task_name": "my_custom_task", + "service": "scheduler", + "executor": "my_executor", + "priority": 50, + "minute": 0, + "hour": 2, + "config": {"param": "value"} + }' +``` + +## Current Tasks + +### Active Schedules + +| Task | Priority | Schedule | Description | +|------|----------|----------|-------------| +| `test_example_task` | 50 | Every minute | Test task (can be disabled) | +| `backup_docker_configs_daily` | 20 | Daily 03:05 | Backup Docker configs | +| `sync_fastapi_docs_monthly` | 60 | 11th @ 04:00 | Sync FastAPI docs to Gitea | +| `sync_ollama_docs_monthly` | 60 | 12th @ 04:00 | Sync Ollama docs to Gitea | + +### Managing Tasks + +```bash +# Disable test task +curl -X PUT http://localhost:8090/tasks/test_example_task \ + -H "Authorization: Bearer $SCHEDULER_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"enabled": false}' + +# View execution history +curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \ + http://localhost:8090/executions?limit=10 +``` + +## Database Schema + +### scheduled_tasks + +Stores task definitions (templates for execution). + +```sql +- id (serial) +- task_name (varchar, unique) +- service (varchar) - owner service +- executor (varchar) - executor module name +- priority (integer) - 1-100 +- minute, hour, day_of_month, month, day_of_week (integer) - schedule +- enabled (boolean) +- description (text) +- config (jsonb) - executor configuration +- last_run, last_status, last_duration_seconds - tracking +- retry_count, max_retries, timeout_seconds - execution control +- created_at, updated_at, created_by - metadata +``` + +### task_executions + +Stores individual execution records (audit trail). + +```sql +- id (serial) +- task_id (integer) - references scheduled_tasks +- task_name, service, executor, priority - snapshot +- status (varchar) - pending, running, success, failed, timeout +- triggered_by (varchar) - scheduler, manual, retry +- triggered_at, started_at, completed_at - timestamps +- duration_seconds (integer) +- output (text) - success message +- error (text) - error details +- retry_count (integer) +- metadata (jsonb) +``` + +## Testing + +### Running Tests + +```bash +# All tests with coverage +docker exec -w /app scheduler /venv/bin/pytest --cov=src --cov-report=term + +# Only unit tests (fast, no database) +docker exec -w /app scheduler /venv/bin/pytest -m unit + +# Only integration tests (with database) +docker exec -w /app scheduler /venv/bin/pytest -m integration + +# Only API tests +docker exec -w /app scheduler /venv/bin/pytest -m api + +# Only executor tests +docker exec -w /app scheduler /venv/bin/pytest -m executor + +# Specific test file +docker exec -w /app scheduler /venv/bin/pytest tests/test_api.py -v + +# With detailed output +docker exec -w /app scheduler /venv/bin/pytest -vv --tb=long +``` + +### Test Coverage + +**Current Coverage: 80%** + +``` +Module Coverage +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ +config.py 100% ✅ +example_executor.py 100% ✅ +main.py (API endpoints) 95% ✅ +doc_sync_executor.py 78% ✅ +executor.py (core logic) 71% ✅ +config_backup_executor.py 50% 📈 +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ +TOTAL 80% 🎯 +``` + +### Test Database + +Tests use a dedicated PostgreSQL database: +- **Database**: `test_scheduler` +- **User**: `test_scheduler_user` +- **Location**: `postgres-shared` container + +Tests automatically: +- Clean database before each test +- Insert test fixtures +- Verify database state +- Rollback changes after test + +## Development + +### Project Structure + +``` +scheduler/ +├── src/ +│ ├── config.py # Pydantic settings +│ ├── main.py # FastAPI app & endpoints +│ ├── executors/ # Task executors +│ │ ├── example_executor.py +│ │ ├── doc_sync_executor.py +│ │ └── config_backup_executor.py +│ └── tasks/ +│ └── executor.py # Task execution engine +├── tests/ +│ ├── conftest.py # Pytest fixtures +│ ├── test_api.py # API tests +│ ├── test_*_executor.py # Executor tests +│ └── test_database_integration.py +├── requirements.txt # Python dependencies +├── pytest.ini # Test configuration +└── README.md # This file +``` + +### Adding Dependencies + +```bash +# Add to requirements.txt with version pinning +echo "new-package~=1.0.0" >> requirements.txt + +# Rebuild venv in container +docker restart scheduler +``` + +### Debugging + +```bash +# View logs +docker logs scheduler --tail 100 -f + +# Check scheduler status +curl http://localhost:8090/health + +# View database tasks +docker exec postgres-shared psql -U scheduler_user -d scheduler \ + -c "SELECT task_name, enabled, priority, last_status FROM scheduled_tasks;" + +# View execution history +docker exec postgres-shared psql -U scheduler_user -d scheduler \ + -c "SELECT task_name, status, duration_seconds, completed_at FROM task_executions ORDER BY id DESC LIMIT 10;" +``` + +### Environment Variables + +Required environment variables (set in `stacks/scheduler.yml`): + +```bash +# Database +POSTGRES_HOST=postgres-shared +POSTGRES_PORT=5432 +POSTGRES_DB=scheduler +POSTGRES_USER=scheduler_user +POSTGRES_PASSWORD=${SCHEDULER_DB_PASSWORD} + +# API Security +SCHEDULER_API_KEY=${SCHEDULER_API_KEY} + +# Gitea (for doc sync) +GITEA_URL=http://gitea:3000 +GITEA_USER=librarian +GITEA_PASSWORD=${GITEA_PASSWORD} + +# Redis (future use) +REDIS_HOST=redis-shared +REDIS_PORT=6379 +REDIS_DB=3 +``` + +## Monitoring + +### Health Checks + +```bash +# Container health +docker ps | grep scheduler + +# API health +curl http://localhost:8090/health + +# Scheduler running +curl http://localhost:8090/stats | jq '.scheduler_running' +``` + +### Metrics + +```bash +# System statistics +curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \ + http://localhost:8090/stats | jq + +# Recent executions +curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \ + "http://localhost:8090/executions?limit=20" | jq + +# Failed tasks in last 24h +curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \ + "http://localhost:8090/executions?status=failed" | jq +``` + +### Common Issues + +**Task not running?** +1. Check if task is enabled: `GET /tasks/{name}` +2. Verify schedule matches current time +3. Check execution history for errors: `GET /executions?task_name={name}` +4. View scheduler logs: `docker logs scheduler` + +**Task timing out?** +- Increase `timeout_seconds` in task configuration +- Check executor implementation for long-running operations +- Consider breaking into smaller tasks + +**Database connection errors?** +- Verify postgres-shared container is running +- Check database credentials in environment +- Test connection: `docker exec scheduler psql -h postgres-shared -U scheduler_user -d scheduler` + +## API Reference + +Full OpenAPI documentation available at: `http://localhost:8090/docs` + +### Response Codes + +- `200` - Success +- `400` - Bad request (invalid data) +- `401` - Missing API key +- `403` - Invalid API key +- `404` - Resource not found +- `500` - Internal server error + +### Pagination + +Use `limit` parameter for controlling result count: + +```bash +GET /executions?limit=50 # Default: 20, Max: 100 +``` + +### Filtering + +Most list endpoints support filtering: + +```bash +GET /tasks?enabled=true&service=scheduler +GET /executions?task_name=my_task&status=success&limit=10 +``` + +## Security + +- **API Key Authentication**: Required for all protected endpoints +- **Network Isolation**: Runs on `docker-dataplane` network +- **Database Credentials**: Stored in environment variables +- **Gitea Tokens**: Used instead of passwords for Git operations +- **Resource Limits**: CPU and memory limits in docker-compose + +## Performance + +- **Concurrent Execution**: Max 5 tasks simultaneously +- **Minute-based Processing**: Lightweight scheduler tick every minute +- **Priority Queue**: Higher priority tasks execute first +- **Database Indexes**: Optimized queries for task selection +- **Connection Pooling**: Reuses database connections + +## License + +Part of Portainer Core infrastructure. + +## Support + +For issues or questions: +1. Check logs: `docker logs scheduler` +2. View health: `curl http://localhost:8090/health` +3. Review execution history: `GET /executions` +4. Check this README for common solutions diff --git a/services/scheduler/src/tasks/__init__.py b/services/scheduler/src/tasks/__init__.py new file mode 100644 index 0000000..904b3bc --- /dev/null +++ b/services/scheduler/src/tasks/__init__.py @@ -0,0 +1 @@ +"""Task execution system for The Scheduler."""