docs(scheduler): add comprehensive README and CHANGELOG

Add complete documentation for The Scheduler service.

README.md (500+ lines):
- Architecture overview with ASCII diagram
- Quick start guide
- Complete API reference with curl examples
- Task scheduling patterns and examples
- Priority system documentation
- Built-in executors documentation (example, doc_sync, config_backup)
- Custom executor development guide
- Current tasks table
- Database schema documentation
- Testing guide with coverage metrics
- Development and debugging information
- Monitoring and troubleshooting
- Security and performance notes
- API reference with response codes and filtering

CHANGELOG.md:
- Initial v1.0.0 release documentation
- Core features and architecture
- REST API endpoints
- Task executors and pre-configured tasks
- Testing infrastructure and metrics
- Technical details and dependencies
- Coverage metrics breakdown
- Planned features for future releases

Documentation covers:
- All API endpoints and authentication
- Scheduling examples (every minute, daily, monthly, etc.)
- Priority ranges and usage
- Executor configuration
- Test database setup
- Docker stack configuration
- Common issues and solutions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

services/scheduler/CHANGELOG.md

@@ -0,0 +1,191 @@
+# Changelog
+
+All notable changes to The Scheduler will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
+
+## [1.0.0] - 2025-12-07
+
+### Added
+
+#### Core Scheduling System
+- **Hybrid APScheduler + PostgreSQL architecture** for minute-based task scheduling
+  - Single scheduler job runs every minute
+  - Queries database for tasks scheduled for current minute
+  - Executes up to 5 tasks concurrently by priority
+  - Priority queue system (1-100, lower = higher priority)
+- **Cron-like scheduling** with wildcard support (`-1` = any)
+  - Supports minute, hour, day_of_month, month, day_of_week patterns
+  - Flexible scheduling from every-minute to specific dates
+- **Task execution tracking** with full audit trail
+  - Database tables: `scheduled_tasks` (definitions) and `task_executions` (history)
+  - Tracks status, duration, output, errors, and retry attempts
+  - Execution metadata stored as JSONB
+
+#### REST API
+- **Full CRUD API** for task management with FastAPI
+  - `POST /tasks` - Create new scheduled task
+  - `GET /tasks` - List all tasks with filtering (service, enabled)
+  - `GET /tasks/{name}` - Get task details
+  - `PUT /tasks/{name}` - Update task configuration
+  - `DELETE /tasks/{name}` - Remove task
+  - `POST /tasks/{name}/trigger` - Manually trigger task execution
+- **Execution history endpoints**
+  - `GET /executions` - Query execution history
+  - Filter by task_name, status, service
+  - Pagination support (limit parameter)
+- **System monitoring endpoints**
+  - `GET /health` - Health check
+  - `GET /stats` - System statistics (enabled tasks, running tasks, 24h execution counts)
+- **API Key authentication** (Bearer token) for all protected endpoints
+- **OpenAPI documentation** at `/docs`
+
+#### Task Executors
+- **Example Executor** (`example_executor.py`)
+  - Simple test implementation with configurable message and delay
+  - Demonstrates executor pattern
+- **Documentation Sync Executor** (`doc_sync_executor.py`)
+  - Mirrors documentation from upstream Git repositories to Gitea
+  - Supports full repository mirroring or selective path syncing
+  - Creates date-tagged snapshots (YYYY-MM-DD format)
+  - Generates `.SYNC_INFO.md` with sync metadata
+  - Configurable upstream repo, paths, branch, and Gitea destination
+- **Config Backup Executor** (`config_backup_executor.py`)
+  - Backs up Docker configurations and data directories
+  - Supports multiple source paths with exclusion patterns
+  - Optional compression (tar.gz)
+  - Retention policy (days-based cleanup)
+  - Creates timestamped backups
+
+#### Pre-configured Tasks
+- **FastAPI Documentation Sync** (monthly on 11th at 04:00)
+  - Syncs entire FastAPI repository to `library/docs-fastapi`
+  - Priority: 60 (maintenance)
+- **Ollama Documentation Sync** (monthly on 12th at 04:00)
+  - Syncs only `/docs` folder from Ollama repository to `library/docs-ollama`
+  - Priority: 60 (maintenance)
+- **Docker Config Backup** (daily at 03:05)
+  - Backs up Docker data and configurations
+  - Priority: 20 (user task)
+  - 30-day retention
+- **Example Test Task** (every minute, can be disabled)
+  - Test task for validation
+  - Priority: 50 (maintenance)
+
+#### Testing Infrastructure
+- **Comprehensive test suite** with 80% code coverage
+  - 85 total tests across multiple test files
+  - Pytest configuration with markers (unit, integration, api, executor)
+  - Coverage reporting with pytest-cov
+- **Test categories**:
+  - Unit tests: Fast tests with mocked dependencies
+  - API tests: Comprehensive endpoint testing (24 tests)
+  - Executor tests: Task executor validation
+  - Integration tests: Real database operations
+- **Test database setup**
+  - Dedicated `test_scheduler` database on postgres-shared
+  - Automatic schema creation and cleanup
+  - Database fixtures for clean test state
+  - Test user: `test_scheduler_user`
+- **Test fixtures** (conftest.py)
+  - Mock database connections
+  - Mock scheduler and executor
+  - Sample task data
+  - Authentication headers
+  - Clean database state management
+
+#### Documentation
+- **Comprehensive README.md** (500+ lines)
+  - Architecture overview with ASCII diagram
+  - Quick start guide
+  - Complete API reference with curl examples
+  - Task scheduling patterns and examples
+  - Executor development guide
+  - Testing guide with coverage metrics
+  - Development and debugging information
+  - Security and performance notes
+- **Test database setup guide** (test_database_setup.sql)
+  - SQL script for creating test environment
+  - Schema matching production
+  - Test data fixtures
+
+#### Configuration
+- **Pydantic Settings** for environment-based configuration
+  - PostgreSQL connection settings
+  - Redis connection (for future use)
+  - Gitea authentication
+  - API key configuration
+  - Computed properties (database_url, redis_url)
+- **Docker stack configuration** (stacks/scheduler.yml)
+  - Virtual environment setup on startup
+  - Git configuration for librarian user
+  - Health checks
+  - Network isolation (docker-dataplane)
+  - Resource limits
+
+### Technical Details
+
+#### Database Schema
+```sql
+scheduled_tasks:
+  - Task definitions/templates
+  - Scheduling configuration (minute/hour/day patterns)
+  - Priority, enabled status, retry settings
+  - Task configuration as JSONB
+  - Execution tracking fields
+
+task_executions:
+  - Individual execution records
+  - Status tracking (pending, running, success, failed, timeout)
+  - Duration and timestamp tracking
+  - Output and error details
+  - Metadata as JSONB
+```
+
+#### Performance
+- Minute-based processing with lightweight scheduler ticks
+- Connection pooling for database efficiency
+- Database indexes for optimized task queries
+- Concurrent execution with configurable limit (default: 5)
+- Priority-based execution order
+
+#### Security
+- API key authentication required for protected endpoints
+- Network isolation on docker-dataplane
+- Environment variable-based secrets
+- Gitea token authentication for git operations
+- Database credentials in environment
+
+### Dependencies
+- FastAPI (web framework)
+- APScheduler (task scheduling)
+- psycopg2-binary (PostgreSQL driver)
+- Pydantic (configuration management)
+- pytest + pytest-asyncio + pytest-cov (testing)
+- GitPython (git operations)
+
+### Coverage Metrics
+```
+Module                          Coverage
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+config.py                       100% ✅
+example_executor.py             100% ✅
+main.py (API endpoints)          95% ✅
+doc_sync_executor.py             78% ✅
+executor.py (core logic)         71% ✅
+config_backup_executor.py        50% 📈
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+TOTAL                            80% 🎯
+```
+
+## [Unreleased]
+
+### Planned
+- Redis integration for distributed locking
+- Webhook notifications for task completion
+- Task dependencies (run task B after task A succeeds)
+- Task groups and tags
+- More executors (database backup, log rotation, etc.)
+- Web UI for task management
+- Metrics export (Prometheus)
+- Advanced scheduling (last business day of month, etc.)

services/scheduler/README.md

@@ -0,0 +1,576 @@
+# The Scheduler
+
+**System-wide maintenance orchestration for automated task scheduling.**
+
+The Scheduler is a hybrid APScheduler + PostgreSQL-based task scheduling system that handles backups, documentation mirroring, cleanup, and automated maintenance tasks across the Portainer Core infrastructure.
+
+## Architecture
+
+**Hybrid Design**: APScheduler + Database-driven Priority System
+
+```
+┌─────────────────────────────────────────────────────────┐
+│  APScheduler (runs every minute)                        │
+│  └─> Query PostgreSQL for tasks scheduled this minute   │
+│      └─> Execute up to 5 tasks concurrently by priority │
+└─────────────────────────────────────────────────────────┘
+```
+
+**Key Features**:
+- **Minute-based scheduling** with cron-like patterns (`-1` = wildcard)
+- **Priority queue** (1-100, lower = higher priority)
+- **Concurrent execution** (max 5 tasks simultaneously)
+- **Execution tracking** (full audit trail in database)
+- **REST API** for task management
+- **Flexible executors** (modular task implementations)
+
+## Quick Start
+
+### Starting the Scheduler
+
+```bash
+# Deploy via Portainer
+# Uses stack: /stacks/scheduler.yml
+
+# Check health
+curl http://localhost:8090/health
+
+# View API docs
+open http://localhost:8090/docs
+```
+
+### API Authentication
+
+All protected endpoints require an API key:
+
+```bash
+# Set in environment or use default
+export SCHEDULER_API_KEY="your-api-key-here"
+
+# Make authenticated request
+curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
+  http://localhost:8090/tasks
+```
+
+## Scheduling Tasks
+
+### Task Configuration
+
+Tasks are defined with:
+- **task_name**: Unique identifier
+- **service**: Which service owns this task
+- **executor**: Python module to execute
+- **priority**: 1-100 (1=emergency, 10-30=user, 40-70=maintenance)
+- **schedule**: Minute, hour, day, month, day_of_week (-1 = any)
+- **config**: JSON configuration for executor
+
+### Scheduling Examples
+
+```python
+# Every minute (wildcard)
+minute=-1, hour=-1, day_of_month=-1, month=-1, day_of_week=-1
+
+# Daily at 3:05 AM
+minute=5, hour=3, day_of_month=-1, month=-1, day_of_week=-1
+
+# 11th of every month at 4:00 AM
+minute=0, hour=4, day_of_month=11, month=-1, day_of_week=-1
+
+# Every Monday at 9:00 AM
+minute=0, hour=9, day_of_month=-1, month=-1, day_of_week=0
+
+# First day of January at midnight
+minute=0, hour=0, day_of_month=1, month=1, day_of_week=-1
+```
+
+### Priority System
+
+```
+Priority Range  Purpose           Examples
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+1-5             Emergency/System  Critical system tasks
+10-30           User Tasks        User-initiated operations
+40-70           Maintenance       Backups, cleanup, doc sync
+71-100          Low Priority      Optional background tasks
+```
+
+## REST API
+
+### Task Management
+
+```bash
+# List all tasks
+GET /tasks
+GET /tasks?enabled=true&service=scheduler
+
+# Get task details
+GET /tasks/{task_name}
+
+# Create new task
+POST /tasks
+{
+  "task_name": "my_task",
+  "service": "scheduler",
+  "executor": "example_executor",
+  "priority": 50,
+  "minute": 0,
+  "hour": 4,
+  "description": "Daily task at 4 AM",
+  "config": {"key": "value"}
+}
+
+# Update task
+PUT /tasks/{task_name}
+{
+  "priority": 60,
+  "enabled": false
+}
+
+# Delete task
+DELETE /tasks/{task_name}
+
+# Manually trigger task
+POST /tasks/{task_name}/trigger
+```
+
+### Execution History
+
+```bash
+# View recent executions
+GET /executions?limit=20
+
+# Filter by task
+GET /executions?task_name=my_task&limit=10
+
+# Filter by status
+GET /executions?status=success
+
+# Filter by service
+GET /executions?service=scheduler
+```
+
+### System Stats
+
+```bash
+# Get system statistics
+GET /stats
+{
+  "scheduler_running": true,
+  "tasks_enabled": 3,
+  "tasks_currently_running": 0,
+  "concurrent_limit": 5,
+  "execution_stats_24h": {
+    "success": 15,
+    "failed": 1
+  }
+}
+```
+
+## Task Executors
+
+Executors are Python modules that implement the actual task logic.
+
+### Built-in Executors
+
+#### 1. Example Executor
+**File**: `src/executors/example_executor.py`
+**Purpose**: Test/example implementation
+
+```python
+config = {
+    "message": "Hello World",
+    "delay_seconds": 2
+}
+```
+
+#### 2. Doc Sync Executor
+**File**: `src/executors/doc_sync_executor.py`
+**Purpose**: Mirror documentation from GitHub to Gitea
+
+```python
+config = {
+    "project": "fastapi",
+    "upstream_repo": "https://github.com/tiangolo/fastapi.git",
+    "docs_paths": ["/docs"],  # Empty = entire repo
+    "gitea_repo": "library/docs-fastapi",
+    "branch": "main"
+}
+```
+
+**Features**:
+- Clones upstream repository
+- Filters to specific paths or mirrors entire repo
+- Pushes to Gitea with authentication
+- Creates date-tagged snapshots
+- Generates `.SYNC_INFO.md` with metadata
+
+#### 3. Config Backup Executor
+**File**: `src/executors/config_backup_executor.py`
+**Purpose**: Backup Docker configs and data
+
+```python
+config = {
+    "sources": [
+        {
+            "path": "/data/docker-data",
+            "name": "docker-data",
+            "excludes": ["*/cache/*", "*.log"]
+        }
+    ],
+    "backup_dir": "/backups/configs",
+    "compress": true,
+    "retention_days": 30
+}
+```
+
+### Creating Custom Executors
+
+1. Create file in `src/executors/your_executor.py`
+2. Implement async `execute(config: dict, settings: Settings) -> str` function
+3. Return success message or raise exception on failure
+
+```python
+# src/executors/my_executor.py
+async def execute(config: dict, settings: Settings) -> str:
+    """
+    Execute custom task.
+
+    Args:
+        config: Task configuration from database
+        settings: Global scheduler settings
+
+    Returns:
+        Success message
+
+    Raises:
+        Exception: On task failure
+    """
+    # Your task logic here
+    result = do_something(config.get('param'))
+
+    return f"Task completed: {result}"
+```
+
+4. Create task using API:
+
+```bash
+curl -X POST http://localhost:8090/tasks \
+  -H "Authorization: Bearer $SCHEDULER_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "task_name": "my_custom_task",
+    "service": "scheduler",
+    "executor": "my_executor",
+    "priority": 50,
+    "minute": 0,
+    "hour": 2,
+    "config": {"param": "value"}
+  }'
+```
+
+## Current Tasks
+
+### Active Schedules
+
+| Task | Priority | Schedule | Description |
+|------|----------|----------|-------------|
+| `test_example_task` | 50 | Every minute | Test task (can be disabled) |
+| `backup_docker_configs_daily` | 20 | Daily 03:05 | Backup Docker configs |
+| `sync_fastapi_docs_monthly` | 60 | 11th @ 04:00 | Sync FastAPI docs to Gitea |
+| `sync_ollama_docs_monthly` | 60 | 12th @ 04:00 | Sync Ollama docs to Gitea |
+
+### Managing Tasks
+
+```bash
+# Disable test task
+curl -X PUT http://localhost:8090/tasks/test_example_task \
+  -H "Authorization: Bearer $SCHEDULER_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{"enabled": false}'
+
+# View execution history
+curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
+  http://localhost:8090/executions?limit=10
+```
+
+## Database Schema
+
+### scheduled_tasks
+
+Stores task definitions (templates for execution).
+
+```sql
+- id (serial)
+- task_name (varchar, unique)
+- service (varchar) - owner service
+- executor (varchar) - executor module name
+- priority (integer) - 1-100
+- minute, hour, day_of_month, month, day_of_week (integer) - schedule
+- enabled (boolean)
+- description (text)
+- config (jsonb) - executor configuration
+- last_run, last_status, last_duration_seconds - tracking
+- retry_count, max_retries, timeout_seconds - execution control
+- created_at, updated_at, created_by - metadata
+```
+
+### task_executions
+
+Stores individual execution records (audit trail).
+
+```sql
+- id (serial)
+- task_id (integer) - references scheduled_tasks
+- task_name, service, executor, priority - snapshot
+- status (varchar) - pending, running, success, failed, timeout
+- triggered_by (varchar) - scheduler, manual, retry
+- triggered_at, started_at, completed_at - timestamps
+- duration_seconds (integer)
+- output (text) - success message
+- error (text) - error details
+- retry_count (integer)
+- metadata (jsonb)
+```
+
+## Testing
+
+### Running Tests
+
+```bash
+# All tests with coverage
+docker exec -w /app scheduler /venv/bin/pytest --cov=src --cov-report=term
+
+# Only unit tests (fast, no database)
+docker exec -w /app scheduler /venv/bin/pytest -m unit
+
+# Only integration tests (with database)
+docker exec -w /app scheduler /venv/bin/pytest -m integration
+
+# Only API tests
+docker exec -w /app scheduler /venv/bin/pytest -m api
+
+# Only executor tests
+docker exec -w /app scheduler /venv/bin/pytest -m executor
+
+# Specific test file
+docker exec -w /app scheduler /venv/bin/pytest tests/test_api.py -v
+
+# With detailed output
+docker exec -w /app scheduler /venv/bin/pytest -vv --tb=long
+```
+
+### Test Coverage
+
+**Current Coverage: 80%**
+
+```
+Module                          Coverage
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+config.py                       100% ✅
+example_executor.py             100% ✅
+main.py (API endpoints)          95% ✅
+doc_sync_executor.py             78% ✅
+executor.py (core logic)         71% ✅
+config_backup_executor.py        50% 📈
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+TOTAL                            80% 🎯
+```
+
+### Test Database
+
+Tests use a dedicated PostgreSQL database:
+- **Database**: `test_scheduler`
+- **User**: `test_scheduler_user`
+- **Location**: `postgres-shared` container
+
+Tests automatically:
+- Clean database before each test
+- Insert test fixtures
+- Verify database state
+- Rollback changes after test
+
+## Development
+
+### Project Structure
+
+```
+scheduler/
+├── src/
+│   ├── config.py              # Pydantic settings
+│   ├── main.py                # FastAPI app & endpoints
+│   ├── executors/             # Task executors
+│   │   ├── example_executor.py
+│   │   ├── doc_sync_executor.py
+│   │   └── config_backup_executor.py
+│   └── tasks/
+│       └── executor.py        # Task execution engine
+├── tests/
+│   ├── conftest.py            # Pytest fixtures
+│   ├── test_api.py            # API tests
+│   ├── test_*_executor.py     # Executor tests
+│   └── test_database_integration.py
+├── requirements.txt           # Python dependencies
+├── pytest.ini                 # Test configuration
+└── README.md                  # This file
+```
+
+### Adding Dependencies
+
+```bash
+# Add to requirements.txt with version pinning
+echo "new-package~=1.0.0" >> requirements.txt
+
+# Rebuild venv in container
+docker restart scheduler
+```
+
+### Debugging
+
+```bash
+# View logs
+docker logs scheduler --tail 100 -f
+
+# Check scheduler status
+curl http://localhost:8090/health
+
+# View database tasks
+docker exec postgres-shared psql -U scheduler_user -d scheduler \
+  -c "SELECT task_name, enabled, priority, last_status FROM scheduled_tasks;"
+
+# View execution history
+docker exec postgres-shared psql -U scheduler_user -d scheduler \
+  -c "SELECT task_name, status, duration_seconds, completed_at FROM task_executions ORDER BY id DESC LIMIT 10;"
+```
+
+### Environment Variables
+
+Required environment variables (set in `stacks/scheduler.yml`):
+
+```bash
+# Database
+POSTGRES_HOST=postgres-shared
+POSTGRES_PORT=5432
+POSTGRES_DB=scheduler
+POSTGRES_USER=scheduler_user
+POSTGRES_PASSWORD=${SCHEDULER_DB_PASSWORD}
+
+# API Security
+SCHEDULER_API_KEY=${SCHEDULER_API_KEY}
+
+# Gitea (for doc sync)
+GITEA_URL=http://gitea:3000
+GITEA_USER=librarian
+GITEA_PASSWORD=${GITEA_PASSWORD}
+
+# Redis (future use)
+REDIS_HOST=redis-shared
+REDIS_PORT=6379
+REDIS_DB=3
+```
+
+## Monitoring
+
+### Health Checks
+
+```bash
+# Container health
+docker ps | grep scheduler
+
+# API health
+curl http://localhost:8090/health
+
+# Scheduler running
+curl http://localhost:8090/stats | jq '.scheduler_running'
+```
+
+### Metrics
+
+```bash
+# System statistics
+curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
+  http://localhost:8090/stats | jq
+
+# Recent executions
+curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
+  "http://localhost:8090/executions?limit=20" | jq
+
+# Failed tasks in last 24h
+curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
+  "http://localhost:8090/executions?status=failed" | jq
+```
+
+### Common Issues
+
+**Task not running?**
+1. Check if task is enabled: `GET /tasks/{name}`
+2. Verify schedule matches current time
+3. Check execution history for errors: `GET /executions?task_name={name}`
+4. View scheduler logs: `docker logs scheduler`
+
+**Task timing out?**
+- Increase `timeout_seconds` in task configuration
+- Check executor implementation for long-running operations
+- Consider breaking into smaller tasks
+
+**Database connection errors?**
+- Verify postgres-shared container is running
+- Check database credentials in environment
+- Test connection: `docker exec scheduler psql -h postgres-shared -U scheduler_user -d scheduler`
+
+## API Reference
+
+Full OpenAPI documentation available at: `http://localhost:8090/docs`
+
+### Response Codes
+
+- `200` - Success
+- `400` - Bad request (invalid data)
+- `401` - Missing API key
+- `403` - Invalid API key
+- `404` - Resource not found
+- `500` - Internal server error
+
+### Pagination
+
+Use `limit` parameter for controlling result count:
+
+```bash
+GET /executions?limit=50  # Default: 20, Max: 100
+```
+
+### Filtering
+
+Most list endpoints support filtering:
+
+```bash
+GET /tasks?enabled=true&service=scheduler
+GET /executions?task_name=my_task&status=success&limit=10
+```
+
+## Security
+
+- **API Key Authentication**: Required for all protected endpoints
+- **Network Isolation**: Runs on `docker-dataplane` network
+- **Database Credentials**: Stored in environment variables
+- **Gitea Tokens**: Used instead of passwords for Git operations
+- **Resource Limits**: CPU and memory limits in docker-compose
+
+## Performance
+
+- **Concurrent Execution**: Max 5 tasks simultaneously
+- **Minute-based Processing**: Lightweight scheduler tick every minute
+- **Priority Queue**: Higher priority tasks execute first
+- **Database Indexes**: Optimized queries for task selection
+- **Connection Pooling**: Reuses database connections
+
+## License
+
+Part of Portainer Core infrastructure.
+
+## Support
+
+For issues or questions:
+1. Check logs: `docker logs scheduler`
+2. View health: `curl http://localhost:8090/health`
+3. Review execution history: `GET /executions`
+4. Check this README for common solutions

services/scheduler/src/tasks/__init__.py

@@ -0,0 +1 @@
+"""Task execution system for The Scheduler."""