scheduler/CHANGELOG.md

Changelog

All notable changes to The Scheduler will be documented in this file.

The format is based on Keep a Changelog.

[1.0.4] - 2025-12-14

Fixed

• CI/CD: Correct Watchtower port (8080)

[1.0.3] - 2025-12-14

Added

• CI/CD: Trigger Watchtower update after successful Docker build

[1.0.2] - 2025-12-14

Fixed

• Removed unused setup_database.sql from Dockerfile (database schema managed externally)

[1.0.1] - 2025-12-14

Changed

• Version tracking now uses pyproject.toml as single source of truth
  • Added pyproject.toml with project metadata and dependencies
  • config.py reads version from pyproject.toml using tomllib
  • FastAPI app title and version dynamically loaded from config
  • Health endpoint now includes version in response
  • Dockerfile updated to include pyproject.toml

[1.0.0] - 2025-12-07

Added

Core Scheduling System

• Hybrid APScheduler + PostgreSQL architecture for minute-based task scheduling
  • Single scheduler job runs every minute
  • Queries database for tasks scheduled for current minute
  • Executes up to 5 tasks concurrently by priority
  • Priority queue system (1-100, lower = higher priority)
• Cron-like scheduling with wildcard support (-1 = any)
  • Supports minute, hour, day_of_month, month, day_of_week patterns
  • Flexible scheduling from every-minute to specific dates
• Task execution tracking with full audit trail
  • Database tables: scheduled_tasks (definitions) and task_executions (history)
  • Tracks status, duration, output, errors, and retry attempts
  • Execution metadata stored as JSONB

REST API

• Full CRUD API for task management with FastAPI
  • POST /tasks - Create new scheduled task
  • GET /tasks - List all tasks with filtering (service, enabled)
  • GET /tasks/{name} - Get task details
  • PUT /tasks/{name} - Update task configuration
  • DELETE /tasks/{name} - Remove task
  • POST /tasks/{name}/trigger - Manually trigger task execution
• Execution history endpoints
  • GET /executions - Query execution history
  • Filter by task_name, status, service
  • Pagination support (limit parameter)
• System monitoring endpoints
  • GET /health - Health check
  • GET /stats - System statistics (enabled tasks, running tasks, 24h execution counts)
• API Key authentication (Bearer token) for all protected endpoints
• OpenAPI documentation at /docs

Task Executors

• Example Executor (example_executor.py)
  • Simple test implementation with configurable message and delay
  • Demonstrates executor pattern
• Documentation Sync Executor (doc_sync_executor.py)
  • Mirrors documentation from upstream Git repositories to Gitea
  • Supports full repository mirroring or selective path syncing
  • Creates date-tagged snapshots (YYYY-MM-DD format)
  • Generates .SYNC_INFO.md with sync metadata
  • Configurable upstream repo, paths, branch, and Gitea destination
• Config Backup Executor (config_backup_executor.py)
  • Backs up Docker configurations and data directories
  • Supports multiple source paths with exclusion patterns
  • Optional compression (tar.gz)
  • Retention policy (days-based cleanup)
  • Creates timestamped backups

Pre-configured Tasks

• FastAPI Documentation Sync (monthly on 11th at 04:00)
  • Syncs entire FastAPI repository to library/docs-fastapi
  • Priority: 60 (maintenance)
• Ollama Documentation Sync (monthly on 12th at 04:00)
  • Syncs only /docs folder from Ollama repository to library/docs-ollama
  • Priority: 60 (maintenance)
• Docker Config Backup (daily at 03:05)
  • Backs up Docker data and configurations
  • Priority: 20 (user task)
  • 30-day retention
• Example Test Task (every minute, can be disabled)
  • Test task for validation
  • Priority: 50 (maintenance)

Testing Infrastructure

• Comprehensive test suite with 80% code coverage
  • 85 total tests across multiple test files
  • Pytest configuration with markers (unit, integration, api, executor)
  • Coverage reporting with pytest-cov
• Test categories:
  • Unit tests: Fast tests with mocked dependencies
  • API tests: Comprehensive endpoint testing (24 tests)
  • Executor tests: Task executor validation
  • Integration tests: Real database operations
• Test database setup
  • Dedicated test_scheduler database on postgres-shared
  • Automatic schema creation and cleanup
  • Database fixtures for clean test state
  • Test user: test_scheduler_user
• Test fixtures (conftest.py)
  • Mock database connections
  • Mock scheduler and executor
  • Sample task data
  • Authentication headers
  • Clean database state management

Documentation

• Comprehensive README.md (500+ lines)
  • Architecture overview with ASCII diagram
  • Quick start guide
  • Complete API reference with curl examples
  • Task scheduling patterns and examples
  • Executor development guide
  • Testing guide with coverage metrics
  • Development and debugging information
  • Security and performance notes
• Test database setup guide (test_database_setup.sql)
  • SQL script for creating test environment
  • Schema matching production
  • Test data fixtures

Configuration

• Pydantic Settings for environment-based configuration
  • PostgreSQL connection settings
  • Redis connection (for future use)
  • Gitea authentication
  • API key configuration
  • Computed properties (database_url, redis_url)
• Docker stack configuration (stacks/scheduler.yml)
  • Virtual environment setup on startup
  • Git configuration for librarian user
  • Health checks
  • Network isolation (docker-dataplane)
  • Resource limits

Technical Details

Database Schema

scheduled_tasks:
  - Task definitions/templates
  - Scheduling configuration (minute/hour/day patterns)
  - Priority, enabled status, retry settings
  - Task configuration as JSONB
  - Execution tracking fields

task_executions:
  - Individual execution records
  - Status tracking (pending, running, success, failed, timeout)
  - Duration and timestamp tracking
  - Output and error details
  - Metadata as JSONB

Performance

• Minute-based processing with lightweight scheduler ticks
• Connection pooling for database efficiency
• Database indexes for optimized task queries
• Concurrent execution with configurable limit (default: 5)
• Priority-based execution order

Security

• API key authentication required for protected endpoints
• Network isolation on docker-dataplane
• Environment variable-based secrets
• Gitea token authentication for git operations
• Database credentials in environment

Dependencies

• FastAPI (web framework)
• APScheduler (task scheduling)
• psycopg2-binary (PostgreSQL driver)
• Pydantic (configuration management)
• pytest + pytest-asyncio + pytest-cov (testing)
• GitPython (git operations)

Coverage Metrics

Module                          Coverage
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
config.py                       100% ✅
example_executor.py             100% ✅
main.py (API endpoints)          95% ✅
doc_sync_executor.py             78% ✅
executor.py (core logic)         71% ✅
config_backup_executor.py        50% 📈
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
TOTAL                            80% 🎯

[Unreleased]

Planned

• Redis integration for distributed locking
• Webhook notifications for task completion
• Task dependencies (run task B after task A succeeds)
• Task groups and tags
• More executors (database backup, log rotation, etc.)
• Web UI for task management
• Metrics export (Prometheus)
• Advanced scheduling (last business day of month, etc.)