scheduler/CHANGELOG.md

Changelog

All notable changes to The Scheduler will be documented in this file.

The format is based on Keep a Changelog.

[Unreleased]

[1.6.0] - 2026-08-11

Fixed

• A restart mid-task no longer unschedules that task forever. An execution row left running excluded its task from every future minute, silently; startup now releases them.
• Timeouts are recorded as timeout instead of a generic failure. The status existed but was unreachable, so all 18,785 executions since December contain zero of them.

Added

• orphaned execution status — an execution whose process died, whose outcome is unknown. Distinct from failed, which asserts the work did not succeed.

[1.5.1] - 2026-08-11

Fixed

• Backups no longer freeze the API. The config backup ran its ~21 minutes of tarring on the event loop, so the whole service was unreachable 03:05-03:25 nightly; it now runs in a worker thread. Health reports moved off the loop too.
• The health report's permission diagnostic prints the database's own message instead of asserting a cause. It claimed the user lacked INSERT on check_history when that grant was present and the missing one was USAGE on the sequence behind its serial id.

Notes

• Requires GRANT USAGE ON SEQUENCE check_history_id_seq TO scheduler_user, applied 2026-08-11. The table grant alone does not permit the insert.

[1.5.0] - 2026-08-11

Added

• Backup executors report their own outcome to the homelab health record — one row in check_history per run, success or failure. Replaces a monitor that inferred backup health from file age and could not tell a failed backup from one that had not run yet.

Notes

• Requires GRANT INSERT ON check_history TO scheduler_user in the sysmon database, applied 2026-08-11. Without it the report is refused, logged, and skipped; the backup itself is unaffected.

[1.4.0] - 2026-08-08

Added

• Portainer Backup Executor (portainer_backup_executor.py) — archives Portainer's own state through its /api/backup endpoint. Portainer's BoltDB lives in a Docker volume that the daily config backup does not cover, so losing that volume would take every stack definition with it. Uses the API rather than tarring the live volume, and rejects a 200 whose body is not a readable archive.

[1.3.0] - 2026-08-08

Added

• Postgres Retention Executor (postgres_retention_executor.py) — deletes rows past a retention window from a table on the shared Postgres server. Uses the Scheduler's own credentials with only the database name overridden, so the target database grants scheduler_user SELECT and DELETE on the table.
• Docker Prune Executor (docker_prune_executor.py) — scheduled reclaim of Docker disk usage. Build cache and dangling images are pruned by default; unused images and volumes are opt-in, since volume pruning also removes volumes belonging to stopped containers.

Fixed

• POST /tasks returned HTTP 500 after successfully creating the task. The response model declared created_at/updated_at as strings while the database returns timestamps, so every create looked like a failure and retrying hit a duplicate-key error.

Changed

• TASK_REGISTRATION.md now lists the executors that exist. It previously advertised shell, python and docker executors that were never implemented.

[1.2.0] - 2026-03-30

Added

• GCS Backup Executor (gcs_backup_executor.py) — offsite backup to Google Cloud Storage.

[1.1.3] - 2026-01-08

Changed

• Test release to validate CI/CD auto-deploy workflow

[1.1.2] - 2026-01-03

Fixed

• CI: Use curl for release creation (release-action requires Go)

[1.1.1] - 2026-01-03

Changed

• CI: Auto-create Gitea release on version tag push (v*) instead of manual release trigger

[1.1.0] - 2025-12-14

Added

• Gitea Release Cleanup Executor (gitea_release_cleanup_executor.py)
  • Automatically cleans up old releases across all Gitea repositories
  • Configurable retention count (default: 5 releases per repo)
  • Repository exclusion list support
  • Dry-run mode for safe testing
  • Designed to run before Watchtower to prevent image tag accumulation
• GITEA_TOKEN setting in config for API token authentication (separate from password)

[1.0.4] - 2025-12-14

Fixed

• CI/CD: Correct Watchtower port (8080)

[1.0.3] - 2025-12-14

Added

• CI/CD: Trigger Watchtower update after successful Docker build

[1.0.2] - 2025-12-14

Fixed

• Removed unused setup_database.sql from Dockerfile (database schema managed externally)

[1.0.1] - 2025-12-14

Changed

• Version tracking now uses pyproject.toml as single source of truth
  • Added pyproject.toml with project metadata and dependencies
  • config.py reads version from pyproject.toml using tomllib
  • FastAPI app title and version dynamically loaded from config
  • Health endpoint now includes version in response
  • Dockerfile updated to include pyproject.toml

[1.0.0] - 2025-12-07

Added

Core Scheduling System

• Hybrid APScheduler + PostgreSQL architecture for minute-based task scheduling
  • Single scheduler job runs every minute
  • Queries database for tasks scheduled for current minute
  • Executes up to 5 tasks concurrently by priority
  • Priority queue system (1-100, lower = higher priority)
• Cron-like scheduling with wildcard support (-1 = any)
  • Supports minute, hour, day_of_month, month, day_of_week patterns
  • Flexible scheduling from every-minute to specific dates
• Task execution tracking with full audit trail
  • Database tables: scheduled_tasks (definitions) and task_executions (history)
  • Tracks status, duration, output, errors, and retry attempts
  • Execution metadata stored as JSONB

REST API

• Full CRUD API for task management with FastAPI
  • POST /tasks - Create new scheduled task
  • GET /tasks - List all tasks with filtering (service, enabled)
  • GET /tasks/{name} - Get task details
  • PUT /tasks/{name} - Update task configuration
  • DELETE /tasks/{name} - Remove task
  • POST /tasks/{name}/trigger - Manually trigger task execution
• Execution history endpoints
  • GET /executions - Query execution history
  • Filter by task_name, status, service
  • Pagination support (limit parameter)
• System monitoring endpoints
  • GET /health - Health check
  • GET /stats - System statistics (enabled tasks, running tasks, 24h execution counts)
• API Key authentication (Bearer token) for all protected endpoints
• OpenAPI documentation at /docs

Task Executors

• Example Executor (example_executor.py)
  • Simple test implementation with configurable message and delay
  • Demonstrates executor pattern
• Documentation Sync Executor (doc_sync_executor.py)
  • Mirrors documentation from upstream Git repositories to Gitea
  • Supports full repository mirroring or selective path syncing
  • Creates date-tagged snapshots (YYYY-MM-DD format)
  • Generates .SYNC_INFO.md with sync metadata
  • Configurable upstream repo, paths, branch, and Gitea destination
• Config Backup Executor (config_backup_executor.py)
  • Backs up Docker configurations and data directories
  • Supports multiple source paths with exclusion patterns
  • Optional compression (tar.gz)
  • Retention policy (days-based cleanup)
  • Creates timestamped backups

Pre-configured Tasks

• FastAPI Documentation Sync (monthly on 11th at 04:00)
  • Syncs entire FastAPI repository to library/docs-fastapi
  • Priority: 60 (maintenance)
• Ollama Documentation Sync (monthly on 12th at 04:00)
  • Syncs only /docs folder from Ollama repository to library/docs-ollama
  • Priority: 60 (maintenance)
• Docker Config Backup (daily at 03:05)
  • Backs up Docker data and configurations
  • Priority: 20 (user task)
  • 30-day retention
• Example Test Task (every minute, can be disabled)
  • Test task for validation
  • Priority: 50 (maintenance)

Testing Infrastructure

• Comprehensive test suite with 80% code coverage
  • 85 total tests across multiple test files
  • Pytest configuration with markers (unit, integration, api, executor)
  • Coverage reporting with pytest-cov
• Test categories:
  • Unit tests: Fast tests with mocked dependencies
  • API tests: Comprehensive endpoint testing (24 tests)
  • Executor tests: Task executor validation
  • Integration tests: Real database operations
• Test database setup
  • Dedicated test_scheduler database on postgres-shared
  • Automatic schema creation and cleanup
  • Database fixtures for clean test state
  • Test user: test_scheduler_user
• Test fixtures (conftest.py)
  • Mock database connections
  • Mock scheduler and executor
  • Sample task data
  • Authentication headers
  • Clean database state management

Documentation

• Comprehensive README.md (500+ lines)
  • Architecture overview with ASCII diagram
  • Quick start guide
  • Complete API reference with curl examples
  • Task scheduling patterns and examples
  • Executor development guide
  • Testing guide with coverage metrics
  • Development and debugging information
  • Security and performance notes
• Test database setup guide (test_database_setup.sql)
  • SQL script for creating test environment
  • Schema matching production
  • Test data fixtures

Configuration

• Pydantic Settings for environment-based configuration
  • PostgreSQL connection settings
  • Redis connection (for future use)
  • Gitea authentication
  • API key configuration
  • Computed properties (database_url, redis_url)
• Docker stack configuration (stacks/scheduler.yml)
  • Virtual environment setup on startup
  • Git configuration for librarian user
  • Health checks
  • Network isolation (docker-dataplane)
  • Resource limits

Technical Details

Database Schema

scheduled_tasks:
  - Task definitions/templates
  - Scheduling configuration (minute/hour/day patterns)
  - Priority, enabled status, retry settings
  - Task configuration as JSONB
  - Execution tracking fields

task_executions:
  - Individual execution records
  - Status tracking (pending, running, success, failed, timeout)
  - Duration and timestamp tracking
  - Output and error details
  - Metadata as JSONB

Performance

• Minute-based processing with lightweight scheduler ticks
• Connection pooling for database efficiency
• Database indexes for optimized task queries
• Concurrent execution with configurable limit (default: 5)
• Priority-based execution order

Security

• API key authentication required for protected endpoints
• Network isolation on docker-dataplane
• Environment variable-based secrets
• Gitea token authentication for git operations
• Database credentials in environment

Dependencies

• FastAPI (web framework)
• APScheduler (task scheduling)
• psycopg2-binary (PostgreSQL driver)
• Pydantic (configuration management)
• pytest + pytest-asyncio + pytest-cov (testing)
• GitPython (git operations)

Coverage Metrics

Module                          Coverage
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
config.py                       100% ✅
example_executor.py             100% ✅
main.py (API endpoints)          95% ✅
doc_sync_executor.py             78% ✅
executor.py (core logic)         71% ✅
config_backup_executor.py        50% 📈
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
TOTAL                            80% 🎯

Planned

• Redis integration for distributed locking
• Webhook notifications for task completion
• Task dependencies (run task B after task A succeeds)
• Task groups and tags
• More executors (database backup, log rotation, etc.)
• Web UI for task management
• Metrics export (Prometheus)
• Advanced scheduling (last business day of month, etc.)