scheduler/CHANGELOG.md

# Changelog

All notable changes to The Scheduler will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

## [Unreleased]

## [1.3.0] - 2026-08-08

### Added
- **Postgres Retention Executor** (`postgres_retention_executor.py`) — deletes rows
  past a retention window from a table on the shared Postgres server. Uses the
  Scheduler's own credentials with only the database name overridden, so the target
  database grants `scheduler_user` SELECT and DELETE on the table.
- **Docker Prune Executor** (`docker_prune_executor.py`) — scheduled reclaim of Docker
  disk usage. Build cache and dangling images are pruned by default; unused images and
  volumes are opt-in, since volume pruning also removes volumes belonging to stopped
  containers.

### Fixed
- `POST /tasks` returned HTTP 500 after successfully creating the task. The response
  model declared `created_at`/`updated_at` as strings while the database returns
  timestamps, so every create looked like a failure and retrying hit a duplicate-key
  error.

### Changed
- `TASK_REGISTRATION.md` now lists the executors that exist. It previously advertised
  `shell`, `python` and `docker` executors that were never implemented.

## [1.2.0] - 2026-03-30

### Added
- **GCS Backup Executor** (`gcs_backup_executor.py`) — offsite backup to Google Cloud
  Storage.

## [1.1.3] - 2026-01-08

### Changed
- Test release to validate CI/CD auto-deploy workflow

## [1.1.2] - 2026-01-03

### Fixed
- CI: Use curl for release creation (release-action requires Go)

## [1.1.1] - 2026-01-03

### Changed
- CI: Auto-create Gitea release on version tag push (v*) instead of manual release trigger

## [1.1.0] - 2025-12-14

### Added
- **Gitea Release Cleanup Executor** (`gitea_release_cleanup_executor.py`)
  - Automatically cleans up old releases across all Gitea repositories
  - Configurable retention count (default: 5 releases per repo)
  - Repository exclusion list support
  - Dry-run mode for safe testing
  - Designed to run before Watchtower to prevent image tag accumulation
- **GITEA_TOKEN setting** in config for API token authentication (separate from password)

## [1.0.4] - 2025-12-14

### Fixed
- CI/CD: Correct Watchtower port (8080)

## [1.0.3] - 2025-12-14

### Added
- CI/CD: Trigger Watchtower update after successful Docker build

## [1.0.2] - 2025-12-14

### Fixed
- Removed unused `setup_database.sql` from Dockerfile (database schema managed externally)

## [1.0.1] - 2025-12-14

### Changed
- **Version tracking now uses pyproject.toml** as single source of truth
  - Added `pyproject.toml` with project metadata and dependencies
  - `config.py` reads version from pyproject.toml using `tomllib`
  - FastAPI app title and version dynamically loaded from config
  - Health endpoint now includes version in response
  - Dockerfile updated to include pyproject.toml

## [1.0.0] - 2025-12-07

### Added

#### Core Scheduling System
- **Hybrid APScheduler + PostgreSQL architecture** for minute-based task scheduling
  - Single scheduler job runs every minute
  - Queries database for tasks scheduled for current minute
  - Executes up to 5 tasks concurrently by priority
  - Priority queue system (1-100, lower = higher priority)
- **Cron-like scheduling** with wildcard support (`-1` = any)
  - Supports minute, hour, day_of_month, month, day_of_week patterns
  - Flexible scheduling from every-minute to specific dates
- **Task execution tracking** with full audit trail
  - Database tables: `scheduled_tasks` (definitions) and `task_executions` (history)
  - Tracks status, duration, output, errors, and retry attempts
  - Execution metadata stored as JSONB

#### REST API
- **Full CRUD API** for task management with FastAPI
  - `POST /tasks` - Create new scheduled task
  - `GET /tasks` - List all tasks with filtering (service, enabled)
  - `GET /tasks/{name}` - Get task details
  - `PUT /tasks/{name}` - Update task configuration
  - `DELETE /tasks/{name}` - Remove task
  - `POST /tasks/{name}/trigger` - Manually trigger task execution
- **Execution history endpoints**
  - `GET /executions` - Query execution history
  - Filter by task_name, status, service
  - Pagination support (limit parameter)
- **System monitoring endpoints**
  - `GET /health` - Health check
  - `GET /stats` - System statistics (enabled tasks, running tasks, 24h execution counts)
- **API Key authentication** (Bearer token) for all protected endpoints
- **OpenAPI documentation** at `/docs`

#### Task Executors
- **Example Executor** (`example_executor.py`)
  - Simple test implementation with configurable message and delay
  - Demonstrates executor pattern
- **Documentation Sync Executor** (`doc_sync_executor.py`)
  - Mirrors documentation from upstream Git repositories to Gitea
  - Supports full repository mirroring or selective path syncing
  - Creates date-tagged snapshots (YYYY-MM-DD format)
  - Generates `.SYNC_INFO.md` with sync metadata
  - Configurable upstream repo, paths, branch, and Gitea destination
- **Config Backup Executor** (`config_backup_executor.py`)
  - Backs up Docker configurations and data directories
  - Supports multiple source paths with exclusion patterns
  - Optional compression (tar.gz)
  - Retention policy (days-based cleanup)
  - Creates timestamped backups

#### Pre-configured Tasks
- **FastAPI Documentation Sync** (monthly on 11th at 04:00)
  - Syncs entire FastAPI repository to `library/docs-fastapi`
  - Priority: 60 (maintenance)
- **Ollama Documentation Sync** (monthly on 12th at 04:00)
  - Syncs only `/docs` folder from Ollama repository to `library/docs-ollama`
  - Priority: 60 (maintenance)
- **Docker Config Backup** (daily at 03:05)
  - Backs up Docker data and configurations
  - Priority: 20 (user task)
  - 30-day retention
- **Example Test Task** (every minute, can be disabled)
  - Test task for validation
  - Priority: 50 (maintenance)

#### Testing Infrastructure
- **Comprehensive test suite** with 80% code coverage
  - 85 total tests across multiple test files
  - Pytest configuration with markers (unit, integration, api, executor)
  - Coverage reporting with pytest-cov
- **Test categories**:
  - Unit tests: Fast tests with mocked dependencies
  - API tests: Comprehensive endpoint testing (24 tests)
  - Executor tests: Task executor validation
  - Integration tests: Real database operations
- **Test database setup**
  - Dedicated `test_scheduler` database on postgres-shared
  - Automatic schema creation and cleanup
  - Database fixtures for clean test state
  - Test user: `test_scheduler_user`
- **Test fixtures** (conftest.py)
  - Mock database connections
  - Mock scheduler and executor
  - Sample task data
  - Authentication headers
  - Clean database state management

#### Documentation
- **Comprehensive README.md** (500+ lines)
  - Architecture overview with ASCII diagram
  - Quick start guide
  - Complete API reference with curl examples
  - Task scheduling patterns and examples
  - Executor development guide
  - Testing guide with coverage metrics
  - Development and debugging information
  - Security and performance notes
- **Test database setup guide** (test_database_setup.sql)
  - SQL script for creating test environment
  - Schema matching production
  - Test data fixtures

#### Configuration
- **Pydantic Settings** for environment-based configuration
  - PostgreSQL connection settings
  - Redis connection (for future use)
  - Gitea authentication
  - API key configuration
  - Computed properties (database_url, redis_url)
- **Docker stack configuration** (stacks/scheduler.yml)
  - Virtual environment setup on startup
  - Git configuration for librarian user
  - Health checks
  - Network isolation (docker-dataplane)
  - Resource limits

### Technical Details

#### Database Schema
```sql
scheduled_tasks:
  - Task definitions/templates
  - Scheduling configuration (minute/hour/day patterns)
  - Priority, enabled status, retry settings
  - Task configuration as JSONB
  - Execution tracking fields

task_executions:
  - Individual execution records
  - Status tracking (pending, running, success, failed, timeout)
  - Duration and timestamp tracking
  - Output and error details
  - Metadata as JSONB
```

#### Performance
- Minute-based processing with lightweight scheduler ticks
- Connection pooling for database efficiency
- Database indexes for optimized task queries
- Concurrent execution with configurable limit (default: 5)
- Priority-based execution order

#### Security
- API key authentication required for protected endpoints
- Network isolation on docker-dataplane
- Environment variable-based secrets
- Gitea token authentication for git operations
- Database credentials in environment

### Dependencies
- FastAPI (web framework)
- APScheduler (task scheduling)
- psycopg2-binary (PostgreSQL driver)
- Pydantic (configuration management)
- pytest + pytest-asyncio + pytest-cov (testing)
- GitPython (git operations)

### Coverage Metrics
```
Module                          Coverage
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
config.py                       100% ✅
example_executor.py             100% ✅
main.py (API endpoints)          95% ✅
doc_sync_executor.py             78% ✅
executor.py (core logic)         71% ✅
config_backup_executor.py        50% 📈
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
TOTAL                            80% 🎯
```

## [Unreleased]

### Planned
- Redis integration for distributed locking
- Webhook notifications for task completion
- Task dependencies (run task B after task A succeeds)
- Task groups and tags
- More executors (database backup, log rotation, etc.)
- Web UI for task management
- Metrics export (Prometheus)
- Advanced scheduling (last business day of month, etc.)