12 Commits
Author SHA1 Message Date
jpmschweitzerandClaude Opus 4.5 ec200c66ba fix(ci): Use curl for release creation
Build and Push / release (push) Successful in 3s
Build and Push / build (push) Successful in 1m10s
The release-action requires Go which isn't in the runner image.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 20:33:47 +01:00
jpmschweitzerandClaude Opus 4.5 2a6a91ed67 chore: Bump version to 1.1.1
Build and Push / release (push) Failing after 7s
Build and Push / build (push) Has been skipped
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 20:30:12 +01:00
jpmschweitzerandClaude Opus 4.5 f334753918 ci: Auto-create release on version tag push
Change workflow trigger from manual release to tag push (v*).
Adds release job that creates Gitea release before building.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 20:28:55 +01:00
jpmschweitzer 8a6bdb9547 update to AGENTS.md 2025-12-25 10:05:58 +01:00
jpmschweitzerandClaude Opus 4.5 424c94d79a feat: Add Gitea release cleanup executor
Build and Push / build (release) Successful in 27s
- Add gitea_release_cleanup_executor for automated release cleanup
- Add GITEA_TOKEN setting for API token authentication
- Configurable retention count, repo exclusions, and dry-run mode
- Designed to run daily before Watchtower (3 AM)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-14 13:25:02 +01:00
jpmschweitzerandClaude Opus 4.5 8f1a4402e7 fix(ci): Correct Watchtower port
Build and Push / build (release) Successful in 28s
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-14 12:57:08 +01:00
jpmschweitzerandClaude Opus 4.5 9ae10af1df ci: Add Watchtower update trigger after build
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-14 12:45:06 +01:00
jpmschweitzerandClaude Opus 4.5 58ac4054a9 chore: Bump version to 1.0.2
Build and Push / build (release) Successful in 11s
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-14 12:17:24 +01:00
jpmschweitzerandClaude Opus 4.5 feb7c9e561 fix: Remove setup_database.sql from Dockerfile
Database schema is managed externally - the SQL file was only copied
but never used by the application.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-14 12:16:27 +01:00
jpmschweitzerandClaude Opus 4.5 6f8e6f1c1f feat: Add version tracking via pyproject.toml
Build and Push / build (release) Failing after 10s
- Add pyproject.toml as single source of truth for version and metadata
- Update config.py to read version from pyproject.toml using tomllib
- FastAPI app now loads title and version dynamically from config
- Health endpoint now includes version in response
- Dockerfile updated to include pyproject.toml

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-14 10:58:28 +01:00
jpmschweitzer 030da66a78 CI/CD network issue 2025-12-14 10:51:24 +01:00
jpmschweitzer 48e82e3a2e cleanup 2025-12-11 19:49:27 +01:00
9 changed files with 426 additions and 130 deletions
+24 -5
View File
@@ -1,19 +1,32 @@
name: Build and Push
on:
release:
types: [published]
push:
tags:
- 'v*'
jobs:
release:
runs-on: ubuntu-latest
steps:
- name: Create Gitea Release
run: |
curl -sf -X POST \
-H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \
-H "Content-Type: application/json" \
-d '{"tag_name": "${{ github.ref_name }}", "name": "Release ${{ github.ref_name }}", "body": "Automated release for ${{ github.ref_name }}"}' \
"${{ github.server_url }}/api/v1/repos/${{ github.repository }}/releases"
build:
runs-on: ubuntu-latest
needs: release
steps:
- uses: actions/checkout@v4
- name: Login to Gitea Registry
uses: docker/login-action@v3
with:
registry: git.schweitz.net
registry: git.schweitz.internal
username: ${{ secrets.REGISTRY_USER }}
password: ${{ secrets.REGISTRY_PASSWORD }}
@@ -23,5 +36,11 @@ jobs:
context: .
push: true
tags: |
git.schweitz.net/jpmschweitzer/scheduler:latest
git.schweitz.net/jpmschweitzer/scheduler:${{ github.ref_name }}
git.schweitz.internal/jpmschweitzer/scheduler:latest
git.schweitz.internal/jpmschweitzer/scheduler:${{ github.ref_name }}
- name: Trigger Watchtower update
if: success()
run: |
curl -sf -H "Authorization: Bearer ${{ secrets.WATCHTOWER_TOKEN }}" \
http://watchtower:8080/v1/update
+72
View File
@@ -0,0 +1,72 @@
# AGENTS.md
> **Start every session by reading this file.**
> This file outlines the operational protocols, coding standards, and architectural decisions for this FastAPI project.
## 1. Agent Operational Protocols
### 🧠 Work Patterns (Plan-Act-Reflect)
* **Plan:** Before writing code, briefly outline your plan. Identify which files you will touch and what the side effects might be.
* **Act:** Execute the changes in small, atomic steps.
* **Reflect:** After coding, verify your work. Did you break existing tests? Did you add new tests?
### 🛡️ Git Discipline
* **NEVER commit to `main` or `master` directly.** Always create a feature branch: `feature/your-feature-name` or `fix/issue-description`.
* **Commit Messages:** Use the [Conventional Commits](https://www.conventionalcommits.org/) format.
* `feat: add user login endpoint`
* `fix: resolve database connection timeout`
* `refactor: split monolith dependency file`
* **Atomic Commits:** Keep commits small. One logical change = one commit.
### 📝 Changelog Maintenance
* **Update `CHANGELOG.md`** with every user-facing change.
* Format: `## [Unreleased] - YYYY-MM-DD` followed by `### Added`, `### Changed`, or `### Fixed`.
### 🚀 Release Flow
When changes are ready for deployment:
1. **Ask user if deploy cycle is desired **
2. **Update version** in `pyproject.toml`:
- Bug fixes: bump patch version (1.8.3 → 1.8.4)
- New features: bump minor version (1.8.4 → 1.9.0)
3. **Update CHANGELOG.md**:
- Move items from `[Unreleased]` to new version section
- Add release date: `## [1.8.4] - 2025-12-16`
4. **Commit and tag**:
```bash
git add -A
git commit -m "fix: description of changes"
git tag v1.8.4
git push origin main --tags
```
5. **CI/CD triggers automatically**:
- Gitea CI builds Docker image on new tag
- Watchtower pulls and deploys to production
- Verify deployment: `curl http://192.168.86.149:8000/health`
---
## 2. FastAPI Architecture & Best Practices
*Reference: [FastAPI Best Practices](https://github.com/zhanymkanov/fastapi-best-practices)*
### 📂 Project Structure (Directory-based, NOT File-type based)
Do **not** group files by type (e.g., one huge `routers` folder). Group by **domain/module** inside a `src/` directory.
**Correct Structure:**
```text
src/
├── auth/
│ ├── router.py # Endpoints
│ ├── schemas.py # Pydantic models
│ ├── service.py # Business logic (CRUD, etc.)
│ ├── dependencies.py# Module-specific dependencies
│ └── config.py # Module-specific settings
├── posts/
│ ├── router.py
│ └── ...
└── main.py # App entry point
+46
View File
@@ -4,6 +4,52 @@ All notable changes to The Scheduler will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
## [1.1.2] - 2026-01-03
### Fixed
- CI: Use curl for release creation (release-action requires Go)
## [1.1.1] - 2026-01-03
### Changed
- CI: Auto-create Gitea release on version tag push (v*) instead of manual release trigger
## [1.1.0] - 2025-12-14
### Added
- **Gitea Release Cleanup Executor** (`gitea_release_cleanup_executor.py`)
- Automatically cleans up old releases across all Gitea repositories
- Configurable retention count (default: 5 releases per repo)
- Repository exclusion list support
- Dry-run mode for safe testing
- Designed to run before Watchtower to prevent image tag accumulation
- **GITEA_TOKEN setting** in config for API token authentication (separate from password)
## [1.0.4] - 2025-12-14
### Fixed
- CI/CD: Correct Watchtower port (8080)
## [1.0.3] - 2025-12-14
### Added
- CI/CD: Trigger Watchtower update after successful Docker build
## [1.0.2] - 2025-12-14
### Fixed
- Removed unused `setup_database.sql` from Dockerfile (database schema managed externally)
## [1.0.1] - 2025-12-14
### Changed
- **Version tracking now uses pyproject.toml** as single source of truth
- Added `pyproject.toml` with project metadata and dependencies
- `config.py` reads version from pyproject.toml using `tomllib`
- FastAPI app title and version dynamically loaded from config
- Health endpoint now includes version in response
- Dockerfile updated to include pyproject.toml
## [1.0.0] - 2025-12-07
### Added
+1 -1
View File
@@ -13,7 +13,7 @@ RUN pip install --no-cache-dir -r requirements.txt
# Copy application
COPY src/ ./src/
COPY setup_database.sql .
COPY pyproject.toml .
# Configure git
RUN git config --global user.name "The Librarian" && \
+45
View File
@@ -0,0 +1,45 @@
[project]
name = "the-scheduler"
version = "1.1.2"
description = "System-wide maintenance orchestration - backups, doc mirroring, cleanup, task automation"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
# FastAPI ecosystem
"fastapi~=0.124.0",
"uvicorn[standard]~=0.38.0",
"pydantic~=2.12.5",
"pydantic-settings~=2.7.0",
# Task scheduling
"apscheduler~=3.11.1",
"sqlalchemy~=2.0.36",
# Database
"psycopg2-binary~=2.9.11",
"redis~=5.2.0",
# HTTP client
"httpx~=0.28.1",
# Web scraping (for doc mirroring)
"scrapy~=2.12.0",
"beautifulsoup4~=4.12.3",
"lxml~=5.1.0",
# Git operations
"gitpython~=3.1.43",
# Security/Auth
"python-jose[cryptography]~=3.3.0",
]
[project.optional-dependencies]
test = [
"pytest~=8.3.4",
"pytest-asyncio~=0.25.2",
"pytest-cov~=6.0.0",
"freezegun~=1.5.1",
]
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"
[tool.setuptools.packages.find]
where = ["."]
include = ["src*"]
-120
View File
@@ -1,120 +0,0 @@
-- The Scheduler Database Setup
-- Run this from postgres-shared container
-- Create database and user
CREATE DATABASE scheduler;
CREATE USER scheduler_user WITH PASSWORD 'a/Ph0NhC4pTDDjSpL6q/DtBI+z0nf43ijVHjTo1KrXc=';
GRANT ALL PRIVILEGES ON DATABASE scheduler TO scheduler_user;
-- Connect to the new database
\c scheduler
-- Grant schema permissions
GRANT ALL ON SCHEMA public TO scheduler_user;
-- Create scheduled_tasks table
CREATE TABLE scheduled_tasks (
id SERIAL PRIMARY KEY,
task_name VARCHAR(100) NOT NULL UNIQUE,
service VARCHAR(50) NOT NULL, -- Which service owns this task
executor VARCHAR(100) NOT NULL, -- Executor module to run
priority INTEGER NOT NULL DEFAULT 30, -- Lower = higher priority (1-100)
-- Scheduling (supports wildcards: -1 = any)
minute INTEGER DEFAULT -1, -- 0-59 or -1 (any)
hour INTEGER DEFAULT -1, -- 0-23 or -1 (any)
day_of_month INTEGER DEFAULT -1, -- 1-31 or -1 (any)
month INTEGER DEFAULT -1, -- 1-12 or -1 (any)
day_of_week INTEGER DEFAULT -1, -- 0-6 (0=Monday) or -1 (any)
enabled BOOLEAN DEFAULT true,
description TEXT,
config JSONB, -- Arguments for executor
-- Execution tracking
last_run TIMESTAMP,
last_status VARCHAR(20), -- success, failed, timeout
last_duration_seconds INTEGER,
retry_count INTEGER DEFAULT 0,
max_retries INTEGER DEFAULT 3,
timeout_seconds INTEGER DEFAULT 3600, -- 1 hour default
-- Metadata
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
created_by VARCHAR(50),
-- Constraints
CHECK (priority >= 1 AND priority <= 100),
CHECK (minute >= -1 AND minute <= 59),
CHECK (hour >= -1 AND hour <= 23),
CHECK (day_of_month >= -1 AND day_of_month <= 31),
CHECK (month >= -1 AND month <= 12),
CHECK (day_of_week >= -1 AND day_of_week <= 6)
);
-- Create task_executions table
CREATE TABLE task_executions (
id SERIAL PRIMARY KEY,
task_id INTEGER NOT NULL REFERENCES scheduled_tasks(id),
task_name VARCHAR(100) NOT NULL,
service VARCHAR(50) NOT NULL,
executor VARCHAR(100) NOT NULL,
priority INTEGER NOT NULL,
status VARCHAR(20) NOT NULL, -- pending, running, success, failed, timeout
triggered_by VARCHAR(50), -- scheduler, manual, retry
triggered_at TIMESTAMP DEFAULT NOW(),
started_at TIMESTAMP,
completed_at TIMESTAMP,
duration_seconds INTEGER,
output TEXT,
error TEXT,
retry_count INTEGER DEFAULT 0,
metadata JSONB
);
-- Create doc_sources table
CREATE TABLE doc_sources (
id SERIAL PRIMARY KEY,
project_name VARCHAR(100) NOT NULL UNIQUE,
current_version VARCHAR(50),
last_mirrored TIMESTAMP,
last_checked TIMESTAMP,
gitea_repo VARCHAR(200),
config JSONB
);
-- Indexes for scheduled_tasks
CREATE INDEX idx_tasks_enabled ON scheduled_tasks(enabled) WHERE enabled = true;
CREATE INDEX idx_tasks_priority ON scheduled_tasks(priority);
CREATE INDEX idx_tasks_service ON scheduled_tasks(service);
CREATE INDEX idx_tasks_schedule ON scheduled_tasks(minute, hour, day_of_month, month, day_of_week) WHERE enabled = true;
-- Indexes for task_executions
CREATE INDEX idx_executions_task_id ON task_executions(task_id);
CREATE INDEX idx_executions_task_name ON task_executions(task_name);
CREATE INDEX idx_executions_status ON task_executions(status);
CREATE INDEX idx_executions_triggered_at ON task_executions(triggered_at DESC);
CREATE INDEX idx_executions_service ON task_executions(service);
CREATE INDEX idx_executions_running ON task_executions(task_id) WHERE status = 'running';
-- Grant permissions
GRANT ALL ON ALL TABLES IN SCHEMA public TO scheduler_user;
GRANT ALL ON ALL SEQUENCES IN SCHEMA public TO scheduler_user;
-- Insert example test task
INSERT INTO scheduled_tasks
(task_name, service, executor, priority, minute, hour, description, config, created_by)
VALUES
('test_example_task', 'scheduler', 'example_executor', 50, -1, -1,
'Example task that runs every minute for testing',
'{"message": "Scheduler is working!", "delay_seconds": 2}'::jsonb,
'setup_script');
-- Verify tables created
\dt
-- Show the test task
SELECT task_name, service, priority, enabled, description FROM scheduled_tasks;
+23 -2
View File
@@ -2,17 +2,37 @@
Configuration management for The Scheduler.
Uses Pydantic BaseSettings for type-safe environment variable loading.
"""
import tomllib
from functools import lru_cache
from pathlib import Path
from pydantic_settings import BaseSettings
from pydantic import Field
def _get_version_from_pyproject() -> tuple[str, str, str]:
"""Read version info from pyproject.toml."""
pyproject_path = Path(__file__).parent.parent / "pyproject.toml"
if pyproject_path.exists():
with open(pyproject_path, "rb") as f:
data = tomllib.load(f)
project = data.get("project", {})
return (
project.get("name", "the-scheduler"),
project.get("version", "0.0.0"),
project.get("description", ""),
)
return ("the-scheduler", "0.0.0", "")
_PROJECT_NAME, _PROJECT_VERSION, _PROJECT_DESCRIPTION = _get_version_from_pyproject()
class Settings(BaseSettings):
"""Application settings loaded from environment variables."""
# Application
app_name: str = Field(default="The Scheduler", alias="APP_NAME")
app_version: str = Field(default="1.0.0", alias="APP_VERSION")
app_name: str = Field(default=_PROJECT_NAME, alias="APP_NAME")
app_version: str = Field(default=_PROJECT_VERSION, alias="APP_VERSION")
debug: bool = Field(default=False, alias="DEBUG")
host: str = Field(default="0.0.0.0", alias="HOST")
port: int = Field(default=8090, alias="PORT")
@@ -37,6 +57,7 @@ class Settings(BaseSettings):
gitea_url: str = Field(default="http://gitea:3000", alias="GITEA_URL")
gitea_user: str = Field(default="library", alias="GITEA_USER")
gitea_password: str = Field(default="", alias="GITEA_PASSWORD")
gitea_token: str = Field(default="", alias="GITEA_TOKEN") # API token with write:repository scope
gitea_ssh_host: str = Field(default="gitea", alias="GITEA_SSH_HOST")
gitea_ssh_port: int = Field(default=22, alias="GITEA_SSH_PORT")
@@ -0,0 +1,211 @@
"""
Gitea Release Cleanup Executor
Cleans up old releases across all Gitea repositories, keeping only the most recent
releases per repository. Designed to run before Watchtower to prevent accumulation
of old container image tags.
Config schema:
{
"keep_count": 5, # Number of releases to keep per repo (default: 5)
"exclude_repos": [], # Repository names to skip (default: [])
"dry_run": false # If true, only log what would be deleted (default: false)
}
Example config:
{
"keep_count": 5,
"exclude_repos": ["important-repo", "legacy-app"],
"dry_run": false
}
Required settings:
- GITEA_URL: Base URL of Gitea instance
- GITEA_TOKEN: API token with write:repository scope
"""
import logging
from typing import Any
import httpx
from src.config import Settings
logger = logging.getLogger(__name__)
# Default configuration values
DEFAULT_KEEP_COUNT = 5
DEFAULT_TIMEOUT = 60
API_BASE = "/api/v1"
async def execute(config: dict, settings: Settings) -> str:
"""
Clean up old releases across all accessible Gitea repositories.
Args:
config: Task configuration (see module docstring)
settings: Global scheduler settings
Returns:
Summary of cleanup actions taken
Raises:
ValueError: On configuration error
Exception: On API call failure
"""
# Validate settings
if not settings.gitea_url:
raise ValueError("GITEA_URL not configured")
if not settings.gitea_token:
raise ValueError("GITEA_TOKEN not configured - required for release deletion")
# Parse configuration
keep_count = config.get("keep_count", DEFAULT_KEEP_COUNT)
exclude_repos = config.get("exclude_repos", [])
dry_run = config.get("dry_run", False)
if keep_count < 1:
raise ValueError(f"keep_count must be at least 1, got {keep_count}")
base_url = settings.gitea_url.rstrip("/")
headers = {"Authorization": f"token {settings.gitea_token}"}
stats = {
"repos_scanned": 0,
"repos_with_releases": 0,
"releases_deleted": 0,
"releases_skipped": 0,
"errors": [],
}
mode = "DRY RUN" if dry_run else "LIVE"
logger.info(f"Starting Gitea release cleanup ({mode}): keeping {keep_count} releases per repo")
async with httpx.AsyncClient(timeout=DEFAULT_TIMEOUT) as client:
# Fetch all repositories
repos = await _fetch_user_repos(client, base_url, headers)
logger.info(f"Found {len(repos)} repositories")
for repo in repos:
owner = repo["owner"]["login"]
name = repo["name"]
full_name = f"{owner}/{name}"
# Check exclusion list
if name in exclude_repos or full_name in exclude_repos:
logger.debug(f"Skipping excluded repo: {full_name}")
continue
stats["repos_scanned"] += 1
try:
# Fetch releases for this repo
releases = await _fetch_releases(client, base_url, headers, owner, name)
if not releases:
continue
stats["repos_with_releases"] += 1
# Determine which releases to delete (beyond keep_count)
to_delete = releases[keep_count:]
if not to_delete:
logger.debug(f"{full_name}: {len(releases)} releases, nothing to delete")
continue
logger.info(f"{full_name}: {len(releases)} releases, deleting {len(to_delete)}")
# Delete old releases
for release in to_delete:
release_id = release["id"]
tag_name = release["tag_name"]
if dry_run:
logger.info(f" [DRY RUN] Would delete: {tag_name} (id={release_id})")
stats["releases_skipped"] += 1
else:
try:
await _delete_release(client, base_url, headers, owner, name, release_id)
logger.info(f" Deleted: {tag_name}")
stats["releases_deleted"] += 1
except Exception as e:
error_msg = f"{full_name}/{tag_name}: {e}"
logger.warning(f" Failed to delete {tag_name}: {e}")
stats["errors"].append(error_msg)
except Exception as e:
error_msg = f"{full_name}: {e}"
logger.error(f"Error processing {full_name}: {e}")
stats["errors"].append(error_msg)
# Build summary
summary = _build_summary(stats, dry_run)
logger.info(f"Cleanup complete: {summary}")
return summary
async def _fetch_user_repos(
client: httpx.AsyncClient,
base_url: str,
headers: dict,
) -> list[dict[str, Any]]:
"""Fetch all repositories accessible to the authenticated user."""
url = f"{base_url}{API_BASE}/user/repos"
params = {"limit": 100}
response = await client.get(url, headers=headers, params=params)
response.raise_for_status()
return response.json()
async def _fetch_releases(
client: httpx.AsyncClient,
base_url: str,
headers: dict,
owner: str,
repo: str,
) -> list[dict[str, Any]]:
"""Fetch releases for a repository, sorted newest first (Gitea default)."""
url = f"{base_url}{API_BASE}/repos/{owner}/{repo}/releases"
params = {"limit": 100}
response = await client.get(url, headers=headers, params=params)
response.raise_for_status()
return response.json()
async def _delete_release(
client: httpx.AsyncClient,
base_url: str,
headers: dict,
owner: str,
repo: str,
release_id: int,
) -> None:
"""Delete a specific release."""
url = f"{base_url}{API_BASE}/repos/{owner}/{repo}/releases/{release_id}"
response = await client.delete(url, headers=headers)
response.raise_for_status()
def _build_summary(stats: dict, dry_run: bool) -> str:
"""Build a human-readable summary of the cleanup operation."""
parts = [
f"Scanned {stats['repos_scanned']} repos",
f"{stats['repos_with_releases']} with releases",
]
if dry_run:
parts.append(f"{stats['releases_skipped']} releases would be deleted")
else:
parts.append(f"{stats['releases_deleted']} releases deleted")
if stats["errors"]:
parts.append(f"{len(stats['errors'])} errors")
return ", ".join(parts)
+4 -2
View File
@@ -102,9 +102,10 @@ async def lifespan(app: FastAPI):
# FastAPI app
settings = get_settings()
app = FastAPI(
title="The Scheduler",
version="1.0.0",
title=settings.app_name,
version=settings.app_version,
description="System-wide maintenance orchestration - backups, doc mirroring, cleanup, task automation",
lifespan=lifespan
)
@@ -134,6 +135,7 @@ async def health(
"""Health check endpoint."""
return {
"status": "healthy",
"version": settings.app_version,
"scheduler_running": sched.running,
"jobs_count": len(sched.get_jobs()),
"database": settings.postgres_db