Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ec200c66ba | ||
|
|
2a6a91ed67 | ||
|
|
f334753918 | ||
|
|
8a6bdb9547 | ||
|
|
424c94d79a | ||
|
|
8f1a4402e7 | ||
|
|
9ae10af1df | ||
|
|
58ac4054a9 | ||
|
|
feb7c9e561 | ||
|
|
6f8e6f1c1f | ||
|
|
030da66a78 | ||
|
|
48e82e3a2e |
@@ -1,19 +1,32 @@
|
|||||||
name: Build and Push
|
name: Build and Push
|
||||||
|
|
||||||
on:
|
on:
|
||||||
release:
|
push:
|
||||||
types: [published]
|
tags:
|
||||||
|
- 'v*'
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
|
release:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- name: Create Gitea Release
|
||||||
|
run: |
|
||||||
|
curl -sf -X POST \
|
||||||
|
-H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"tag_name": "${{ github.ref_name }}", "name": "Release ${{ github.ref_name }}", "body": "Automated release for ${{ github.ref_name }}"}' \
|
||||||
|
"${{ github.server_url }}/api/v1/repos/${{ github.repository }}/releases"
|
||||||
|
|
||||||
build:
|
build:
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
|
needs: release
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@v4
|
||||||
|
|
||||||
- name: Login to Gitea Registry
|
- name: Login to Gitea Registry
|
||||||
uses: docker/login-action@v3
|
uses: docker/login-action@v3
|
||||||
with:
|
with:
|
||||||
registry: git.schweitz.net
|
registry: git.schweitz.internal
|
||||||
username: ${{ secrets.REGISTRY_USER }}
|
username: ${{ secrets.REGISTRY_USER }}
|
||||||
password: ${{ secrets.REGISTRY_PASSWORD }}
|
password: ${{ secrets.REGISTRY_PASSWORD }}
|
||||||
|
|
||||||
@@ -23,5 +36,11 @@ jobs:
|
|||||||
context: .
|
context: .
|
||||||
push: true
|
push: true
|
||||||
tags: |
|
tags: |
|
||||||
git.schweitz.net/jpmschweitzer/scheduler:latest
|
git.schweitz.internal/jpmschweitzer/scheduler:latest
|
||||||
git.schweitz.net/jpmschweitzer/scheduler:${{ github.ref_name }}
|
git.schweitz.internal/jpmschweitzer/scheduler:${{ github.ref_name }}
|
||||||
|
|
||||||
|
- name: Trigger Watchtower update
|
||||||
|
if: success()
|
||||||
|
run: |
|
||||||
|
curl -sf -H "Authorization: Bearer ${{ secrets.WATCHTOWER_TOKEN }}" \
|
||||||
|
http://watchtower:8080/v1/update
|
||||||
|
|||||||
@@ -0,0 +1,72 @@
|
|||||||
|
|
||||||
|
# AGENTS.md
|
||||||
|
|
||||||
|
> **Start every session by reading this file.**
|
||||||
|
> This file outlines the operational protocols, coding standards, and architectural decisions for this FastAPI project.
|
||||||
|
|
||||||
|
## 1. Agent Operational Protocols
|
||||||
|
|
||||||
|
### 🧠 Work Patterns (Plan-Act-Reflect)
|
||||||
|
* **Plan:** Before writing code, briefly outline your plan. Identify which files you will touch and what the side effects might be.
|
||||||
|
* **Act:** Execute the changes in small, atomic steps.
|
||||||
|
* **Reflect:** After coding, verify your work. Did you break existing tests? Did you add new tests?
|
||||||
|
|
||||||
|
### 🛡️ Git Discipline
|
||||||
|
* **NEVER commit to `main` or `master` directly.** Always create a feature branch: `feature/your-feature-name` or `fix/issue-description`.
|
||||||
|
* **Commit Messages:** Use the [Conventional Commits](https://www.conventionalcommits.org/) format.
|
||||||
|
* `feat: add user login endpoint`
|
||||||
|
* `fix: resolve database connection timeout`
|
||||||
|
* `refactor: split monolith dependency file`
|
||||||
|
* **Atomic Commits:** Keep commits small. One logical change = one commit.
|
||||||
|
|
||||||
|
### 📝 Changelog Maintenance
|
||||||
|
* **Update `CHANGELOG.md`** with every user-facing change.
|
||||||
|
* Format: `## [Unreleased] - YYYY-MM-DD` followed by `### Added`, `### Changed`, or `### Fixed`.
|
||||||
|
|
||||||
|
### 🚀 Release Flow
|
||||||
|
When changes are ready for deployment:
|
||||||
|
|
||||||
|
1. **Ask user if deploy cycle is desired **
|
||||||
|
|
||||||
|
2. **Update version** in `pyproject.toml`:
|
||||||
|
- Bug fixes: bump patch version (1.8.3 → 1.8.4)
|
||||||
|
- New features: bump minor version (1.8.4 → 1.9.0)
|
||||||
|
|
||||||
|
3. **Update CHANGELOG.md**:
|
||||||
|
- Move items from `[Unreleased]` to new version section
|
||||||
|
- Add release date: `## [1.8.4] - 2025-12-16`
|
||||||
|
|
||||||
|
4. **Commit and tag**:
|
||||||
|
```bash
|
||||||
|
git add -A
|
||||||
|
git commit -m "fix: description of changes"
|
||||||
|
git tag v1.8.4
|
||||||
|
git push origin main --tags
|
||||||
|
```
|
||||||
|
|
||||||
|
5. **CI/CD triggers automatically**:
|
||||||
|
- Gitea CI builds Docker image on new tag
|
||||||
|
- Watchtower pulls and deploys to production
|
||||||
|
- Verify deployment: `curl http://192.168.86.149:8000/health`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. FastAPI Architecture & Best Practices
|
||||||
|
*Reference: [FastAPI Best Practices](https://github.com/zhanymkanov/fastapi-best-practices)*
|
||||||
|
|
||||||
|
### 📂 Project Structure (Directory-based, NOT File-type based)
|
||||||
|
Do **not** group files by type (e.g., one huge `routers` folder). Group by **domain/module** inside a `src/` directory.
|
||||||
|
|
||||||
|
**Correct Structure:**
|
||||||
|
```text
|
||||||
|
src/
|
||||||
|
├── auth/
|
||||||
|
│ ├── router.py # Endpoints
|
||||||
|
│ ├── schemas.py # Pydantic models
|
||||||
|
│ ├── service.py # Business logic (CRUD, etc.)
|
||||||
|
│ ├── dependencies.py# Module-specific dependencies
|
||||||
|
│ └── config.py # Module-specific settings
|
||||||
|
├── posts/
|
||||||
|
│ ├── router.py
|
||||||
|
│ └── ...
|
||||||
|
└── main.py # App entry point
|
||||||
@@ -4,6 +4,52 @@ All notable changes to The Scheduler will be documented in this file.
|
|||||||
|
|
||||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
||||||
|
|
||||||
|
## [1.1.2] - 2026-01-03
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- CI: Use curl for release creation (release-action requires Go)
|
||||||
|
|
||||||
|
## [1.1.1] - 2026-01-03
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- CI: Auto-create Gitea release on version tag push (v*) instead of manual release trigger
|
||||||
|
|
||||||
|
## [1.1.0] - 2025-12-14
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- **Gitea Release Cleanup Executor** (`gitea_release_cleanup_executor.py`)
|
||||||
|
- Automatically cleans up old releases across all Gitea repositories
|
||||||
|
- Configurable retention count (default: 5 releases per repo)
|
||||||
|
- Repository exclusion list support
|
||||||
|
- Dry-run mode for safe testing
|
||||||
|
- Designed to run before Watchtower to prevent image tag accumulation
|
||||||
|
- **GITEA_TOKEN setting** in config for API token authentication (separate from password)
|
||||||
|
|
||||||
|
## [1.0.4] - 2025-12-14
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- CI/CD: Correct Watchtower port (8080)
|
||||||
|
|
||||||
|
## [1.0.3] - 2025-12-14
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- CI/CD: Trigger Watchtower update after successful Docker build
|
||||||
|
|
||||||
|
## [1.0.2] - 2025-12-14
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- Removed unused `setup_database.sql` from Dockerfile (database schema managed externally)
|
||||||
|
|
||||||
|
## [1.0.1] - 2025-12-14
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- **Version tracking now uses pyproject.toml** as single source of truth
|
||||||
|
- Added `pyproject.toml` with project metadata and dependencies
|
||||||
|
- `config.py` reads version from pyproject.toml using `tomllib`
|
||||||
|
- FastAPI app title and version dynamically loaded from config
|
||||||
|
- Health endpoint now includes version in response
|
||||||
|
- Dockerfile updated to include pyproject.toml
|
||||||
|
|
||||||
## [1.0.0] - 2025-12-07
|
## [1.0.0] - 2025-12-07
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|||||||
+1
-1
@@ -13,7 +13,7 @@ RUN pip install --no-cache-dir -r requirements.txt
|
|||||||
|
|
||||||
# Copy application
|
# Copy application
|
||||||
COPY src/ ./src/
|
COPY src/ ./src/
|
||||||
COPY setup_database.sql .
|
COPY pyproject.toml .
|
||||||
|
|
||||||
# Configure git
|
# Configure git
|
||||||
RUN git config --global user.name "The Librarian" && \
|
RUN git config --global user.name "The Librarian" && \
|
||||||
|
|||||||
@@ -0,0 +1,45 @@
|
|||||||
|
[project]
|
||||||
|
name = "the-scheduler"
|
||||||
|
version = "1.1.2"
|
||||||
|
description = "System-wide maintenance orchestration - backups, doc mirroring, cleanup, task automation"
|
||||||
|
readme = "README.md"
|
||||||
|
requires-python = ">=3.12"
|
||||||
|
dependencies = [
|
||||||
|
# FastAPI ecosystem
|
||||||
|
"fastapi~=0.124.0",
|
||||||
|
"uvicorn[standard]~=0.38.0",
|
||||||
|
"pydantic~=2.12.5",
|
||||||
|
"pydantic-settings~=2.7.0",
|
||||||
|
# Task scheduling
|
||||||
|
"apscheduler~=3.11.1",
|
||||||
|
"sqlalchemy~=2.0.36",
|
||||||
|
# Database
|
||||||
|
"psycopg2-binary~=2.9.11",
|
||||||
|
"redis~=5.2.0",
|
||||||
|
# HTTP client
|
||||||
|
"httpx~=0.28.1",
|
||||||
|
# Web scraping (for doc mirroring)
|
||||||
|
"scrapy~=2.12.0",
|
||||||
|
"beautifulsoup4~=4.12.3",
|
||||||
|
"lxml~=5.1.0",
|
||||||
|
# Git operations
|
||||||
|
"gitpython~=3.1.43",
|
||||||
|
# Security/Auth
|
||||||
|
"python-jose[cryptography]~=3.3.0",
|
||||||
|
]
|
||||||
|
|
||||||
|
[project.optional-dependencies]
|
||||||
|
test = [
|
||||||
|
"pytest~=8.3.4",
|
||||||
|
"pytest-asyncio~=0.25.2",
|
||||||
|
"pytest-cov~=6.0.0",
|
||||||
|
"freezegun~=1.5.1",
|
||||||
|
]
|
||||||
|
|
||||||
|
[build-system]
|
||||||
|
requires = ["setuptools>=61.0"]
|
||||||
|
build-backend = "setuptools.build_meta"
|
||||||
|
|
||||||
|
[tool.setuptools.packages.find]
|
||||||
|
where = ["."]
|
||||||
|
include = ["src*"]
|
||||||
@@ -1,120 +0,0 @@
|
|||||||
-- The Scheduler Database Setup
|
|
||||||
-- Run this from postgres-shared container
|
|
||||||
|
|
||||||
-- Create database and user
|
|
||||||
CREATE DATABASE scheduler;
|
|
||||||
CREATE USER scheduler_user WITH PASSWORD 'a/Ph0NhC4pTDDjSpL6q/DtBI+z0nf43ijVHjTo1KrXc=';
|
|
||||||
GRANT ALL PRIVILEGES ON DATABASE scheduler TO scheduler_user;
|
|
||||||
|
|
||||||
-- Connect to the new database
|
|
||||||
\c scheduler
|
|
||||||
|
|
||||||
-- Grant schema permissions
|
|
||||||
GRANT ALL ON SCHEMA public TO scheduler_user;
|
|
||||||
|
|
||||||
-- Create scheduled_tasks table
|
|
||||||
CREATE TABLE scheduled_tasks (
|
|
||||||
id SERIAL PRIMARY KEY,
|
|
||||||
task_name VARCHAR(100) NOT NULL UNIQUE,
|
|
||||||
service VARCHAR(50) NOT NULL, -- Which service owns this task
|
|
||||||
executor VARCHAR(100) NOT NULL, -- Executor module to run
|
|
||||||
priority INTEGER NOT NULL DEFAULT 30, -- Lower = higher priority (1-100)
|
|
||||||
|
|
||||||
-- Scheduling (supports wildcards: -1 = any)
|
|
||||||
minute INTEGER DEFAULT -1, -- 0-59 or -1 (any)
|
|
||||||
hour INTEGER DEFAULT -1, -- 0-23 or -1 (any)
|
|
||||||
day_of_month INTEGER DEFAULT -1, -- 1-31 or -1 (any)
|
|
||||||
month INTEGER DEFAULT -1, -- 1-12 or -1 (any)
|
|
||||||
day_of_week INTEGER DEFAULT -1, -- 0-6 (0=Monday) or -1 (any)
|
|
||||||
|
|
||||||
enabled BOOLEAN DEFAULT true,
|
|
||||||
description TEXT,
|
|
||||||
config JSONB, -- Arguments for executor
|
|
||||||
|
|
||||||
-- Execution tracking
|
|
||||||
last_run TIMESTAMP,
|
|
||||||
last_status VARCHAR(20), -- success, failed, timeout
|
|
||||||
last_duration_seconds INTEGER,
|
|
||||||
retry_count INTEGER DEFAULT 0,
|
|
||||||
max_retries INTEGER DEFAULT 3,
|
|
||||||
timeout_seconds INTEGER DEFAULT 3600, -- 1 hour default
|
|
||||||
|
|
||||||
-- Metadata
|
|
||||||
created_at TIMESTAMP DEFAULT NOW(),
|
|
||||||
updated_at TIMESTAMP DEFAULT NOW(),
|
|
||||||
created_by VARCHAR(50),
|
|
||||||
|
|
||||||
-- Constraints
|
|
||||||
CHECK (priority >= 1 AND priority <= 100),
|
|
||||||
CHECK (minute >= -1 AND minute <= 59),
|
|
||||||
CHECK (hour >= -1 AND hour <= 23),
|
|
||||||
CHECK (day_of_month >= -1 AND day_of_month <= 31),
|
|
||||||
CHECK (month >= -1 AND month <= 12),
|
|
||||||
CHECK (day_of_week >= -1 AND day_of_week <= 6)
|
|
||||||
);
|
|
||||||
|
|
||||||
-- Create task_executions table
|
|
||||||
CREATE TABLE task_executions (
|
|
||||||
id SERIAL PRIMARY KEY,
|
|
||||||
task_id INTEGER NOT NULL REFERENCES scheduled_tasks(id),
|
|
||||||
task_name VARCHAR(100) NOT NULL,
|
|
||||||
service VARCHAR(50) NOT NULL,
|
|
||||||
executor VARCHAR(100) NOT NULL,
|
|
||||||
priority INTEGER NOT NULL,
|
|
||||||
|
|
||||||
status VARCHAR(20) NOT NULL, -- pending, running, success, failed, timeout
|
|
||||||
triggered_by VARCHAR(50), -- scheduler, manual, retry
|
|
||||||
triggered_at TIMESTAMP DEFAULT NOW(),
|
|
||||||
started_at TIMESTAMP,
|
|
||||||
completed_at TIMESTAMP,
|
|
||||||
duration_seconds INTEGER,
|
|
||||||
|
|
||||||
output TEXT,
|
|
||||||
error TEXT,
|
|
||||||
retry_count INTEGER DEFAULT 0,
|
|
||||||
metadata JSONB
|
|
||||||
);
|
|
||||||
|
|
||||||
-- Create doc_sources table
|
|
||||||
CREATE TABLE doc_sources (
|
|
||||||
id SERIAL PRIMARY KEY,
|
|
||||||
project_name VARCHAR(100) NOT NULL UNIQUE,
|
|
||||||
current_version VARCHAR(50),
|
|
||||||
last_mirrored TIMESTAMP,
|
|
||||||
last_checked TIMESTAMP,
|
|
||||||
gitea_repo VARCHAR(200),
|
|
||||||
config JSONB
|
|
||||||
);
|
|
||||||
|
|
||||||
-- Indexes for scheduled_tasks
|
|
||||||
CREATE INDEX idx_tasks_enabled ON scheduled_tasks(enabled) WHERE enabled = true;
|
|
||||||
CREATE INDEX idx_tasks_priority ON scheduled_tasks(priority);
|
|
||||||
CREATE INDEX idx_tasks_service ON scheduled_tasks(service);
|
|
||||||
CREATE INDEX idx_tasks_schedule ON scheduled_tasks(minute, hour, day_of_month, month, day_of_week) WHERE enabled = true;
|
|
||||||
|
|
||||||
-- Indexes for task_executions
|
|
||||||
CREATE INDEX idx_executions_task_id ON task_executions(task_id);
|
|
||||||
CREATE INDEX idx_executions_task_name ON task_executions(task_name);
|
|
||||||
CREATE INDEX idx_executions_status ON task_executions(status);
|
|
||||||
CREATE INDEX idx_executions_triggered_at ON task_executions(triggered_at DESC);
|
|
||||||
CREATE INDEX idx_executions_service ON task_executions(service);
|
|
||||||
CREATE INDEX idx_executions_running ON task_executions(task_id) WHERE status = 'running';
|
|
||||||
|
|
||||||
-- Grant permissions
|
|
||||||
GRANT ALL ON ALL TABLES IN SCHEMA public TO scheduler_user;
|
|
||||||
GRANT ALL ON ALL SEQUENCES IN SCHEMA public TO scheduler_user;
|
|
||||||
|
|
||||||
-- Insert example test task
|
|
||||||
INSERT INTO scheduled_tasks
|
|
||||||
(task_name, service, executor, priority, minute, hour, description, config, created_by)
|
|
||||||
VALUES
|
|
||||||
('test_example_task', 'scheduler', 'example_executor', 50, -1, -1,
|
|
||||||
'Example task that runs every minute for testing',
|
|
||||||
'{"message": "Scheduler is working!", "delay_seconds": 2}'::jsonb,
|
|
||||||
'setup_script');
|
|
||||||
|
|
||||||
-- Verify tables created
|
|
||||||
\dt
|
|
||||||
|
|
||||||
-- Show the test task
|
|
||||||
SELECT task_name, service, priority, enabled, description FROM scheduled_tasks;
|
|
||||||
+23
-2
@@ -2,17 +2,37 @@
|
|||||||
Configuration management for The Scheduler.
|
Configuration management for The Scheduler.
|
||||||
Uses Pydantic BaseSettings for type-safe environment variable loading.
|
Uses Pydantic BaseSettings for type-safe environment variable loading.
|
||||||
"""
|
"""
|
||||||
|
import tomllib
|
||||||
from functools import lru_cache
|
from functools import lru_cache
|
||||||
|
from pathlib import Path
|
||||||
from pydantic_settings import BaseSettings
|
from pydantic_settings import BaseSettings
|
||||||
from pydantic import Field
|
from pydantic import Field
|
||||||
|
|
||||||
|
|
||||||
|
def _get_version_from_pyproject() -> tuple[str, str, str]:
|
||||||
|
"""Read version info from pyproject.toml."""
|
||||||
|
pyproject_path = Path(__file__).parent.parent / "pyproject.toml"
|
||||||
|
if pyproject_path.exists():
|
||||||
|
with open(pyproject_path, "rb") as f:
|
||||||
|
data = tomllib.load(f)
|
||||||
|
project = data.get("project", {})
|
||||||
|
return (
|
||||||
|
project.get("name", "the-scheduler"),
|
||||||
|
project.get("version", "0.0.0"),
|
||||||
|
project.get("description", ""),
|
||||||
|
)
|
||||||
|
return ("the-scheduler", "0.0.0", "")
|
||||||
|
|
||||||
|
|
||||||
|
_PROJECT_NAME, _PROJECT_VERSION, _PROJECT_DESCRIPTION = _get_version_from_pyproject()
|
||||||
|
|
||||||
|
|
||||||
class Settings(BaseSettings):
|
class Settings(BaseSettings):
|
||||||
"""Application settings loaded from environment variables."""
|
"""Application settings loaded from environment variables."""
|
||||||
|
|
||||||
# Application
|
# Application
|
||||||
app_name: str = Field(default="The Scheduler", alias="APP_NAME")
|
app_name: str = Field(default=_PROJECT_NAME, alias="APP_NAME")
|
||||||
app_version: str = Field(default="1.0.0", alias="APP_VERSION")
|
app_version: str = Field(default=_PROJECT_VERSION, alias="APP_VERSION")
|
||||||
debug: bool = Field(default=False, alias="DEBUG")
|
debug: bool = Field(default=False, alias="DEBUG")
|
||||||
host: str = Field(default="0.0.0.0", alias="HOST")
|
host: str = Field(default="0.0.0.0", alias="HOST")
|
||||||
port: int = Field(default=8090, alias="PORT")
|
port: int = Field(default=8090, alias="PORT")
|
||||||
@@ -37,6 +57,7 @@ class Settings(BaseSettings):
|
|||||||
gitea_url: str = Field(default="http://gitea:3000", alias="GITEA_URL")
|
gitea_url: str = Field(default="http://gitea:3000", alias="GITEA_URL")
|
||||||
gitea_user: str = Field(default="library", alias="GITEA_USER")
|
gitea_user: str = Field(default="library", alias="GITEA_USER")
|
||||||
gitea_password: str = Field(default="", alias="GITEA_PASSWORD")
|
gitea_password: str = Field(default="", alias="GITEA_PASSWORD")
|
||||||
|
gitea_token: str = Field(default="", alias="GITEA_TOKEN") # API token with write:repository scope
|
||||||
gitea_ssh_host: str = Field(default="gitea", alias="GITEA_SSH_HOST")
|
gitea_ssh_host: str = Field(default="gitea", alias="GITEA_SSH_HOST")
|
||||||
gitea_ssh_port: int = Field(default=22, alias="GITEA_SSH_PORT")
|
gitea_ssh_port: int = Field(default=22, alias="GITEA_SSH_PORT")
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,211 @@
|
|||||||
|
"""
|
||||||
|
Gitea Release Cleanup Executor
|
||||||
|
|
||||||
|
Cleans up old releases across all Gitea repositories, keeping only the most recent
|
||||||
|
releases per repository. Designed to run before Watchtower to prevent accumulation
|
||||||
|
of old container image tags.
|
||||||
|
|
||||||
|
Config schema:
|
||||||
|
{
|
||||||
|
"keep_count": 5, # Number of releases to keep per repo (default: 5)
|
||||||
|
"exclude_repos": [], # Repository names to skip (default: [])
|
||||||
|
"dry_run": false # If true, only log what would be deleted (default: false)
|
||||||
|
}
|
||||||
|
|
||||||
|
Example config:
|
||||||
|
{
|
||||||
|
"keep_count": 5,
|
||||||
|
"exclude_repos": ["important-repo", "legacy-app"],
|
||||||
|
"dry_run": false
|
||||||
|
}
|
||||||
|
|
||||||
|
Required settings:
|
||||||
|
- GITEA_URL: Base URL of Gitea instance
|
||||||
|
- GITEA_TOKEN: API token with write:repository scope
|
||||||
|
"""
|
||||||
|
import logging
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
from src.config import Settings
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# Default configuration values
|
||||||
|
DEFAULT_KEEP_COUNT = 5
|
||||||
|
DEFAULT_TIMEOUT = 60
|
||||||
|
API_BASE = "/api/v1"
|
||||||
|
|
||||||
|
|
||||||
|
async def execute(config: dict, settings: Settings) -> str:
|
||||||
|
"""
|
||||||
|
Clean up old releases across all accessible Gitea repositories.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
config: Task configuration (see module docstring)
|
||||||
|
settings: Global scheduler settings
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Summary of cleanup actions taken
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: On configuration error
|
||||||
|
Exception: On API call failure
|
||||||
|
"""
|
||||||
|
# Validate settings
|
||||||
|
if not settings.gitea_url:
|
||||||
|
raise ValueError("GITEA_URL not configured")
|
||||||
|
if not settings.gitea_token:
|
||||||
|
raise ValueError("GITEA_TOKEN not configured - required for release deletion")
|
||||||
|
|
||||||
|
# Parse configuration
|
||||||
|
keep_count = config.get("keep_count", DEFAULT_KEEP_COUNT)
|
||||||
|
exclude_repos = config.get("exclude_repos", [])
|
||||||
|
dry_run = config.get("dry_run", False)
|
||||||
|
|
||||||
|
if keep_count < 1:
|
||||||
|
raise ValueError(f"keep_count must be at least 1, got {keep_count}")
|
||||||
|
|
||||||
|
base_url = settings.gitea_url.rstrip("/")
|
||||||
|
headers = {"Authorization": f"token {settings.gitea_token}"}
|
||||||
|
|
||||||
|
stats = {
|
||||||
|
"repos_scanned": 0,
|
||||||
|
"repos_with_releases": 0,
|
||||||
|
"releases_deleted": 0,
|
||||||
|
"releases_skipped": 0,
|
||||||
|
"errors": [],
|
||||||
|
}
|
||||||
|
|
||||||
|
mode = "DRY RUN" if dry_run else "LIVE"
|
||||||
|
logger.info(f"Starting Gitea release cleanup ({mode}): keeping {keep_count} releases per repo")
|
||||||
|
|
||||||
|
async with httpx.AsyncClient(timeout=DEFAULT_TIMEOUT) as client:
|
||||||
|
# Fetch all repositories
|
||||||
|
repos = await _fetch_user_repos(client, base_url, headers)
|
||||||
|
logger.info(f"Found {len(repos)} repositories")
|
||||||
|
|
||||||
|
for repo in repos:
|
||||||
|
owner = repo["owner"]["login"]
|
||||||
|
name = repo["name"]
|
||||||
|
full_name = f"{owner}/{name}"
|
||||||
|
|
||||||
|
# Check exclusion list
|
||||||
|
if name in exclude_repos or full_name in exclude_repos:
|
||||||
|
logger.debug(f"Skipping excluded repo: {full_name}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
stats["repos_scanned"] += 1
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Fetch releases for this repo
|
||||||
|
releases = await _fetch_releases(client, base_url, headers, owner, name)
|
||||||
|
|
||||||
|
if not releases:
|
||||||
|
continue
|
||||||
|
|
||||||
|
stats["repos_with_releases"] += 1
|
||||||
|
|
||||||
|
# Determine which releases to delete (beyond keep_count)
|
||||||
|
to_delete = releases[keep_count:]
|
||||||
|
|
||||||
|
if not to_delete:
|
||||||
|
logger.debug(f"{full_name}: {len(releases)} releases, nothing to delete")
|
||||||
|
continue
|
||||||
|
|
||||||
|
logger.info(f"{full_name}: {len(releases)} releases, deleting {len(to_delete)}")
|
||||||
|
|
||||||
|
# Delete old releases
|
||||||
|
for release in to_delete:
|
||||||
|
release_id = release["id"]
|
||||||
|
tag_name = release["tag_name"]
|
||||||
|
|
||||||
|
if dry_run:
|
||||||
|
logger.info(f" [DRY RUN] Would delete: {tag_name} (id={release_id})")
|
||||||
|
stats["releases_skipped"] += 1
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
await _delete_release(client, base_url, headers, owner, name, release_id)
|
||||||
|
logger.info(f" Deleted: {tag_name}")
|
||||||
|
stats["releases_deleted"] += 1
|
||||||
|
except Exception as e:
|
||||||
|
error_msg = f"{full_name}/{tag_name}: {e}"
|
||||||
|
logger.warning(f" Failed to delete {tag_name}: {e}")
|
||||||
|
stats["errors"].append(error_msg)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
error_msg = f"{full_name}: {e}"
|
||||||
|
logger.error(f"Error processing {full_name}: {e}")
|
||||||
|
stats["errors"].append(error_msg)
|
||||||
|
|
||||||
|
# Build summary
|
||||||
|
summary = _build_summary(stats, dry_run)
|
||||||
|
logger.info(f"Cleanup complete: {summary}")
|
||||||
|
|
||||||
|
return summary
|
||||||
|
|
||||||
|
|
||||||
|
async def _fetch_user_repos(
|
||||||
|
client: httpx.AsyncClient,
|
||||||
|
base_url: str,
|
||||||
|
headers: dict,
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
"""Fetch all repositories accessible to the authenticated user."""
|
||||||
|
url = f"{base_url}{API_BASE}/user/repos"
|
||||||
|
params = {"limit": 100}
|
||||||
|
|
||||||
|
response = await client.get(url, headers=headers, params=params)
|
||||||
|
response.raise_for_status()
|
||||||
|
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
|
||||||
|
async def _fetch_releases(
|
||||||
|
client: httpx.AsyncClient,
|
||||||
|
base_url: str,
|
||||||
|
headers: dict,
|
||||||
|
owner: str,
|
||||||
|
repo: str,
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
"""Fetch releases for a repository, sorted newest first (Gitea default)."""
|
||||||
|
url = f"{base_url}{API_BASE}/repos/{owner}/{repo}/releases"
|
||||||
|
params = {"limit": 100}
|
||||||
|
|
||||||
|
response = await client.get(url, headers=headers, params=params)
|
||||||
|
response.raise_for_status()
|
||||||
|
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
|
||||||
|
async def _delete_release(
|
||||||
|
client: httpx.AsyncClient,
|
||||||
|
base_url: str,
|
||||||
|
headers: dict,
|
||||||
|
owner: str,
|
||||||
|
repo: str,
|
||||||
|
release_id: int,
|
||||||
|
) -> None:
|
||||||
|
"""Delete a specific release."""
|
||||||
|
url = f"{base_url}{API_BASE}/repos/{owner}/{repo}/releases/{release_id}"
|
||||||
|
|
||||||
|
response = await client.delete(url, headers=headers)
|
||||||
|
response.raise_for_status()
|
||||||
|
|
||||||
|
|
||||||
|
def _build_summary(stats: dict, dry_run: bool) -> str:
|
||||||
|
"""Build a human-readable summary of the cleanup operation."""
|
||||||
|
parts = [
|
||||||
|
f"Scanned {stats['repos_scanned']} repos",
|
||||||
|
f"{stats['repos_with_releases']} with releases",
|
||||||
|
]
|
||||||
|
|
||||||
|
if dry_run:
|
||||||
|
parts.append(f"{stats['releases_skipped']} releases would be deleted")
|
||||||
|
else:
|
||||||
|
parts.append(f"{stats['releases_deleted']} releases deleted")
|
||||||
|
|
||||||
|
if stats["errors"]:
|
||||||
|
parts.append(f"{len(stats['errors'])} errors")
|
||||||
|
|
||||||
|
return ", ".join(parts)
|
||||||
+4
-2
@@ -102,9 +102,10 @@ async def lifespan(app: FastAPI):
|
|||||||
|
|
||||||
|
|
||||||
# FastAPI app
|
# FastAPI app
|
||||||
|
settings = get_settings()
|
||||||
app = FastAPI(
|
app = FastAPI(
|
||||||
title="The Scheduler",
|
title=settings.app_name,
|
||||||
version="1.0.0",
|
version=settings.app_version,
|
||||||
description="System-wide maintenance orchestration - backups, doc mirroring, cleanup, task automation",
|
description="System-wide maintenance orchestration - backups, doc mirroring, cleanup, task automation",
|
||||||
lifespan=lifespan
|
lifespan=lifespan
|
||||||
)
|
)
|
||||||
@@ -134,6 +135,7 @@ async def health(
|
|||||||
"""Health check endpoint."""
|
"""Health check endpoint."""
|
||||||
return {
|
return {
|
||||||
"status": "healthy",
|
"status": "healthy",
|
||||||
|
"version": settings.app_version,
|
||||||
"scheduler_running": sched.running,
|
"scheduler_running": sched.running,
|
||||||
"jobs_count": len(sched.get_jobs()),
|
"jobs_count": len(sched.get_jobs()),
|
||||||
"database": settings.postgres_db
|
"database": settings.postgres_db
|
||||||
|
|||||||
Reference in New Issue
Block a user