jpmschweitzer/portainer-core
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor: extract scheduler to standalone repository

Browse Source 
Scheduler service moved to git.schweitz.net/jpmschweitzer/scheduler
- Source code removed from this repository
- Stack file updated to use container image from Gitea registry
- Added Gitea Actions runner to gitea stack for CI/CD
- Documentation updated with external service reference

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
jpmschweitzer
2025-12-11 15:18:42 +01:00
co-authored by Claude Opus 4.5
parent 9af231d72d
commit ca83bbffb5
31 changed files with 79 additions and 5943 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/settings.local.json
+4 -1
View File
@@ -6,7 +6,10 @@
"Bash(docker ps:*)",
"Bash(docker inspect:*)",
"Bash(tree:*)",
"Bash(nvidia-smi:*)"
"Bash(nvidia-smi:*)",
"Bash(find:*)",
"Bash(grep:*)",
"Bash(cat:*)"
],
"deny": [],
"ask": []
CONTAINERS.md
+1 -1
View File
@@ -24,7 +24,7 @@
| **Maintenance** | None | N/A (background) | No | No | ✅ Running |
| **Open WebUI** | 82 | http://192.168.86.149:82 | LAN | No | ✅ Running |
| **Core API** | 8083 | http://192.168.86.149:8083 | LAN | No | ✅ Running |
| **Scheduler** | 8090 | http://192.168.86.149:8090 | LAN | No | 🔧 Deploying |
| **Scheduler** | 8090 | http://192.168.86.149:8090 | LAN | No | ✅ Running (external) |
| **SearXNG** | 8087 | http://192.168.86.149:8087 | LAN | No | ✅ Running |
| **Jellyfin** | 8096, 8920, 7359, 1900 | https://media.schweitz.net | Internet | Yes (RTX 2080 Ti) | ✅ Running |
| **Nextcloud** | 8082 | https://cloud.schweitz.net | Internet | No | ⚠️ Stopped |
docs/EXTERNAL_SERVICES.md
+53
View File
@@ -0,0 +1,53 @@
# External Services
Services that have been extracted to their own repositories but are still part of the overall infrastructure.
---
## Scheduler
**Repository:** https://git.schweitz.net/jpmschweitzer/scheduler
**Container Image:** `git.schweitz.net/jpmschweitzer/scheduler:latest`
**Stack File:** `stacks/scheduler.yml` (remains in portainer-core)
### Overview
The Scheduler service handles system-wide maintenance orchestration including:
- Docker config backups
- Documentation mirroring
- Task automation via REST API
- Integration with Library Desk for consolidation jobs
### Deployment
- Container image built via Gitea Actions on release
- Stack file in portainer-core defines volumes, environment, and network
- Watchtower monitors for image updates
### Development
To make changes:
1. Clone: `git clone gitea:jpmschweitzer/scheduler.git`
2. Make changes
3. Create a release in Gitea to trigger build
4. Watchtower will auto-update the running container
### API
- Health: `http://scheduler:8090/health`
- Tasks: `http://scheduler:8090/tasks` (requires API key)
- Docs: `http://scheduler:8090/docs`
---
## Tatlock
**Repository:** https://git.schweitz.net/jpmschweitzer/tatlock
**Location:** `/home/jpmschweitzer/Projects/tatlock`
### Overview
Tatlock is a separate project maintained in its own repository.
---
## Future Migrations
The following services are planned for extraction:
- **core-api** - Infrastructure management API
- **library-desk** - Knowledge management and consolidation service
services/scheduler/CHANGELOG.md
-191
View File
@@ -1,191 +0,0 @@
# Changelog
All notable changes to The Scheduler will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
## [1.0.0] - 2025-12-07
### Added
#### Core Scheduling System
- **Hybrid APScheduler + PostgreSQL architecture** for minute-based task scheduling
- Single scheduler job runs every minute
- Queries database for tasks scheduled for current minute
- Executes up to 5 tasks concurrently by priority
- Priority queue system (1-100, lower = higher priority)
- **Cron-like scheduling** with wildcard support (`-1` = any)
- Supports minute, hour, day_of_month, month, day_of_week patterns
- Flexible scheduling from every-minute to specific dates
- **Task execution tracking** with full audit trail
- Database tables: `scheduled_tasks` (definitions) and `task_executions` (history)
- Tracks status, duration, output, errors, and retry attempts
- Execution metadata stored as JSONB
#### REST API
- **Full CRUD API** for task management with FastAPI
- `POST /tasks` - Create new scheduled task
- `GET /tasks` - List all tasks with filtering (service, enabled)
- `GET /tasks/{name}` - Get task details
- `PUT /tasks/{name}` - Update task configuration
- `DELETE /tasks/{name}` - Remove task
- `POST /tasks/{name}/trigger` - Manually trigger task execution
- **Execution history endpoints**
- `GET /executions` - Query execution history
- Filter by task_name, status, service
- Pagination support (limit parameter)
- **System monitoring endpoints**
- `GET /health` - Health check
- `GET /stats` - System statistics (enabled tasks, running tasks, 24h execution counts)
- **API Key authentication** (Bearer token) for all protected endpoints
- **OpenAPI documentation** at `/docs`
#### Task Executors
- **Example Executor** (`example_executor.py`)
- Simple test implementation with configurable message and delay
- Demonstrates executor pattern
- **Documentation Sync Executor** (`doc_sync_executor.py`)
- Mirrors documentation from upstream Git repositories to Gitea
- Supports full repository mirroring or selective path syncing
- Creates date-tagged snapshots (YYYY-MM-DD format)
- Generates `.SYNC_INFO.md` with sync metadata
- Configurable upstream repo, paths, branch, and Gitea destination
- **Config Backup Executor** (`config_backup_executor.py`)
- Backs up Docker configurations and data directories
- Supports multiple source paths with exclusion patterns
- Optional compression (tar.gz)
- Retention policy (days-based cleanup)
- Creates timestamped backups
#### Pre-configured Tasks
- **FastAPI Documentation Sync** (monthly on 11th at 04:00)
- Syncs entire FastAPI repository to `library/docs-fastapi`
- Priority: 60 (maintenance)
- **Ollama Documentation Sync** (monthly on 12th at 04:00)
- Syncs only `/docs` folder from Ollama repository to `library/docs-ollama`
- Priority: 60 (maintenance)
- **Docker Config Backup** (daily at 03:05)
- Backs up Docker data and configurations
- Priority: 20 (user task)
- 30-day retention
- **Example Test Task** (every minute, can be disabled)
- Test task for validation
- Priority: 50 (maintenance)
#### Testing Infrastructure
- **Comprehensive test suite** with 80% code coverage
- 85 total tests across multiple test files
- Pytest configuration with markers (unit, integration, api, executor)
- Coverage reporting with pytest-cov
- **Test categories**:
- Unit tests: Fast tests with mocked dependencies
- API tests: Comprehensive endpoint testing (24 tests)
- Executor tests: Task executor validation
- Integration tests: Real database operations
- **Test database setup**
- Dedicated `test_scheduler` database on postgres-shared
- Automatic schema creation and cleanup
- Database fixtures for clean test state
- Test user: `test_scheduler_user`
- **Test fixtures** (conftest.py)
- Mock database connections
- Mock scheduler and executor
- Sample task data
- Authentication headers
- Clean database state management
#### Documentation
- **Comprehensive README.md** (500+ lines)
- Architecture overview with ASCII diagram
- Quick start guide
- Complete API reference with curl examples
- Task scheduling patterns and examples
- Executor development guide
- Testing guide with coverage metrics
- Development and debugging information
- Security and performance notes
- **Test database setup guide** (test_database_setup.sql)
- SQL script for creating test environment
- Schema matching production
- Test data fixtures
#### Configuration
- **Pydantic Settings** for environment-based configuration
- PostgreSQL connection settings
- Redis connection (for future use)
- Gitea authentication
- API key configuration
- Computed properties (database_url, redis_url)
- **Docker stack configuration** (stacks/scheduler.yml)
- Virtual environment setup on startup
- Git configuration for librarian user
- Health checks
- Network isolation (docker-dataplane)
- Resource limits
### Technical Details
#### Database Schema
```sql
scheduled_tasks:
- Task definitions/templates
- Scheduling configuration (minute/hour/day patterns)
- Priority, enabled status, retry settings
- Task configuration as JSONB
- Execution tracking fields
task_executions:
- Individual execution records
- Status tracking (pending, running, success, failed, timeout)
- Duration and timestamp tracking
- Output and error details
- Metadata as JSONB
```
#### Performance
- Minute-based processing with lightweight scheduler ticks
- Connection pooling for database efficiency
- Database indexes for optimized task queries
- Concurrent execution with configurable limit (default: 5)
- Priority-based execution order
#### Security
- API key authentication required for protected endpoints
- Network isolation on docker-dataplane
- Environment variable-based secrets
- Gitea token authentication for git operations
- Database credentials in environment
### Dependencies
- FastAPI (web framework)
- APScheduler (task scheduling)
- psycopg2-binary (PostgreSQL driver)
- Pydantic (configuration management)
- pytest + pytest-asyncio + pytest-cov (testing)
- GitPython (git operations)
### Coverage Metrics
```
Module Coverage
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
config.py 100% ✅
example_executor.py 100% ✅
main.py (API endpoints) 95% ✅
doc_sync_executor.py 78% ✅
executor.py (core logic) 71% ✅
config_backup_executor.py 50% 📈
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
TOTAL 80% 🎯
```
## [Unreleased]
### Planned
- Redis integration for distributed locking
- Webhook notifications for task completion
- Task dependencies (run task B after task A succeeds)
- Task groups and tags
- More executors (database backup, log rotation, etc.)
- Web UI for task management
- Metrics export (Prometheus)
- Advanced scheduling (last business day of month, etc.)
services/scheduler/README.md
-576
View File
@@ -1,576 +0,0 @@
# The Scheduler
**System-wide maintenance orchestration for automated task scheduling.**
The Scheduler is a hybrid APScheduler + PostgreSQL-based task scheduling system that handles backups, documentation mirroring, cleanup, and automated maintenance tasks across the Portainer Core infrastructure.
## Architecture
**Hybrid Design**: APScheduler + Database-driven Priority System
```
┌─────────────────────────────────────────────────────────┐
│ APScheduler (runs every minute) │
│ └─> Query PostgreSQL for tasks scheduled this minute │
│ └─> Execute up to 5 tasks concurrently by priority │
└─────────────────────────────────────────────────────────┘
```
**Key Features**:
- **Minute-based scheduling** with cron-like patterns (`-1` = wildcard)
- **Priority queue** (1-100, lower = higher priority)
- **Concurrent execution** (max 5 tasks simultaneously)
- **Execution tracking** (full audit trail in database)
- **REST API** for task management
- **Flexible executors** (modular task implementations)
## Quick Start
### Starting the Scheduler
```bash
# Deploy via Portainer
# Uses stack: /stacks/scheduler.yml
# Check health
curl http://localhost:8090/health
# View API docs
open http://localhost:8090/docs
```
### API Authentication
All protected endpoints require an API key:
```bash
# Set in environment or use default
export SCHEDULER_API_KEY="your-api-key-here"
# Make authenticated request
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
http://localhost:8090/tasks
```
## Scheduling Tasks
### Task Configuration
Tasks are defined with:
- **task_name**: Unique identifier
- **service**: Which service owns this task
- **executor**: Python module to execute
- **priority**: 1-100 (1=emergency, 10-30=user, 40-70=maintenance)
- **schedule**: Minute, hour, day, month, day_of_week (-1 = any)
- **config**: JSON configuration for executor
### Scheduling Examples
```python
# Every minute (wildcard)
minute=-1, hour=-1, day_of_month=-1, month=-1, day_of_week=-1
# Daily at 3:05 AM
minute=5, hour=3, day_of_month=-1, month=-1, day_of_week=-1
# 11th of every month at 4:00 AM
minute=0, hour=4, day_of_month=11, month=-1, day_of_week=-1
# Every Monday at 9:00 AM
minute=0, hour=9, day_of_month=-1, month=-1, day_of_week=0
# First day of January at midnight
minute=0, hour=0, day_of_month=1, month=1, day_of_week=-1
```
### Priority System
```
Priority Range Purpose Examples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1-5 Emergency/System Critical system tasks
10-30 User Tasks User-initiated operations
40-70 Maintenance Backups, cleanup, doc sync
71-100 Low Priority Optional background tasks
```
## REST API
### Task Management
```bash
# List all tasks
GET /tasks
GET /tasks?enabled=true&service=scheduler
# Get task details
GET /tasks/{task_name}
# Create new task
POST /tasks
{
"task_name": "my_task",
"service": "scheduler",
"executor": "example_executor",
"priority": 50,
"minute": 0,
"hour": 4,
"description": "Daily task at 4 AM",
"config": {"key": "value"}
}
# Update task
PUT /tasks/{task_name}
{
"priority": 60,
"enabled": false
}
# Delete task
DELETE /tasks/{task_name}
# Manually trigger task
POST /tasks/{task_name}/trigger
```
### Execution History
```bash
# View recent executions
GET /executions?limit=20
# Filter by task
GET /executions?task_name=my_task&limit=10
# Filter by status
GET /executions?status=success
# Filter by service
GET /executions?service=scheduler
```
### System Stats
```bash
# Get system statistics
GET /stats
{
"scheduler_running": true,
"tasks_enabled": 3,
"tasks_currently_running": 0,
"concurrent_limit": 5,
"execution_stats_24h": {
"success": 15,
"failed": 1
}
}
```
## Task Executors
Executors are Python modules that implement the actual task logic.
### Built-in Executors
#### 1. Example Executor
**File**: `src/executors/example_executor.py`
**Purpose**: Test/example implementation
```python
config = {
"message": "Hello World",
"delay_seconds": 2
}
```
#### 2. Doc Sync Executor
**File**: `src/executors/doc_sync_executor.py`
**Purpose**: Mirror documentation from GitHub to Gitea
```python
config = {
"project": "fastapi",
"upstream_repo": "https://github.com/tiangolo/fastapi.git",
"docs_paths": ["/docs"], # Empty = entire repo
"gitea_repo": "library/docs-fastapi",
"branch": "main"
}
```
**Features**:
- Clones upstream repository
- Filters to specific paths or mirrors entire repo
- Pushes to Gitea with authentication
- Creates date-tagged snapshots
- Generates `.SYNC_INFO.md` with metadata
#### 3. Config Backup Executor
**File**: `src/executors/config_backup_executor.py`
**Purpose**: Backup Docker configs and data
```python
config = {
"sources": [
{
"path": "/data/docker-data",
"name": "docker-data",
"excludes": ["*/cache/*", "*.log"]
}
],
"backup_dir": "/backups/configs",
"compress": true,
"retention_days": 30
}
```
### Creating Custom Executors
1. Create file in `src/executors/your_executor.py`
2. Implement async `execute(config: dict, settings: Settings) -> str` function
3. Return success message or raise exception on failure
```python
# src/executors/my_executor.py
async def execute(config: dict, settings: Settings) -> str:
"""
Execute custom task.
Args:
config: Task configuration from database
settings: Global scheduler settings
Returns:
Success message
Raises:
Exception: On task failure
"""
# Your task logic here
result = do_something(config.get('param'))
return f"Task completed: {result}"
```
4. Create task using API:
```bash
curl -X POST http://localhost:8090/tasks \
-H "Authorization: Bearer $SCHEDULER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"task_name": "my_custom_task",
"service": "scheduler",
"executor": "my_executor",
"priority": 50,
"minute": 0,
"hour": 2,
"config": {"param": "value"}
}'
```
## Current Tasks
### Active Schedules
| Task | Priority | Schedule | Description |
|------|----------|----------|-------------|
| `test_example_task` | 50 | Every minute | Test task (can be disabled) |
| `backup_docker_configs_daily` | 20 | Daily 03:05 | Backup Docker configs |
| `sync_fastapi_docs_monthly` | 60 | 11th @ 04:00 | Sync FastAPI docs to Gitea |
| `sync_ollama_docs_monthly` | 60 | 12th @ 04:00 | Sync Ollama docs to Gitea |
### Managing Tasks
```bash
# Disable test task
curl -X PUT http://localhost:8090/tasks/test_example_task \
-H "Authorization: Bearer $SCHEDULER_API_KEY" \
-H "Content-Type: application/json" \
-d '{"enabled": false}'
# View execution history
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
http://localhost:8090/executions?limit=10
```
## Database Schema
### scheduled_tasks
Stores task definitions (templates for execution).
```sql
- id (serial)
- task_name (varchar, unique)
- service (varchar) - owner service
- executor (varchar) - executor module name
- priority (integer) - 1-100
- minute, hour, day_of_month, month, day_of_week (integer) - schedule
- enabled (boolean)
- description (text)
- config (jsonb) - executor configuration
- last_run, last_status, last_duration_seconds - tracking
- retry_count, max_retries, timeout_seconds - execution control
- created_at, updated_at, created_by - metadata
```
### task_executions
Stores individual execution records (audit trail).
```sql
- id (serial)
- task_id (integer) - references scheduled_tasks
- task_name, service, executor, priority - snapshot
- status (varchar) - pending, running, success, failed, timeout
- triggered_by (varchar) - scheduler, manual, retry
- triggered_at, started_at, completed_at - timestamps
- duration_seconds (integer)
- output (text) - success message
- error (text) - error details
- retry_count (integer)
- metadata (jsonb)
```
## Testing
### Running Tests
```bash
# All tests with coverage
docker exec -w /app scheduler /venv/bin/pytest --cov=src --cov-report=term
# Only unit tests (fast, no database)
docker exec -w /app scheduler /venv/bin/pytest -m unit
# Only integration tests (with database)
docker exec -w /app scheduler /venv/bin/pytest -m integration
# Only API tests
docker exec -w /app scheduler /venv/bin/pytest -m api
# Only executor tests
docker exec -w /app scheduler /venv/bin/pytest -m executor
# Specific test file
docker exec -w /app scheduler /venv/bin/pytest tests/test_api.py -v
# With detailed output
docker exec -w /app scheduler /venv/bin/pytest -vv --tb=long
```
### Test Coverage
**Current Coverage: 80%**
```
Module Coverage
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
config.py 100% ✅
example_executor.py 100% ✅
main.py (API endpoints) 95% ✅
doc_sync_executor.py 78% ✅
executor.py (core logic) 71% ✅
config_backup_executor.py 50% 📈
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
TOTAL 80% 🎯
```
### Test Database
Tests use a dedicated PostgreSQL database:
- **Database**: `test_scheduler`
- **User**: `test_scheduler_user`
- **Location**: `postgres-shared` container
Tests automatically:
- Clean database before each test
- Insert test fixtures
- Verify database state
- Rollback changes after test
## Development
### Project Structure
```
scheduler/
├── src/
│ ├── config.py # Pydantic settings
│ ├── main.py # FastAPI app & endpoints
│ ├── executors/ # Task executors
│ │ ├── example_executor.py
│ │ ├── doc_sync_executor.py
│ │ └── config_backup_executor.py
│ └── tasks/
│ └── executor.py # Task execution engine
├── tests/
│ ├── conftest.py # Pytest fixtures
│ ├── test_api.py # API tests
│ ├── test_*_executor.py # Executor tests
│ └── test_database_integration.py
├── requirements.txt # Python dependencies
├── pytest.ini # Test configuration
└── README.md # This file
```
### Adding Dependencies
```bash
# Add to requirements.txt with version pinning
echo "new-package~=1.0.0" >> requirements.txt
# Rebuild venv in container
docker restart scheduler
```
### Debugging
```bash
# View logs
docker logs scheduler --tail 100 -f
# Check scheduler status
curl http://localhost:8090/health
# View database tasks
docker exec postgres-shared psql -U scheduler_user -d scheduler \
-c "SELECT task_name, enabled, priority, last_status FROM scheduled_tasks;"
# View execution history
docker exec postgres-shared psql -U scheduler_user -d scheduler \
-c "SELECT task_name, status, duration_seconds, completed_at FROM task_executions ORDER BY id DESC LIMIT 10;"
```
### Environment Variables
Required environment variables (set in `stacks/scheduler.yml`):
```bash
# Database
POSTGRES_HOST=postgres-shared
POSTGRES_PORT=5432
POSTGRES_DB=scheduler
POSTGRES_USER=scheduler_user
POSTGRES_PASSWORD=${SCHEDULER_DB_PASSWORD}
# API Security
SCHEDULER_API_KEY=${SCHEDULER_API_KEY}
# Gitea (for doc sync)
GITEA_URL=http://gitea:3000
GITEA_USER=librarian
GITEA_PASSWORD=${GITEA_PASSWORD}
# Redis (future use)
REDIS_HOST=redis-shared
REDIS_PORT=6379
REDIS_DB=3
```
## Monitoring
### Health Checks
```bash
# Container health
docker ps | grep scheduler
# API health
curl http://localhost:8090/health
# Scheduler running
curl http://localhost:8090/stats | jq '.scheduler_running'
```
### Metrics
```bash
# System statistics
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
http://localhost:8090/stats | jq
# Recent executions
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
"http://localhost:8090/executions?limit=20" | jq
# Failed tasks in last 24h
curl -H "Authorization: Bearer $SCHEDULER_API_KEY" \
"http://localhost:8090/executions?status=failed" | jq
```
### Common Issues
**Task not running?**
1. Check if task is enabled: `GET /tasks/{name}`
2. Verify schedule matches current time
3. Check execution history for errors: `GET /executions?task_name={name}`
4. View scheduler logs: `docker logs scheduler`
**Task timing out?**
- Increase `timeout_seconds` in task configuration
- Check executor implementation for long-running operations
- Consider breaking into smaller tasks
**Database connection errors?**
- Verify postgres-shared container is running
- Check database credentials in environment
- Test connection: `docker exec scheduler psql -h postgres-shared -U scheduler_user -d scheduler`
## API Reference
Full OpenAPI documentation available at: `http://localhost:8090/docs`
### Response Codes
- `200` - Success
- `400` - Bad request (invalid data)
- `401` - Missing API key
- `403` - Invalid API key
- `404` - Resource not found
- `500` - Internal server error
### Pagination
Use `limit` parameter for controlling result count:
```bash
GET /executions?limit=50 # Default: 20, Max: 100
```
### Filtering
Most list endpoints support filtering:
```bash
GET /tasks?enabled=true&service=scheduler
GET /executions?task_name=my_task&status=success&limit=10
```
## Security
- **API Key Authentication**: Required for all protected endpoints
- **Network Isolation**: Runs on `docker-dataplane` network
- **Database Credentials**: Stored in environment variables
- **Gitea Tokens**: Used instead of passwords for Git operations
- **Resource Limits**: CPU and memory limits in docker-compose
## Performance
- **Concurrent Execution**: Max 5 tasks simultaneously
- **Minute-based Processing**: Lightweight scheduler tick every minute
- **Priority Queue**: Higher priority tasks execute first
- **Database Indexes**: Optimized queries for task selection
- **Connection Pooling**: Reuses database connections
## License
Part of Portainer Core infrastructure.
## Support
For issues or questions:
1. Check logs: `docker logs scheduler`
2. View health: `curl http://localhost:8090/health`
3. Review execution history: `GET /executions`
4. Check this README for common solutions
services/scheduler/TASK_REGISTRATION.md
-289
View File
@@ -1,289 +0,0 @@
# Scheduler Task Registration Guide
## Overview
The Scheduler is **DB-driven** (not YAML-based). Tasks are registered via REST API and stored in PostgreSQL.
## Quick Reference
### Register a New Task
```bash
curl -X POST http://192.168.86.149:8090/tasks \
-H "Authorization: Bearer $SCHEDULER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"task_name": "my_task",
"service": "my-service",
"executor": "rest_api",
"priority": 25,
"description": "Task description",
"minute": 0,
"hour": -1,
"config": {
"method": "POST",
"url": "http://service:port/endpoint",
"headers": {
"Authorization": "Bearer ${ENV_VAR}"
},
"body": {}
}
}'
```
### List All Tasks
```bash
curl http://192.168.86.149:8090/tasks \
-H "Authorization: Bearer $SCHEDULER_API_KEY"
```
### Trigger Task Manually
```bash
curl -X POST http://192.168.86.149:8090/tasks/{task_name}/trigger \
-H "Authorization: Bearer $SCHEDULER_API_KEY"
```
### View Task Execution History
```bash
curl http://192.168.86.149:8090/executions?task_name={task_name} \
-H "Authorization: Bearer $SCHEDULER_API_KEY"
```
## Schedule Patterns
Use cron-style fields where **`-1` means "every"**:
| Pattern | minute | hour | day_of_month | month | day_of_week | Description |
|---------|--------|------|--------------|-------|-------------|-------------|
| Every minute | -1 | -1 | -1 | -1 | -1 | Runs every minute |
| Hourly | 0 | -1 | -1 | -1 | -1 | Runs at minute 0 of every hour |
| Daily at 2 AM | 0 | 2 | -1 | -1 | -1 | Runs at 2:00 AM every day |
| Weekly (Monday 2 AM) | 0 | 2 | -1 | -1 | 0 | Runs at 2:00 AM every Monday |
| Monthly (1st at 2 AM) | 0 | 2 | 1 | -1 | -1 | Runs at 2:00 AM on 1st of month |
**Note:** `day_of_week` is 0-6 (Monday-Sunday)
## Priority Levels
| Priority Range | Level | Use Cases |
|----------------|-------|-----------|
| 1-5 | Emergency/System | Critical system tasks |
| 10-30 | User | User-initiated tasks |
| 40-70 | Maintenance | Background maintenance |
| 70+ | Low | Low priority cleanup |
## Executor Types
### `rest_api` Executor
Calls HTTP endpoints. Supports environment variable substitution in headers/body.
**Config Format:**
```json
{
"method": "POST|GET|PUT|DELETE|PATCH",
"url": "http://service:port/endpoint",
"headers": {
"Authorization": "Bearer ${ENV_VAR}",
"Content-Type": "application/json"
},
"body": {
"key": "value"
},
"timeout": 30,
"auth": {
"type": "bearer|basic|api_key",
"value": "${TOKEN_ENV_VAR}"
}
}
```
**Environment Variables:** Use `${VAR_NAME}` syntax to reference env vars from the scheduler container.
### Other Executors
- `shell`: Execute shell commands
- `python`: Execute Python scripts
- `docker`: Docker operations
- `backup`: Backup operations
- `doc_sync`: Documentation sync
## Complete Task Schema
### Required Fields
- `task_name` (string): Unique task identifier
- `service` (string): Service that owns this task (e.g., "library-desk", "core-api")
- `executor` (string): Executor type ("rest_api", "shell", etc.)
- `priority` (int): Priority level (1-100)
### Optional Fields
- `description` (string): Human-readable description
- `minute` (int): -1 to 59, default: -1 (every minute)
- `hour` (int): -1 to 23, default: -1 (every hour)
- `day_of_month` (int): -1 to 31, default: -1 (every day)
- `month` (int): -1 to 12, default: -1 (every month)
- `day_of_week` (int): -1 to 6, default: -1 (every day)
- `enabled` (bool): Whether task is enabled, default: true
- `max_retries` (int): Max retry attempts, default: 3
- `timeout_seconds` (int): Execution timeout, default: 3600
- `config` (object): Executor-specific configuration
- `created_by` (string): Creator identifier, default: "api"
## Example: Librarian Knowledge Consolidation
This task processes unprocessed search queries and consolidates knowledge into wiki pages.
**Registration:**
```bash
curl -X POST http://192.168.86.149:8090/tasks \
-H "Authorization: Bearer $SCHEDULER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"task_name": "librarian_consolidation",
"service": "library-desk",
"executor": "rest_api",
"priority": 25,
"description": "Processes unprocessed search queries and consolidates knowledge",
"minute": 0,
"hour": -1,
"config": {
"method": "POST",
"url": "http://library-desk:8089/consolidation/run",
"headers": {
"Authorization": "Bearer ${LIBRARY_API_KEY}",
"Content-Type": "application/json"
},
"body": {
"dry_run": false,
"process_limit": 10,
"lookback_days": 7,
"min_web_results": 2
}
}
}'
```
**Schedule:** Runs hourly (at minute 0 of every hour)
**What it does:**
1. Queries Neo4j for unprocessed `SearchQuery` nodes
2. Analyzes web results with Ollama
3. Creates/updates wiki pages via WikiPageWriter
4. Extracts and adds new entities to knowledge graph
5. Marks searches as processed
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/tasks` | List all tasks |
| GET | `/tasks/{task_name}` | Get task details |
| POST | `/tasks` | Create new task |
| PUT | `/tasks/{task_name}` | Update task |
| DELETE | `/tasks/{task_name}` | Delete task |
| POST | `/tasks/{task_name}/trigger` | Manually trigger task |
| GET | `/executions` | View execution history |
| GET | `/stats` | Get scheduler statistics |
| GET | `/health` | Health check |
## Interactive Documentation
Visit **http://192.168.86.149:8090/docs** for interactive Swagger UI with:
- Full API documentation
- Request/response schemas
- Try-it-out functionality
- Example payloads
## Environment Variables
Get API keys from containers:
```bash
# Scheduler API key
docker exec scheduler printenv SCHEDULER_API_KEY
# Library Desk API key
docker exec library-desk printenv LIBRARY_API_KEY
```
## Task Management Tips
1. **Test with dry_run**: Set `enabled: false` initially or use dry_run in task config
2. **Monitor executions**: Check `/executions` endpoint regularly
3. **Start conservative**: Use longer intervals (hourly, daily) before going to frequent runs
4. **Use priorities wisely**: Reserve 1-10 for critical tasks, use 20-30 for user tasks
5. **Set timeouts appropriately**: Long-running tasks need higher timeout_seconds
## Troubleshooting
### Task not running
1. Check if task is enabled: `GET /tasks/{task_name}`
2. Check execution history: `GET /executions?task_name={task_name}`
3. Verify schedule matches current time
4. Check scheduler logs: `docker logs scheduler`
### Task failing
1. Check execution history for error messages
2. Verify endpoint is accessible from scheduler container
3. Test endpoint manually with curl
4. Check API keys and authentication
5. Verify executor configuration (especially for rest_api)
### Debug a task
```bash
# Get task details
curl http://192.168.86.149:8090/tasks/librarian_consolidation \
-H "Authorization: Bearer $SCHEDULER_API_KEY"
# View recent executions
curl http://192.168.86.149:8090/executions?task_name=librarian_consolidation&limit=5 \
-H "Authorization: Bearer $SCHEDULER_API_KEY"
# Trigger manually
curl -X POST http://192.168.86.149:8090/tasks/librarian_consolidation/trigger \
-H "Authorization: Bearer $SCHEDULER_API_KEY"
```
## Architecture
```
┌─────────────────────────────────────────────────────────┐
│ The Scheduler (AsyncIOScheduler) │
│ │
│ ┌────────────────────────────────────────────┐ │
│ │ Cron Job (every minute) │ │
│ │ ↓ │ │
│ │ Query PostgreSQL for tasks scheduled │ │
│ │ for current minute/hour/day │ │
│ │ ↓ │ │
│ │ Execute up to 5 tasks concurrently │ │
│ │ (based on priority) │ │
│ │ ↓ │ │
│ │ Call appropriate executor: │ │
│ │ - rest_api → HTTP request │ │
│ │ - shell → subprocess │ │
│ │ - python → exec │ │
│ └────────────────────────────────────────────┘ │
│ │
│ PostgreSQL: │
│ - scheduled_tasks (task definitions) │
│ - task_executions (execution history) │
└─────────────────────────────────────────────────────────┘
```
## Current Registered Tasks
As of 2025-12-09:
1. **backup_docker_configs_daily** - Daily at 3:05 AM
2. **librarian_consolidation** - Hourly (minute 0)
3. **test_example_task** - Every minute (testing)
4. **sync_fastapi_docs_monthly** - 11th of month at 4:00 AM
5. **sync_ollama_docs_monthly** - 12th of month at 4:00 AM
services/scheduler/pytest.ini
-45
View File
@@ -1,45 +0,0 @@
[pytest]
# Pytest configuration for The Scheduler
# Test discovery patterns
python_files = test_*.py
python_classes = Test*
python_functions = test_*
# Test paths
testpaths = tests
# Asyncio mode
asyncio_mode = auto
asyncio_default_fixture_loop_scope = function
# Coverage options
addopts =
--verbose
--strict-markers
--tb=short
--cov=src
--cov-report=term-missing
--cov-report=html:htmlcov
--cov-report=json:coverage.json
--cov-branch
# Custom markers
markers =
unit: Unit tests (fast, no external dependencies)
integration: Integration tests (may use database, slower)
executor: Tests for task executors
api: API endpoint tests
slow: Tests that take a while to run
# Ignore patterns
norecursedirs = .git .tox dist build *.egg venv __pycache__ node_modules
# Console output
console_output_style = progress
# Fail on first error (disable for CI)
# addopts = --exitfirst
# Show local variables in tracebacks
# addopts = --showlocals
services/scheduler/requirements.txt
-34
View File
@@ -1,34 +0,0 @@
# FastAPI ecosystem - Latest CVE-safe versions (Dec 2025)
fastapi~=0.124.0 # Latest non-vulnerable (no known CVEs)
uvicorn[standard]~=0.38.0 # Latest non-vulnerable (CVE-2025-43859 fixed in h11 0.16.0)
pydantic~=2.12.5 # Latest (CVE-2024-3772 fixed in 2.4.0+)
pydantic-settings~=2.7.0 # Settings management
# Task scheduling
apscheduler~=3.11.1 # Stable release (avoid 4.x alpha)
sqlalchemy~=2.0.36 # Required by APScheduler jobstore
# Database
psycopg2-binary~=2.9.11 # Latest stable PostgreSQL adapter
redis~=5.2.0 # Redis client
# HTTP client
httpx~=0.28.1 # Latest stable async HTTP client
# Web scraping (for doc mirroring)
scrapy~=2.12.0 # Latest stable
beautifulsoup4~=4.12.3 # HTML parsing
lxml~=5.1.0 # XML/HTML parser
# Git operations
gitpython~=3.1.43 # Latest stable
# Security/Auth
python-jose[cryptography]~=3.3.0 # JWT handling
# Testing
pytest~=8.3.4 # Test framework
pytest-asyncio~=0.25.2 # Async test support
pytest-cov~=6.0.0 # Coverage reporting
httpx~=0.28.1 # Already included above, used for API testing
freezegun~=1.5.1 # Time mocking for scheduler tests
services/scheduler/src/config.py
-73
View File
@@ -1,73 +0,0 @@
"""
Configuration management for The Scheduler.
Uses Pydantic BaseSettings for type-safe environment variable loading.
"""
from functools import lru_cache
from pydantic_settings import BaseSettings
from pydantic import Field
class Settings(BaseSettings):
"""Application settings loaded from environment variables."""
# Application
app_name: str = Field(default="The Scheduler", alias="APP_NAME")
app_version: str = Field(default="1.0.0", alias="APP_VERSION")
debug: bool = Field(default=False, alias="DEBUG")
host: str = Field(default="0.0.0.0", alias="HOST")
port: int = Field(default=8090, alias="PORT")
log_level: str = Field(default="INFO", alias="LOG_LEVEL")
# Security
scheduler_api_key: str = Field(default="dev-key-change-me", alias="SCHEDULER_API_KEY")
# PostgreSQL
postgres_host: str = Field(default="postgres-shared", alias="POSTGRES_HOST")
postgres_port: int = Field(default=5432, alias="POSTGRES_PORT")
postgres_db: str = Field(default="library_scheduler", alias="POSTGRES_DB")
postgres_user: str = Field(default="library_scheduler_user", alias="POSTGRES_USER")
postgres_password: str = Field(default="", alias="POSTGRES_PASSWORD")
# Redis
redis_host: str = Field(default="redis-shared", alias="REDIS_HOST")
redis_port: int = Field(default=6379, alias="REDIS_PORT")
redis_db: int = Field(default=3, alias="REDIS_DB")
# Gitea
gitea_url: str = Field(default="http://gitea:3000", alias="GITEA_URL")
gitea_user: str = Field(default="library", alias="GITEA_USER")
gitea_password: str = Field(default="", alias="GITEA_PASSWORD")
gitea_ssh_host: str = Field(default="gitea", alias="GITEA_SSH_HOST")
gitea_ssh_port: int = Field(default=22, alias="GITEA_SSH_PORT")
# Backup Configuration
backup_retention_daily: int = Field(default=7, alias="BACKUP_RETENTION_DAILY")
backup_retention_weekly: int = Field(default=4, alias="BACKUP_RETENTION_WEEKLY")
backup_retention_monthly: int = Field(default=12, alias="BACKUP_RETENTION_MONTHLY")
# Documentation Mirroring
docs_mirror_path: str = Field(default="/docs-mirror", alias="DOCS_MIRROR_PATH")
docs_check_interval: int = Field(default=21600, alias="DOCS_CHECK_INTERVAL") # 6 hours
@property
def database_url(self) -> str:
"""PostgreSQL connection URL for APScheduler."""
return f"postgresql://{self.postgres_user}:{self.postgres_password}@{self.postgres_host}:{self.postgres_port}/{self.postgres_db}"
@property
def redis_url(self) -> str:
"""Redis connection URL."""
return f"redis://{self.redis_host}:{self.redis_port}/{self.redis_db}"
class Config:
env_file = ".env"
case_sensitive = False
@lru_cache
def get_settings() -> Settings:
"""
Get cached settings instance.
Using lru_cache ensures we only create one Settings instance.
"""
return Settings()
services/scheduler/src/executors/__init__.py
-23
View File
@@ -1,23 +0,0 @@
"""
Executor modules for The Scheduler.
Each executor must implement an async execute(config, settings) function.
Example:
async def execute(config: dict, settings: Settings) -> str:
'''
Perform the task.
Args:
config: Task-specific configuration from scheduled_tasks.config
settings: Global scheduler settings
Returns:
Output message (success)
Raises:
Exception: On failure (will be logged as error)
'''
# Your task logic here
return "Task completed successfully"
"""
services/scheduler/src/executors/config_backup_executor.py
-159
View File
@@ -1,159 +0,0 @@
"""
Config Backup Executor
Backs up Docker container configs and host-based service configs.
Replicates functionality of maintenance container's backup-configs.sh
"""
import asyncio
import logging
import tarfile
import tempfile
from datetime import datetime, timedelta
from pathlib import Path
from typing import List, Dict, Any
from src.config import Settings
logger = logging.getLogger(__name__)
async def execute(config: dict, settings: Settings) -> str:
"""
Execute config backup task.
Config schema:
{
"sources": [
{
"path": "/data/docker-data",
"name": "docker-data",
"excludes": ["*/cache/*", "*/temp/*", "*.log"]
}
],
"backup_dir": "/backups/docker-configs",
"retention_days": 30,
"compress": true
}
Args:
config: Backup configuration
settings: Global scheduler settings
Returns:
Summary of backup operation
Raises:
Exception: On backup failure
"""
sources = config.get('sources', [])
backup_dir = Path(config.get('backup_dir', '/backups/docker-configs'))
retention_days = config.get('retention_days', 30)
compress = config.get('compress', True)
if not sources:
raise ValueError("No backup sources configured")
timestamp = datetime.now().strftime('%Y%m%d-%H%M%S')
backup_filename = f"docker-configs-{timestamp}.tar.gz"
backup_file = backup_dir / backup_filename
logger.info(f"Starting Docker configs backup: {backup_filename}")
# Create backup directory
backup_dir.mkdir(parents=True, exist_ok=True)
# Create temporary directory for staging
with tempfile.TemporaryDirectory(prefix='backup-') as temp_dir:
temp_path = Path(temp_dir)
results = []
# Backup each source
for source in sources:
source_path = Path(source['path'])
source_name = source['name']
excludes = source.get('excludes', [])
if not source_path.exists():
logger.warning(f"Source path does not exist: {source_path}")
continue
logger.info(f"Backing up {source_name} from {source_path}")
# Create tar for this source
source_tar = temp_path / f"{source_name}.tar.gz"
def tar_filter(tarinfo):
"""Filter function to exclude patterns."""
for pattern in excludes:
# Simple pattern matching (could be enhanced with fnmatch)
if pattern.replace('*/', '').replace('/*', '') in tarinfo.name:
logger.debug(f"Excluding: {tarinfo.name}")
return None
return tarinfo
with tarfile.open(source_tar, 'w:gz') as tar:
tar.add(
source_path,
arcname=source_name,
filter=tar_filter,
recursive=True
)
source_size = source_tar.stat().st_size / (1024 * 1024) # MB
results.append(f"{source_name}: {source_size:.2f}MB")
logger.info(f"Backed up {source_name}: {source_size:.2f}MB")
# Combine all source backups into final archive
logger.info("Creating combined backup archive...")
with tarfile.open(backup_file, 'w:gz') as final_tar:
for item in temp_path.glob('*.tar.gz'):
final_tar.add(item, arcname=item.name)
# Verify backup created
if not backup_file.exists():
raise Exception("Backup file was not created")
backup_size = backup_file.stat().st_size / (1024 * 1024) # MB
logger.info(f"Backup created successfully: {backup_size:.2f}MB")
# Clean up old backups
await cleanup_old_backups(backup_dir, retention_days)
# Count remaining backups
backup_count = len(list(backup_dir.glob('docker-configs-*.tar.gz')))
total_size = sum(f.stat().st_size for f in backup_dir.glob('docker-configs-*.tar.gz'))
total_size_mb = total_size / (1024 * 1024)
output = (
f"Backup completed: {backup_filename} ({backup_size:.2f}MB). "
f"Sources: {', '.join(results)}. "
f"Retention: {backup_count} backups, {total_size_mb:.2f}MB total."
)
logger.info(output)
return output
async def cleanup_old_backups(backup_dir: Path, retention_days: int):
"""Remove backups older than retention period."""
cutoff_date = datetime.now() - timedelta(days=retention_days)
removed_count = 0
removed_size = 0
logger.info(f"Cleaning up backups older than {retention_days} days...")
for backup_file in backup_dir.glob('docker-configs-*.tar.gz'):
# Get file modification time
file_mtime = datetime.fromtimestamp(backup_file.stat().st_mtime)
if file_mtime < cutoff_date:
file_size = backup_file.stat().st_size
logger.info(f"Removing old backup: {backup_file.name} (from {file_mtime:%Y-%m-%d})")
backup_file.unlink()
removed_count += 1
removed_size += file_size
if removed_count > 0:
removed_size_mb = removed_size / (1024 * 1024)
logger.info(f"Removed {removed_count} old backups, freed {removed_size_mb:.2f}MB")
else:
logger.info("No old backups to remove")
services/scheduler/src/executors/doc_sync_executor.py
-259
View File
@@ -1,259 +0,0 @@
"""
Documentation Sync Executor
Syncs documentation from upstream Git repositories to Gitea.
Clones source repos, extracts docs directories, pushes to Gitea.
"""
import asyncio
import logging
import shutil
from datetime import datetime
from pathlib import Path
from typing import List, Optional
from src.config import Settings
logger = logging.getLogger(__name__)
async def execute(config: dict, settings: Settings) -> str:
"""
Sync documentation from upstream Git repo to Gitea.
Config schema:
{
"project": "fastapi",
"upstream_repo": "https://github.com/tiangolo/fastapi.git",
"docs_paths": ["/docs", "/docs_src"],
"gitea_repo": "library/docs-fastapi",
"gitea_url": "http://gitea:3000",
"branch": "main"
}
Args:
config: Documentation sync configuration
settings: Global scheduler settings
Returns:
Summary of sync operation
Raises:
Exception: On sync failure
"""
project = config.get('project')
upstream_repo = config.get('upstream_repo')
docs_paths = config.get('docs_paths', ['/docs'])
gitea_repo = config.get('gitea_repo')
gitea_url = config.get('gitea_url', settings.gitea_url)
branch = config.get('branch', 'main')
if not all([project, upstream_repo, gitea_repo]):
raise ValueError("Missing required config: project, upstream_repo, or gitea_repo")
logger.info(f"Starting doc sync for {project}")
logger.info(f"Upstream: {upstream_repo}")
logger.info(f"Gitea: {gitea_repo}")
# Work directory
work_dir = Path(f"/app/task-data/doc-sync/{project}")
upstream_dir = work_dir / "upstream"
gitea_dir = work_dir / "gitea"
try:
# Clean work directory
if work_dir.exists():
logger.info(f"Cleaning work directory: {work_dir}")
shutil.rmtree(work_dir)
work_dir.mkdir(parents=True)
# Clone upstream repo (shallow clone for speed)
logger.info(f"Cloning upstream repo...")
await _run_command([
'git', 'clone',
'--depth', '1',
'--branch', branch,
upstream_repo,
str(upstream_dir)
])
# Get upstream version/commit
upstream_commit = await _get_git_commit(upstream_dir)
upstream_date = datetime.now().strftime('%Y-%m-%d')
logger.info(f"Upstream commit: {upstream_commit[:8]}")
# Clone Gitea repo (or create if doesn't exist)
# Build authenticated URL for Gitea
gitea_url_clean = gitea_url.replace('http://', '').replace('https://', '')
gitea_clone_url = f"http://{settings.gitea_user}:{settings.gitea_password}@{gitea_url_clean}/{gitea_repo}.git"
# Log without credentials
logger.info(f"Cloning Gitea repo: {gitea_url}/{gitea_repo}.git")
# Try to clone, if fails create new repo
try:
await _run_command([
'git', 'clone',
gitea_clone_url,
str(gitea_dir)
])
except Exception as e:
logger.warning(f"Gitea repo doesn't exist, will create: {e}")
gitea_dir.mkdir(parents=True)
await _run_command(['git', 'init'], cwd=gitea_dir)
await _run_command(['git', 'checkout', '-b', branch], cwd=gitea_dir)
# Set remote
await _run_command([
'git', 'remote', 'add', 'origin',
gitea_clone_url
], cwd=gitea_dir)
# Clear existing content in Gitea repo (except .git)
for item in gitea_dir.iterdir():
if item.name != '.git':
if item.is_dir():
shutil.rmtree(item)
else:
item.unlink()
# Determine what to copy based on docs_paths
copied_paths = []
# If docs_paths is empty or contains "." or "/", sync entire repo
if not docs_paths or any(p in [".", "/", ""] for p in docs_paths):
logger.info(f"Copying entire repository...")
for item in upstream_dir.iterdir():
if item.name != '.git':
dest = gitea_dir / item.name
if item.is_dir():
shutil.copytree(item, dest)
else:
shutil.copy2(item, dest)
copied_paths = ["entire repository"]
else:
# Copy only specified paths
logger.info(f"Copying specific paths: {docs_paths}")
for doc_path in docs_paths:
source = upstream_dir / doc_path.lstrip('/')
if source.exists():
dest = gitea_dir / source.name
logger.info(f"Copying {source.name}...")
if source.is_dir():
shutil.copytree(source, dest)
else:
shutil.copy2(source, dest)
copied_paths.append(source.name)
else:
logger.warning(f"Path not found in upstream: {doc_path}")
if not copied_paths:
raise Exception("No documentation paths were copied")
# Create .SYNC_INFO.md with metadata (don't overwrite README.md from upstream)
sync_info_path = gitea_dir / ".SYNC_INFO.md"
content_desc = "Complete repository mirror" if "entire repository" in copied_paths else f"Paths: {', '.join(copied_paths)}"
sync_info_content = f"""# Sync Information
This is a mirror of the {project.title()} repository.
**Synced from:** {upstream_repo}
**Branch:** {branch}
**Commit:** {upstream_commit}
**Sync Date:** {upstream_date}
**Content:** {content_desc}
---
This repository is automatically synced monthly by The Scheduler.
For the latest updates, visit the official repository.
"""
sync_info_path.write_text(sync_info_content)
# Git add, commit, push
await _run_command(['git', 'add', '.'], cwd=gitea_dir)
# Check if there are changes
status = await _run_command(
['git', 'status', '--porcelain'],
cwd=gitea_dir,
capture=True
)
if not status.strip():
logger.info("No changes detected, skipping commit")
return f"Documentation already up to date (commit: {upstream_commit[:8]})"
# Commit changes
commit_msg = f"Sync {project} docs from {upstream_commit[:8]} on {upstream_date}"
await _run_command([
'git', 'commit',
'-m', commit_msg
], cwd=gitea_dir)
# Tag with date
tag = f"sync-{upstream_date}"
await _run_command([
'git', 'tag', '-f', tag,
'-m', f"Documentation snapshot {upstream_date}"
], cwd=gitea_dir)
# Push to Gitea
logger.info("Pushing to Gitea...")
await _run_command([
'git', 'push', 'origin', branch, '--force'
], cwd=gitea_dir)
await _run_command([
'git', 'push', 'origin', tag, '--force'
], cwd=gitea_dir)
# Cleanup
logger.info("Cleaning up work directory...")
shutil.rmtree(work_dir)
result = (
f"Successfully synced {project} documentation. "
f"Copied: {', '.join(copied_paths)}. "
f"Upstream commit: {upstream_commit[:8]}. "
f"Tagged: {tag}"
)
logger.info(result)
return result
except Exception as e:
# Cleanup on error
if work_dir.exists():
shutil.rmtree(work_dir)
raise
async def _run_command(
cmd: List[str],
cwd: Optional[Path] = None,
capture: bool = False
) -> str:
"""Run shell command asynchronously."""
logger.debug(f"Running: {' '.join(cmd)} (cwd: {cwd})")
proc = await asyncio.create_subprocess_exec(
*cmd,
cwd=cwd,
stdout=asyncio.subprocess.PIPE if capture else asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE
)
stdout, stderr = await proc.communicate()
if proc.returncode != 0:
error_msg = stderr.decode() if stderr else "Unknown error"
raise Exception(f"Command failed: {' '.join(cmd)}\n{error_msg}")
return stdout.decode() if capture else ""
async def _get_git_commit(repo_dir: Path) -> str:
"""Get current git commit hash."""
output = await _run_command(
['git', 'rev-parse', 'HEAD'],
cwd=repo_dir,
capture=True
)
return output.strip()
services/scheduler/src/executors/example_executor.py
-42
View File
@@ -1,42 +0,0 @@
"""
Example executor demonstrating the pattern.
Shows how to write task executors for The Scheduler.
"""
import asyncio
import logging
from src.config import Settings
logger = logging.getLogger(__name__)
async def execute(config: dict, settings: Settings) -> str:
"""
Example task executor.
Args:
config: Task configuration from scheduled_tasks.config JSONB field
Example: {"message": "Hello", "delay_seconds": 2}
settings: Global scheduler settings (database, API keys, etc.)
Returns:
Output message describing what was done
Raises:
Exception: On failure (will be caught and logged by executor framework)
"""
message = config.get('message', 'No message configured')
delay = config.get('delay_seconds', 1)
logger.info(f"Example executor starting: {message}")
# Simulate some work
await asyncio.sleep(delay)
# You can access settings
logger.info(f"Using database: {settings.postgres_db}")
# Return success message
output = f"Executed example task: {message} (took {delay}s)"
logger.info(output)
return output
services/scheduler/src/executors/rest_api_executor.py
-266
View File
@@ -1,266 +0,0 @@
"""
Generic REST API Executor
Universal executor for calling any REST API endpoint across the system.
Supports GET, POST, PUT, DELETE with configurable payloads, headers, and authentication.
This executor can be used to trigger any service endpoint:
- Library Desk knowledge consolidation
- Core API operations
- External webhooks
- Any HTTP-based task
Config schema:
{
"url": "http://service:port/endpoint",
"method": "POST", # GET, POST, PUT, DELETE, PATCH
"payload": {...}, # Request body (for POST/PUT/PATCH)
"headers": {...}, # Additional headers
"auth": {
"type": "bearer", # bearer, basic, api_key
"token": "${ENV_VAR}", # Use ${VAR} for env vars
"header": "Authorization" # Optional: header name for API key
},
"timeout": 300, # Timeout in seconds (default: 300)
"verify_ssl": true, # SSL verification (default: true)
"success_codes": [200, 201, 202], # Expected success codes
"response_path": "result.message" # JSONPath to extract from response
}
Example configs:
1. Library Desk Knowledge Consolidation:
{
"url": "http://library-desk:8089/consolidate/knowledge",
"method": "POST",
"payload": {"process_limit": 10, "lookback_days": 7, "dry_run": false},
"auth": {"type": "bearer", "token": "${LIBRARY_DESK_API_KEY}"}
}
2. Core API Container Restart:
{
"url": "http://core-api:8088/v1/infrastructure/containers/nginx/restart",
"method": "POST",
"auth": {"type": "bearer", "token": "${CORE_API_KEY}"}
}
3. External Webhook:
{
"url": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL",
"method": "POST",
"payload": {"text": "Scheduled task completed"},
"verify_ssl": true
}
"""
import logging
import os
import re
import httpx
from typing import Any, Dict, Optional
from src.config import Settings
logger = logging.getLogger(__name__)
async def execute(config: dict, settings: Settings) -> str:
"""
Execute REST API call with configured parameters.
Args:
config: REST API call configuration (see module docstring)
settings: Global scheduler settings
Returns:
Response summary or extracted result
Raises:
ValueError: On configuration error
Exception: On API call failure
"""
# Required configuration
url = config.get('url')
if not url:
raise ValueError("Missing required config: 'url'")
method = config.get('method', 'POST').upper()
if method not in ['GET', 'POST', 'PUT', 'DELETE', 'PATCH']:
raise ValueError(f"Invalid HTTP method: {method}")
# Optional configuration
payload = config.get('payload', {})
headers = config.get('headers', {})
timeout = config.get('timeout', 300)
verify_ssl = config.get('verify_ssl', True)
success_codes = config.get('success_codes', [200, 201, 202, 204])
response_path = config.get('response_path')
# Handle authentication
auth_config = config.get('auth', {})
if auth_config:
auth_header = _build_auth_header(auth_config, settings)
if auth_header:
headers.update(auth_header)
# Substitute environment variables in URL and payload
url = _substitute_env_vars(url)
payload = _substitute_env_vars_recursive(payload)
logger.info(f"Executing REST API call: {method} {url}")
if payload:
logger.debug(f"Payload: {_redact_sensitive(payload)}")
# Make HTTP request
try:
async with httpx.AsyncClient(timeout=timeout, verify=verify_ssl) as client:
if method == 'GET':
response = await client.get(url, headers=headers)
elif method == 'POST':
response = await client.post(url, json=payload, headers=headers)
elif method == 'PUT':
response = await client.put(url, json=payload, headers=headers)
elif method == 'DELETE':
response = await client.delete(url, headers=headers)
elif method == 'PATCH':
response = await client.patch(url, json=payload, headers=headers)
# Check status code
if response.status_code not in success_codes:
error_msg = (
f"API call failed with status {response.status_code}: "
f"{response.text[:500]}"
)
logger.error(error_msg)
raise Exception(error_msg)
# Parse response
try:
response_data = response.json()
except:
response_data = {"text": response.text}
# Extract specific field if response_path provided
result_text = None
if response_path and isinstance(response_data, dict):
result_text = _extract_json_path(response_data, response_path)
if not result_text:
# Build summary from response
if isinstance(response_data, dict):
# Look for common result fields
result_text = (
response_data.get('message') or
response_data.get('result') or
response_data.get('summary') or
f"Success ({response.status_code})"
)
else:
result_text = f"Success ({response.status_code})"
logger.info(f"API call succeeded: {result_text}")
return str(result_text)
except httpx.HTTPStatusError as e:
error_msg = f"HTTP {e.response.status_code}: {e.response.text[:500]}"
logger.error(error_msg)
raise Exception(error_msg)
except httpx.RequestError as e:
error_msg = f"Request failed: {str(e)}"
logger.error(error_msg)
raise Exception(error_msg)
except Exception as e:
logger.error(f"REST API call failed: {e}", exc_info=True)
raise
def _build_auth_header(auth_config: dict, settings: Settings) -> Optional[Dict[str, str]]:
"""Build authentication header from config."""
auth_type = auth_config.get('type', '').lower()
if auth_type == 'bearer':
token = auth_config.get('token', '')
token = _substitute_env_vars(token)
if token:
return {"Authorization": f"Bearer {token}"}
elif auth_type == 'basic':
username = _substitute_env_vars(auth_config.get('username', ''))
password = _substitute_env_vars(auth_config.get('password', ''))
if username and password:
import base64
credentials = base64.b64encode(f"{username}:{password}".encode()).decode()
return {"Authorization": f"Basic {credentials}"}
elif auth_type == 'api_key':
key = _substitute_env_vars(auth_config.get('key', ''))
header_name = auth_config.get('header', 'X-API-Key')
if key:
return {header_name: key}
return None
def _substitute_env_vars(text: str) -> str:
"""Substitute ${ENV_VAR} placeholders with environment variables."""
if not isinstance(text, str):
return text
# Find all ${VAR} patterns
pattern = r'\$\{([A-Z_][A-Z0-9_]*)\}'
matches = re.findall(pattern, text)
for var_name in matches:
env_value = os.getenv(var_name, '')
if not env_value:
logger.warning(f"Environment variable not found: {var_name}")
text = text.replace(f"${{{var_name}}}", env_value)
return text
def _substitute_env_vars_recursive(data: Any) -> Any:
"""Recursively substitute environment variables in nested structures."""
if isinstance(data, dict):
return {k: _substitute_env_vars_recursive(v) for k, v in data.items()}
elif isinstance(data, list):
return [_substitute_env_vars_recursive(item) for item in data]
elif isinstance(data, str):
return _substitute_env_vars(data)
else:
return data
def _extract_json_path(data: dict, path: str) -> Optional[str]:
"""
Extract value from nested dict using dot notation.
Example: "result.message" -> data["result"]["message"]
"""
try:
keys = path.split('.')
value = data
for key in keys:
if isinstance(value, dict):
value = value.get(key)
else:
return None
return str(value) if value is not None else None
except:
return None
def _redact_sensitive(data: Any) -> Any:
"""Redact sensitive fields from logs."""
if isinstance(data, dict):
redacted = {}
sensitive_keys = ['password', 'token', 'api_key', 'secret', 'auth']
for k, v in data.items():
if any(s in k.lower() for s in sensitive_keys):
redacted[k] = '***REDACTED***'
else:
redacted[k] = _redact_sensitive(v)
return redacted
elif isinstance(data, list):
return [_redact_sensitive(item) for item in data]
else:
return data
services/scheduler/src/main.py
-530
View File
@@ -1,530 +0,0 @@
"""
The Scheduler - System-wide maintenance orchestration.
Handles backups, documentation mirroring, cleanup, and automated tasks.
Architecture: Hybrid APScheduler + DB-based priority system
- APScheduler runs a single job every minute
- Job queries DB for tasks scheduled in that minute
- Executes up to 5 tasks concurrently based on priority
"""
from fastapi import FastAPI, HTTPException, Depends, Header
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore
from apscheduler.triggers.cron import CronTrigger
from contextlib import asynccontextmanager
import logging
from src.config import get_settings, Settings
from src.tasks.executor import TaskExecutor
from src.models import TaskCreate, TaskUpdate, TaskResponse
# Configure logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
# Global instances
scheduler: AsyncIOScheduler | None = None
task_executor: TaskExecutor | None = None
def get_scheduler() -> AsyncIOScheduler:
"""Dependency to get scheduler instance."""
if scheduler is None:
raise HTTPException(500, "Scheduler not initialized")
return scheduler
def get_task_executor() -> TaskExecutor:
"""Dependency to get task executor instance."""
if task_executor is None:
raise HTTPException(500, "Task executor not initialized")
return task_executor
# Lifespan manager for startup/shutdown
@asynccontextmanager
async def lifespan(app: FastAPI):
"""Manage application lifecycle - startup and shutdown."""
global scheduler, task_executor
settings = get_settings()
# Startup
logger.info("=" * 60)
logger.info("The Scheduler - System-wide Maintenance Orchestration")
logger.info("=" * 60)
logger.info(f"Architecture: Hybrid APScheduler + DB-based priority system")
logger.info(f"Database: {settings.postgres_host}:{settings.postgres_port}/{settings.postgres_db}")
logger.info(f"API: http://{settings.host}:{settings.port}")
logger.info(f"Docs: http://{settings.host}:{settings.port}/docs")
logger.info("=" * 60)
# Initialize task executor
task_executor = TaskExecutor(settings)
logger.info("Task executor initialized (max 5 concurrent tasks)")
# Initialize APScheduler with minimal configuration
# No jobstore needed - we only have one in-memory job
scheduler = AsyncIOScheduler(
job_defaults={
'coalesce': True, # Combine missed runs
'max_instances': 1, # Only one instance running
'misfire_grace_time': 30 # 30s grace period for minute-based execution
}
)
# Add the single minute-based task processor
scheduler.add_job(
func=task_executor.process_minute,
trigger=CronTrigger(minute='*'), # Run every minute
id='process_tasks',
name='Process scheduled tasks',
replace_existing=True
)
try:
scheduler.start()
logger.info("Scheduler started - processing tasks every minute")
logger.info("Priority system: 1-5 (emergency/system), 10-30 (user), 40-70+ (maintenance)")
except Exception as e:
logger.error(f"Failed to start scheduler: {e}")
raise
yield
# Shutdown
logger.info("Stopping The Scheduler...")
if scheduler:
scheduler.shutdown(wait=True)
logger.info("Scheduler stopped")
# FastAPI app
app = FastAPI(
title="The Scheduler",
version="1.0.0",
description="System-wide maintenance orchestration - backups, doc mirroring, cleanup, task automation",
lifespan=lifespan
)
# Dependencies
async def verify_api_key(
authorization: str = Header(None),
settings: Settings = Depends(get_settings)
):
"""Verify API key from Authorization header."""
if not authorization or not authorization.startswith("Bearer "):
raise HTTPException(401, "Missing API key")
key = authorization.replace("Bearer ", "")
if key != settings.scheduler_api_key:
raise HTTPException(403, "Invalid API key")
return key
# ============================================================================
# Public Endpoints (no auth required)
# ============================================================================
@app.get("/health")
async def health(
settings: Settings = Depends(get_settings),
sched: AsyncIOScheduler = Depends(get_scheduler)
):
"""Health check endpoint."""
return {
"status": "healthy",
"scheduler_running": sched.running,
"jobs_count": len(sched.get_jobs()),
"database": settings.postgres_db
}
# ============================================================================
# Protected Endpoints (require API key)
# ============================================================================
@app.get("/tasks")
async def list_tasks(
enabled: bool = None,
service: str = None,
api_key: str = Depends(verify_api_key),
executor: TaskExecutor = Depends(get_task_executor)
):
"""List all scheduled tasks from database."""
import psycopg2.extras
query = "SELECT * FROM scheduled_tasks WHERE 1=1"
params = []
if enabled is not None:
query += " AND enabled = %s"
params.append(enabled)
if service:
query += " AND service = %s"
params.append(service)
query += " ORDER BY priority ASC, task_name ASC"
with executor.get_db_connection() as conn:
with conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor) as cur:
cur.execute(query, params)
tasks = [dict(task) for task in cur.fetchall()]
return {
"tasks": tasks,
"count": len(tasks)
}
@app.get("/tasks/{task_name}")
async def get_task_details(
task_name: str,
api_key: str = Depends(verify_api_key),
executor: TaskExecutor = Depends(get_task_executor)
):
"""Get details for a specific task."""
import psycopg2.extras
with executor.get_db_connection() as conn:
with conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor) as cur:
cur.execute("SELECT * FROM scheduled_tasks WHERE task_name = %s", (task_name,))
task = cur.fetchone()
if not task:
raise HTTPException(404, f"Task '{task_name}' not found")
return dict(task)
@app.post("/tasks", response_model=TaskResponse, tags=["Task Management"])
async def create_task(
task: TaskCreate,
api_key: str = Depends(verify_api_key),
executor: TaskExecutor = Depends(get_task_executor)
):
"""
Create a new scheduled task.
## Schedule Pattern
Use cron-style fields where `-1` means "every":
- `minute: -1, hour: -1` → Runs every minute
- `minute: 0, hour: -1` → Runs at minute 0 of every hour (hourly)
- `minute: 0, hour: 2` → Runs at 2:00 AM every day
- `minute: 0, hour: 2, day_of_week: 0` → Runs at 2:00 AM every Monday
## Priority Levels
- **1-5**: Emergency/critical system tasks
- **10-30**: User-initiated tasks
- **40-70**: Background maintenance
- **70+**: Low priority cleanup
## Executor Types
### `rest_api` Executor
Calls HTTP endpoints. Config format:
```json
{
"method": "POST",
"url": "http://service:port/endpoint",
"headers": {
"Authorization": "Bearer ${ENV_VAR}",
"Content-Type": "application/json"
},
"body": {"key": "value"}
}
```
Environment variables can be referenced with `${VAR_NAME}` syntax.
## Example: Librarian Consolidation Task
```json
{
"task_name": "librarian_consolidation",
"service": "library-desk",
"executor": "rest_api",
"priority": 25,
"description": "Processes search queries and consolidates knowledge",
"minute": 0,
"hour": -1,
"config": {
"method": "POST",
"url": "http://library-desk:8089/consolidation/run",
"headers": {
"Authorization": "Bearer ${LIBRARY_DESK_API_KEY}",
"Content-Type": "application/json"
},
"body": {
"dry_run": false,
"process_limit": 10
}
}
}
```
This creates an hourly task that calls the consolidation endpoint.
"""
import psycopg2.extras
import json
# Convert task model to dict
task_data = task.model_dump()
# Convert config dict to JSON string if present
if 'config' in task_data and isinstance(task_data['config'], dict):
task_data['config'] = json.dumps(task_data['config'])
with executor.get_db_connection() as conn:
with conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor) as cur:
cur.execute("""
INSERT INTO scheduled_tasks
(task_name, service, executor, priority, minute, hour,
day_of_month, month, day_of_week, enabled, description,
config, max_retries, timeout_seconds, created_by)
VALUES
(%(task_name)s, %(service)s, %(executor)s, %(priority)s,
%(minute)s, %(hour)s, %(day_of_month)s, %(month)s,
%(day_of_week)s, %(enabled)s, %(description)s,
%(config)s::jsonb, %(max_retries)s, %(timeout_seconds)s,
%(created_by)s)
RETURNING *
""", task_data)
new_task = dict(cur.fetchone())
conn.commit()
logger.info(f"Created task: {new_task['task_name']}")
return new_task
@app.put("/tasks/{task_name}")
async def update_task(
task_name: str,
task_data: dict,
api_key: str = Depends(verify_api_key),
executor: TaskExecutor = Depends(get_task_executor)
):
"""Update an existing scheduled task."""
import psycopg2.extras
import json
# Build update query dynamically
allowed_fields = ['service', 'executor', 'priority', 'minute', 'hour',
'day_of_month', 'month', 'day_of_week', 'enabled',
'description', 'config', 'max_retries', 'timeout_seconds']
updates = {k: v for k, v in task_data.items() if k in allowed_fields}
if not updates:
raise HTTPException(400, "No valid fields to update")
# Convert config dict to JSON string if present
if 'config' in updates and isinstance(updates['config'], dict):
updates['config'] = json.dumps(updates['config'])
set_clause = ', '.join([f"{k} = %({k})s" for k in updates.keys()])
with executor.get_db_connection() as conn:
with conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor) as cur:
cur.execute(f"""
UPDATE scheduled_tasks
SET {set_clause}, updated_at = NOW()
WHERE task_name = %(task_name)s
RETURNING *
""", {**updates, 'task_name': task_name})
updated_task = cur.fetchone()
if not updated_task:
raise HTTPException(404, f"Task '{task_name}' not found")
conn.commit()
logger.info(f"Updated task: {task_name}")
return dict(updated_task)
@app.delete("/tasks/{task_name}")
async def delete_task(
task_name: str,
api_key: str = Depends(verify_api_key),
executor: TaskExecutor = Depends(get_task_executor)
):
"""Delete a scheduled task."""
with executor.get_db_connection() as conn:
with conn.cursor() as cur:
cur.execute("""
DELETE FROM scheduled_tasks
WHERE task_name = %s
RETURNING task_name
""", (task_name,))
deleted = cur.fetchone()
if not deleted:
raise HTTPException(404, f"Task '{task_name}' not found")
conn.commit()
logger.info(f"Deleted task: {task_name}")
return {"message": f"Task '{task_name}' deleted successfully"}
@app.post("/tasks/{task_name}/trigger")
async def trigger_task(
task_name: str,
api_key: str = Depends(verify_api_key),
executor: TaskExecutor = Depends(get_task_executor)
):
"""Manually trigger a task to run immediately."""
import psycopg2.extras
# Get task details
with executor.get_db_connection() as conn:
with conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor) as cur:
cur.execute("SELECT * FROM scheduled_tasks WHERE task_name = %s", (task_name,))
task = cur.fetchone()
if not task:
raise HTTPException(404, f"Task '{task_name}' not found")
if not task['enabled']:
raise HTTPException(400, f"Task '{task_name}' is disabled")
# Execute task immediately in background
import asyncio
asyncio.create_task(executor.execute_task(dict(task)))
logger.info(f"Manually triggered task: {task_name}")
return {
"message": f"Task '{task_name}' triggered successfully",
"task_name": task_name,
"priority": task['priority'],
"executor": task['executor']
}
# Legacy endpoints (deprecated)
@app.post("/tasks/backup")
async def trigger_backup(api_key: str = Depends(verify_api_key)):
"""Trigger backup tasks manually (deprecated - use POST /tasks/{name}/trigger)"""
# TODO: Implement backup executor
logger.info("Manual backup triggered")
return {
"message": "Backup task triggered",
"status": "not_implemented",
"note": "Backup executor needs to be implemented"
}
@app.post("/tasks/docs/update")
async def trigger_docs_update(
project: str = None,
api_key: str = Depends(verify_api_key)
):
"""Trigger documentation mirror update"""
# TODO: Implement doc mirror executor
logger.info(f"Doc mirror update triggered for project: {project or 'all'}")
return {
"message": f"Documentation update triggered for {project or 'all projects'}",
"status": "not_implemented",
"note": "Doc mirror executor needs to be implemented"
}
@app.post("/tasks/docs/check-versions")
async def check_doc_versions(api_key: str = Depends(verify_api_key)):
"""Check for new documentation versions"""
# TODO: Implement version check executor
logger.info("Version check triggered")
return {
"message": "Version check triggered",
"status": "not_implemented",
"note": "Version check executor needs to be implemented"
}
@app.post("/tasks/cleanup")
async def trigger_cleanup(api_key: str = Depends(verify_api_key)):
"""Run cleanup tasks"""
# TODO: Implement cleanup executor
logger.info("Cleanup task triggered")
return {
"message": "Cleanup task triggered",
"status": "not_implemented",
"note": "Cleanup executor needs to be implemented"
}
@app.get("/executions")
async def task_history(
limit: int = 20,
task_name: str = None,
service: str = None,
status: str = None,
api_key: str = Depends(verify_api_key),
executor: TaskExecutor = Depends(get_task_executor)
):
"""View task execution history."""
import psycopg2.extras
query = "SELECT * FROM task_executions WHERE 1=1"
params = []
if task_name:
query += " AND task_name = %s"
params.append(task_name)
if service:
query += " AND service = %s"
params.append(service)
if status:
query += " AND status = %s"
params.append(status)
query += " ORDER BY triggered_at DESC LIMIT %s"
params.append(limit)
with executor.get_db_connection() as conn:
with conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor) as cur:
cur.execute(query, params)
executions = [dict(ex) for ex in cur.fetchall()]
return {
"executions": executions,
"count": len(executions),
"limit": limit
}
@app.get("/stats")
async def stats(
api_key: str = Depends(verify_api_key),
settings: Settings = Depends(get_settings),
sched: AsyncIOScheduler = Depends(get_scheduler),
task_exec: TaskExecutor = Depends(get_task_executor)
):
"""Get system statistics."""
import psycopg2.extras
# Query task stats from database
with task_exec.get_db_connection() as conn:
with conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor) as cur:
# Count enabled tasks
cur.execute("SELECT COUNT(*) as count FROM scheduled_tasks WHERE enabled = true")
enabled_tasks = cur.fetchone()['count']
# Count running tasks
cur.execute("SELECT COUNT(*) as count FROM task_executions WHERE status = 'running'")
running_tasks = cur.fetchone()['count']
# Recent execution stats (last 24 hours)
cur.execute("""
SELECT status, COUNT(*) as count
FROM task_executions
WHERE triggered_at > NOW() - INTERVAL '24 hours'
GROUP BY status
""")
execution_stats = {row['status']: row['count'] for row in cur.fetchall()}
return {
"scheduler_running": sched.running,
"minute_processor_active": True, # If we got here, it's running
"database": settings.postgres_db,
"tasks_enabled": enabled_tasks,
"tasks_currently_running": running_tasks,
"concurrent_limit": 5,
"execution_stats_24h": execution_stats,
"priority_system": "1-5 (emergency/system), 10-30 (user), 40-70+ (maintenance)"
}
services/scheduler/src/models.py
-210
View File
@@ -1,210 +0,0 @@
"""
Pydantic models for The Scheduler API.
"""
from pydantic import BaseModel, Field
from typing import Optional, Dict, Any
from enum import Enum
class SchedulePattern(str, Enum):
"""Common schedule patterns."""
EVERY_MINUTE = "every_minute"
HOURLY = "hourly"
DAILY = "daily"
WEEKLY = "weekly"
MONTHLY = "monthly"
class TaskExecutor(str, Enum):
"""Available task executors."""
REST_API = "rest_api"
SHELL = "shell"
PYTHON = "python"
class TaskPriorityLevel(str, Enum):
"""Task priority levels."""
EMERGENCY = "emergency" # 1-5: Critical system tasks
SYSTEM = "system" # 5-10: System maintenance
USER = "user" # 10-30: User-initiated tasks
MAINTENANCE = "maintenance" # 40-70: Background maintenance
LOW = "low" # 70+: Low priority cleanup
class TaskBase(BaseModel):
"""Base task fields."""
task_name: str = Field(
...,
description="Unique task identifier (e.g., 'librarian_consolidation')",
example="librarian_consolidation"
)
service: str = Field(
...,
description="Service that owns this task (e.g., 'library-desk', 'core-api')",
example="library-desk"
)
executor: str = Field(
...,
description="Executor type: 'rest_api', 'shell', or 'python'",
example="rest_api"
)
priority: int = Field(
...,
ge=1,
le=100,
description="Priority level (1-5: emergency, 10-30: user, 40-70: maintenance, 70+: low)",
example=25
)
description: Optional[str] = Field(
None,
description="Human-readable task description",
example="Processes unprocessed search queries and consolidates knowledge into wiki pages"
)
class TaskSchedule(BaseModel):
"""Cron-style schedule fields."""
minute: int = Field(
-1,
ge=-1,
le=59,
description="Minute to run (-1 = every minute, 0-59 = specific minute)",
example=-1
)
hour: int = Field(
-1,
ge=-1,
le=23,
description="Hour to run (-1 = every hour, 0-23 = specific hour)",
example=-1
)
day_of_month: int = Field(
-1,
ge=-1,
le=31,
description="Day of month to run (-1 = every day, 1-31 = specific day)",
example=-1
)
month: int = Field(
-1,
ge=-1,
le=12,
description="Month to run (-1 = every month, 1-12 = specific month)",
example=-1
)
day_of_week: int = Field(
-1,
ge=-1,
le=6,
description="Day of week to run (-1 = every day, 0-6 = Monday-Sunday)",
example=-1
)
class TaskConfig(BaseModel):
"""Task execution configuration."""
enabled: bool = Field(
True,
description="Whether task is enabled",
example=True
)
max_retries: int = Field(
3,
ge=0,
le=10,
description="Maximum retry attempts on failure",
example=3
)
timeout_seconds: int = Field(
3600,
ge=1,
description="Execution timeout in seconds",
example=3600
)
config: Optional[Dict[str, Any]] = Field(
None,
description="Executor-specific configuration (varies by executor type)",
example={
"method": "POST",
"url": "http://library-desk:8089/consolidation/run",
"headers": {
"Authorization": "Bearer ${LIBRARY_DESK_API_KEY}",
"Content-Type": "application/json"
},
"body": {
"dry_run": False,
"process_limit": 10,
"lookback_days": 7,
"min_web_results": 2
}
}
)
class TaskCreate(TaskBase, TaskSchedule, TaskConfig):
"""Request model for creating a new scheduled task."""
created_by: Optional[str] = Field(
"api",
description="User or system that created this task",
example="api"
)
class Config:
json_schema_extra = {
"example": {
"task_name": "librarian_consolidation",
"service": "library-desk",
"executor": "rest_api",
"priority": 25,
"description": "Processes unprocessed search queries and consolidates knowledge",
"minute": 0,
"hour": -1,
"day_of_month": -1,
"month": -1,
"day_of_week": -1,
"enabled": True,
"max_retries": 3,
"timeout_seconds": 3600,
"config": {
"method": "POST",
"url": "http://library-desk:8089/consolidation/run",
"headers": {
"Authorization": "Bearer ${LIBRARY_DESK_API_KEY}",
"Content-Type": "application/json"
},
"body": {
"dry_run": False,
"process_limit": 10,
"lookback_days": 7,
"min_web_results": 2
}
},
"created_by": "api"
}
}
class TaskUpdate(BaseModel):
"""Request model for updating a scheduled task."""
service: Optional[str] = None
executor: Optional[str] = None
priority: Optional[int] = Field(None, ge=1, le=100)
minute: Optional[int] = Field(None, ge=-1, le=59)
hour: Optional[int] = Field(None, ge=-1, le=23)
day_of_month: Optional[int] = Field(None, ge=-1, le=31)
month: Optional[int] = Field(None, ge=-1, le=12)
day_of_week: Optional[int] = Field(None, ge=-1, le=6)
enabled: Optional[bool] = None
description: Optional[str] = None
config: Optional[Dict[str, Any]] = None
max_retries: Optional[int] = Field(None, ge=0, le=10)
timeout_seconds: Optional[int] = Field(None, ge=1)
class TaskResponse(TaskCreate):
"""Response model for task operations."""
created_at: str
updated_at: Optional[str] = None
class Config:
from_attributes = True
services/scheduler/src/tasks/__init__.py
-1
View File
@@ -1 +0,0 @@
"""Task execution system for The Scheduler."""
services/scheduler/src/tasks/executor.py
-278
View File
@@ -1,278 +0,0 @@
"""
Task execution engine for The Scheduler.
Implements minute-based polling with priority-based concurrent execution.
"""
import asyncio
import logging
from datetime import datetime, timezone
from typing import List, Dict, Any, Optional
import psycopg2
from psycopg2.extras import RealDictCursor
import traceback
from src.config import Settings
logger = logging.getLogger(__name__)
# Priority ranges (for reference)
# 1: Emergency/recovery tasks
# 2: Primary system tasks
# 3: Secondary system tasks
# 5: Urgent user-triggered tasks
# 10: High-priority user tasks
# 15: Reserved
# 20: Backup tasks
# 25: Reserved
# 30: Low-priority user tasks
# 40: Cleanup tasks
# 50: Documentation version checks
# 60: Documentation mirroring
# 70+: Future/experimental tasks
MAX_CONCURRENT_TASKS = 5
class TaskExecutor:
"""Executes scheduled tasks based on priority with concurrency control."""
def __init__(self, settings: Settings):
self.settings = settings
self.running_tasks: Dict[int, asyncio.Task] = {} # task_id -> asyncio.Task
self.semaphore = asyncio.Semaphore(MAX_CONCURRENT_TASKS)
def get_db_connection(self):
"""Create database connection."""
return psycopg2.connect(
host=self.settings.postgres_host,
port=self.settings.postgres_port,
database=self.settings.postgres_db,
user=self.settings.postgres_user,
password=self.settings.postgres_password
)
def get_tasks_for_minute(self, now: datetime) -> List[Dict[str, Any]]:
"""
Query database for tasks scheduled for this minute.
Supports wildcards (-1 = any value).
"""
minute = now.minute
hour = now.hour
day = now.day
month = now.month
# Python: Monday=0, Sunday=6; PostgreSQL: Monday=0, Sunday=6 (same)
weekday = now.weekday()
query = """
SELECT
id, task_name, service, executor, priority,
config, timeout_seconds, max_retries, retry_count,
last_run, last_status
FROM scheduled_tasks
WHERE enabled = true
AND (minute = -1 OR minute = %s)
AND (hour = -1 OR hour = %s)
AND (day_of_month = -1 OR day_of_month = %s)
AND (month = -1 OR month = %s)
AND (day_of_week = -1 OR day_of_week = %s)
AND id NOT IN (
SELECT task_id
FROM task_executions
WHERE status = 'running'
)
ORDER BY priority ASC, task_name ASC
"""
with self.get_db_connection() as conn:
with conn.cursor(cursor_factory=RealDictCursor) as cur:
cur.execute(query, (minute, hour, day, month, weekday))
tasks = cur.fetchall()
logger.info(f"Found {len(tasks)} tasks scheduled for {now.strftime('%Y-%m-%d %H:%M')}")
return [dict(task) for task in tasks]
def should_run_task(self, task: Dict[str, Any], now: datetime) -> bool:
"""
Determine if task should run based on last execution.
Prevents running the same task multiple times in the same minute.
"""
if not task['last_run']:
return True
last_run = task['last_run']
if last_run.tzinfo is None:
last_run = last_run.replace(tzinfo=timezone.utc)
if now.tzinfo is None:
now = now.replace(tzinfo=timezone.utc)
# Don't run if already executed this minute
if (last_run.year == now.year and
last_run.month == now.month and
last_run.day == now.day and
last_run.hour == now.hour and
last_run.minute == now.minute):
logger.debug(f"Task {task['task_name']} already ran this minute")
return False
return True
async def execute_task(self, task: Dict[str, Any]):
"""
Execute a single task with timeout and error handling.
Updates task_executions table with results.
"""
task_id = task['id']
task_name = task['task_name']
executor_name = task['executor']
timeout = task.get('timeout_seconds', 3600)
execution_id = None
started_at = datetime.now(timezone.utc)
logger.info(f"[Priority {task['priority']}] Starting task: {task_name} (executor: {executor_name})")
try:
# Create execution record
with self.get_db_connection() as conn:
with conn.cursor() as cur:
cur.execute("""
INSERT INTO task_executions
(task_id, task_name, service, executor, priority,
status, triggered_by, started_at)
VALUES (%s, %s, %s, %s, %s, 'running', 'scheduler', %s)
RETURNING id
""", (task_id, task_name, task['service'], executor_name,
task['priority'], started_at))
execution_id = cur.fetchone()[0]
conn.commit()
# Load and execute the task
output, error = await self._run_executor(executor_name, task, timeout)
completed_at = datetime.now(timezone.utc)
duration = int((completed_at - started_at).total_seconds())
status = 'success' if error is None else 'failed'
# Update execution record
with self.get_db_connection() as conn:
with conn.cursor() as cur:
cur.execute("""
UPDATE task_executions
SET status = %s, completed_at = %s, duration_seconds = %s,
output = %s, error = %s
WHERE id = %s
""", (status, completed_at, duration, output, error, execution_id))
# Update scheduled_tasks
cur.execute("""
UPDATE scheduled_tasks
SET last_run = %s, last_status = %s, last_duration_seconds = %s,
retry_count = 0, updated_at = %s
WHERE id = %s
""", (completed_at, status, duration, completed_at, task_id))
conn.commit()
if error:
logger.error(f"Task {task_name} failed: {error}")
else:
logger.info(f"Task {task_name} completed successfully in {duration}s")
except asyncio.TimeoutError:
logger.error(f"Task {task_name} timed out after {timeout}s")
self._update_execution_status(execution_id, 'timeout',
error=f"Task exceeded timeout of {timeout}s")
except Exception as e:
logger.error(f"Task {task_name} failed with exception: {e}")
logger.error(traceback.format_exc())
self._update_execution_status(execution_id, 'failed',
error=f"{str(e)}\n{traceback.format_exc()}")
finally:
# Remove from running tasks
if task_id in self.running_tasks:
del self.running_tasks[task_id]
async def _run_executor(self, executor_name: str, task: Dict[str, Any], timeout: int) -> tuple[Optional[str], Optional[str]]:
"""
Dynamically load and run the executor module.
Returns (output, error) tuple.
"""
try:
# Import executor dynamically
module_path = f"src.executors.{executor_name}"
module = __import__(module_path, fromlist=['execute'])
if not hasattr(module, 'execute'):
return None, f"Executor {executor_name} missing execute() function"
# Run with timeout
config = task.get('config', {})
result = await asyncio.wait_for(
module.execute(config, self.settings),
timeout=timeout
)
return result, None
except ModuleNotFoundError:
return None, f"Executor module not found: {executor_name}"
except Exception as e:
return None, f"Executor error: {str(e)}\n{traceback.format_exc()}"
def _update_execution_status(self, execution_id: int, status: str, error: str = None):
"""Update execution record with final status."""
if execution_id is None:
return
try:
with self.get_db_connection() as conn:
with conn.cursor() as cur:
cur.execute("""
UPDATE task_executions
SET status = %s, completed_at = %s, error = %s
WHERE id = %s
""", (status, datetime.now(timezone.utc), error, execution_id))
conn.commit()
except Exception as e:
logger.error(f"Failed to update execution status: {e}")
async def process_minute(self):
"""
Main entry point: Process all tasks scheduled for the current minute.
Executes up to MAX_CONCURRENT_TASKS in parallel, prioritized by priority field.
"""
now = datetime.now(timezone.utc)
logger.info(f"Processing tasks for {now.strftime('%Y-%m-%d %H:%M')}")
# Get tasks scheduled for this minute
tasks = self.get_tasks_for_minute(now)
# Filter out tasks that already ran this minute
tasks_to_run = [task for task in tasks if self.should_run_task(task, now)]
if not tasks_to_run:
logger.debug("No tasks to run this minute")
return
logger.info(f"Will execute {len(tasks_to_run)} tasks (max {MAX_CONCURRENT_TASKS} concurrent)")
# Process tasks in priority order, respecting concurrency limit
for task in tasks_to_run:
# Wait for available slot
await self.semaphore.acquire()
# Start task
task_coro = self._execute_with_semaphore(task)
asyncio_task = asyncio.create_task(task_coro)
self.running_tasks[task['id']] = asyncio_task
# Wait for all tasks in this batch to complete
if self.running_tasks:
await asyncio.gather(*self.running_tasks.values(), return_exceptions=True)
async def _execute_with_semaphore(self, task: Dict[str, Any]):
"""Execute task and release semaphore when done."""
try:
await self.execute_task(task)
finally:
self.semaphore.release()
services/scheduler/tests/README.md
-202
View File
@@ -1,202 +0,0 @@
# Scheduler Tests
Comprehensive test suite for The Scheduler using pytest.
## Test Structure
```
tests/
├── conftest.py # Shared fixtures and configuration
├── test_api.py # API endpoint tests
├── test_example_executor.py # Example executor tests
├── test_doc_sync_executor.py # Documentation sync executor tests
└── README.md # This file
```
## Running Tests
### Run all tests
```bash
docker exec scheduler pytest
```
### Run with coverage report
```bash
docker exec scheduler pytest --cov=src --cov-report=term-missing
```
### Run specific test file
```bash
docker exec scheduler pytest tests/test_api.py
```
### Run specific test class
```bash
docker exec scheduler pytest tests/test_api.py::TestHealthEndpoint
```
### Run specific test
```bash
docker exec scheduler pytest tests/test_api.py::TestHealthEndpoint::test_health_endpoint_returns_healthy
```
### Run tests by marker
```bash
# Run only unit tests
docker exec scheduler pytest -m unit
# Run only API tests
docker exec scheduler pytest -m api
# Run only executor tests
docker exec scheduler pytest -m executor
# Run only fast tests (exclude slow)
docker exec scheduler pytest -m "not slow"
```
### Run with verbose output
```bash
docker exec scheduler pytest -v
```
### Run with detailed failure output
```bash
docker exec scheduler pytest -vv --tb=long
```
### Stop on first failure
```bash
docker exec scheduler pytest -x
```
## Test Markers
Tests are organized with pytest markers:
- `@pytest.mark.unit` - Fast unit tests with no external dependencies
- `@pytest.mark.integration` - Integration tests (may use database)
- `@pytest.mark.api` - API endpoint tests
- `@pytest.mark.executor` - Task executor tests
- `@pytest.mark.slow` - Tests that take a while to run
## Coverage Reports
After running tests with coverage:
- **Terminal**: Shows missing lines in terminal output
- **HTML**: Open `htmlcov/index.html` in browser for detailed report
- **JSON**: Machine-readable coverage data in `coverage.json`
View HTML coverage report:
```bash
# From host machine
open /home/jpmschweitzer/docker-data/scheduler/htmlcov/index.html
```
## Writing New Tests
### Test File Naming
- Test files must start with `test_`
- Place in `tests/` directory
- Use descriptive names: `test_<module_name>.py`
### Test Function Naming
- Test functions must start with `test_`
- Use descriptive names: `test_<what_is_being_tested>`
### Using Fixtures
Fixtures are defined in `conftest.py`:
```python
def test_something(test_settings, auth_headers):
# Use fixtures as function parameters
assert test_settings.app_name == "Test Scheduler"
```
### Adding Markers
```python
@pytest.mark.unit
@pytest.mark.api
def test_health_endpoint(client):
response = client.get("/health")
assert response.status_code == 200
```
### Async Tests
```python
@pytest.mark.asyncio
async def test_async_function():
result = await some_async_function()
assert result is not None
```
### Mocking
```python
from unittest.mock import patch, MagicMock
def test_with_mock():
with patch('module.function') as mock_func:
mock_func.return_value = "mocked"
result = call_function_that_uses_it()
assert result == "mocked"
```
## Continuous Integration
These tests are designed to run in CI pipelines:
```yaml
# Example GitHub Actions
- name: Run tests
run: |
docker exec scheduler pytest --cov=src --cov-report=xml
docker exec scheduler pytest --cov=src --cov-report=html
```
## Test Database
Tests use mocked database connections by default. For integration tests that require a real database:
1. Set up a test database
2. Use `POSTGRES_DB=test_scheduler` environment variable
3. Run migrations before tests
4. Clean up after tests
## Troubleshooting
### Tests fail with "ModuleNotFoundError"
```bash
# Install test dependencies
docker exec scheduler /venv/bin/pip install -r requirements.txt
```
### Tests fail with database errors
- Tests use mocked connections by default
- Check that mocks are properly set up in conftest.py
- For integration tests, ensure test database exists
### Coverage not working
```bash
# Reinstall pytest-cov
docker exec scheduler /venv/bin/pip install --upgrade pytest-cov
```
## Best Practices
1. **Fast by default**: Unit tests should be fast (<1s each)
2. **Isolated**: Tests should not depend on each other
3. **Descriptive**: Test names should clearly describe what they test
4. **Single assertion**: Test one thing per test when possible
5. **Use fixtures**: Reuse common setup via fixtures
6. **Mock external deps**: Don't hit real databases/APIs in unit tests
7. **Mark appropriately**: Use markers to categorize tests
## Current Coverage
Run this to see current coverage:
```bash
docker exec scheduler pytest --cov=src --cov-report=term
```
Target: 80%+ code coverage for critical paths
services/scheduler/tests/conftest.py
-223
View File
@@ -1,223 +0,0 @@
"""
Shared pytest fixtures for The Scheduler tests.
"""
import asyncio
import os
from typing import Generator, AsyncGenerator
from unittest.mock import Mock, MagicMock
import pytest
from fastapi.testclient import TestClient
from httpx import AsyncClient, ASGITransport
# Set test environment variables before importing app
os.environ["POSTGRES_HOST"] = "postgres-shared" # Use real postgres for integration tests
os.environ["POSTGRES_DB"] = "test_scheduler"
os.environ["POSTGRES_USER"] = "test_scheduler_user"
os.environ["POSTGRES_PASSWORD"] = "test_password_12345"
os.environ["SCHEDULER_API_KEY"] = "test-api-key-12345"
os.environ["GITEA_USER"] = "test-librarian"
os.environ["GITEA_PASSWORD"] = "test-gitea-token"
os.environ["REDIS_HOST"] = "redis-shared"
from src.main import app
from src.config import Settings, get_settings
# Override settings for tests
@pytest.fixture
def test_settings() -> Settings:
"""Provide test settings loaded from environment variables."""
# Settings are already loaded from environment (set at module level)
return get_settings()
@pytest.fixture
def api_key() -> str:
"""Test API key for authenticated requests."""
return "test-api-key-12345"
@pytest.fixture
def auth_headers(api_key: str) -> dict:
"""Authentication headers for test requests."""
return {"Authorization": f"Bearer {api_key}"}
# Synchronous test client
@pytest.fixture
def client() -> Generator[TestClient, None, None]:
"""
FastAPI test client for synchronous tests.
Note: Some endpoints may fail if they require real database connections.
"""
with TestClient(app) as c:
yield c
# Async test client
@pytest.fixture
async def async_client() -> AsyncGenerator[AsyncClient, None]:
"""
Async HTTP client for testing async endpoints.
Note: Some endpoints may fail if they require real database connections.
"""
async with AsyncClient(
transport=ASGITransport(app=app),
base_url="http://test"
) as ac:
yield ac
# Mock database connection
@pytest.fixture
def mock_db_connection():
"""Mock database connection for testing without real database."""
mock_conn = MagicMock()
mock_cursor = MagicMock()
# Setup context managers
mock_conn.__enter__ = Mock(return_value=mock_conn)
mock_conn.__exit__ = Mock(return_value=None)
mock_conn.cursor.return_value.__enter__ = Mock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = Mock(return_value=None)
return mock_conn, mock_cursor
# Mock scheduler
@pytest.fixture
def mock_scheduler():
"""Mock APScheduler for testing without real scheduler."""
mock = MagicMock()
mock.running = True
mock.get_jobs.return_value = []
return mock
# Mock task executor
@pytest.fixture
def mock_task_executor():
"""Mock TaskExecutor for testing without real executor."""
mock = MagicMock()
return mock
# Sample task data
@pytest.fixture
def sample_task_data() -> dict:
"""Sample task data for testing."""
return {
"id": 1,
"task_name": "test_task",
"service": "scheduler",
"executor": "example_executor",
"priority": 50,
"minute": 0,
"hour": 4,
"day_of_month": -1,
"month": -1,
"day_of_week": -1,
"enabled": True,
"description": "Test task for unit tests",
"config": {
"message": "Test message",
"delay_seconds": 1
},
"max_retries": 3,
"timeout_seconds": 60,
"retry_count": 0,
"last_run": None,
"last_status": None,
"last_duration_seconds": None,
"created_at": "2025-12-07T00:00:00",
"updated_at": "2025-12-07T00:00:00",
"created_by": "test"
}
# Sample doc sync config
@pytest.fixture
def sample_doc_sync_config() -> dict:
"""Sample doc sync configuration for testing."""
return {
"project": "test-project",
"upstream_repo": "https://github.com/test/repo.git",
"docs_paths": ["/docs"],
"gitea_repo": "library/test-docs",
"branch": "main"
}
# Temporary directory for file operations
@pytest.fixture
def temp_work_dir(tmp_path):
"""Temporary directory for testing file operations."""
work_dir = tmp_path / "test-work"
work_dir.mkdir()
return work_dir
# Database fixtures for integration tests
@pytest.fixture(scope="function")
def db_connection():
"""Provide a database connection for integration tests."""
import psycopg2
conn = psycopg2.connect(
host="postgres-shared",
database="test_scheduler",
user="test_scheduler_user",
password="test_password_12345"
)
yield conn
# Cleanup: rollback any uncommitted changes
conn.rollback()
conn.close()
@pytest.fixture(scope="function")
def clean_database(db_connection):
"""Clean test database before each test."""
cursor = db_connection.cursor()
# Delete all test data
cursor.execute("DELETE FROM task_executions")
cursor.execute("DELETE FROM scheduled_tasks")
db_connection.commit()
yield db_connection
# Cleanup after test
cursor.execute("DELETE FROM task_executions")
cursor.execute("DELETE FROM scheduled_tasks")
db_connection.commit()
cursor.close()
@pytest.fixture
def sample_task_in_db(clean_database):
"""Insert a sample task into the test database."""
cursor = clean_database.cursor()
cursor.execute("""
INSERT INTO scheduled_tasks
(task_name, service, executor, priority, minute, hour,
day_of_month, month, day_of_week, enabled, description,
config, max_retries, timeout_seconds, created_by)
VALUES
(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s::jsonb, %s, %s, %s)
RETURNING id
""", (
'test_task', 'scheduler', 'example_executor', 50,
-1, -1, -1, -1, -1, True, 'Test task',
'{"message": "Test", "delay_seconds": 0}',
3, 60, 'test'
))
task_id = cursor.fetchone()[0]
clean_database.commit()
cursor.close()
return task_id
services/scheduler/tests/test_api.py
-236
View File
@@ -1,236 +0,0 @@
"""
Tests for The Scheduler API endpoints.
"""
import pytest
from fastapi.testclient import TestClient
from unittest.mock import patch, MagicMock
import json
@pytest.mark.api
@pytest.mark.unit
class TestHealthEndpoint:
"""Tests for /health endpoint."""
def test_health_endpoint_returns_healthy(self, client: TestClient):
"""Test that health endpoint returns healthy status."""
with patch('src.main.get_scheduler') as mock_get_scheduler:
mock_scheduler = MagicMock()
mock_scheduler.running = True
mock_scheduler.get_jobs.return_value = [MagicMock()]
mock_get_scheduler.return_value = mock_scheduler
response = client.get("/health")
assert response.status_code == 200
data = response.json()
assert data["status"] == "healthy"
assert data["scheduler_running"] is True
assert data["jobs_count"] == 1
def test_health_endpoint_no_auth_required(self, client: TestClient):
"""Test that health endpoint doesn't require authentication."""
with patch('src.main.get_scheduler') as mock_get_scheduler:
mock_scheduler = MagicMock()
mock_scheduler.running = True
mock_get_scheduler.return_value = mock_scheduler
response = client.get("/health")
assert response.status_code == 200
@pytest.mark.api
@pytest.mark.unit
class TestAuthenticationEndpoints:
"""Tests for API authentication."""
def test_missing_api_key_returns_401(self, client: TestClient):
"""Test that missing API key returns 401."""
response = client.get("/tasks")
assert response.status_code == 401
def test_invalid_api_key_returns_403(self, client: TestClient):
"""Test that invalid API key returns 403."""
response = client.get(
"/tasks",
headers={"Authorization": "Bearer wrong-key"}
)
assert response.status_code == 403
def test_valid_api_key_allows_access(self, client: TestClient, auth_headers: dict):
"""Test that valid API key allows access."""
with patch('src.main.get_task_executor') as mock_executor:
mock_executor.return_value.get_db_connection.return_value.__enter__.return_value.cursor.return_value.__enter__.return_value.fetchall.return_value = []
response = client.get("/tasks", headers=auth_headers)
# May fail with 500 due to DB, but should not be 401/403
assert response.status_code not in [401, 403]
@pytest.mark.api
@pytest.mark.unit
class TestTaskEndpoints:
"""Tests for task management endpoints."""
def test_create_task_missing_fields_returns_400(self, client: TestClient, auth_headers: dict):
"""Test that creating task without required fields returns 400."""
incomplete_task = {
"task_name": "test",
# Missing service, executor, priority
}
response = client.post(
"/tasks",
headers=auth_headers,
json=incomplete_task
)
assert response.status_code == 400
def test_create_task_with_valid_data(self, client: TestClient, auth_headers: dict, sample_task_data: dict):
"""Test creating a task with valid data."""
with patch('src.main.get_task_executor') as mock_executor:
mock_conn = MagicMock()
mock_cursor = MagicMock()
# Setup mock to return task data
mock_cursor.fetchone.return_value = {**sample_task_data, "id": 1}
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_executor.return_value.get_db_connection.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_executor.return_value.get_db_connection.return_value.__exit__ = MagicMock(return_value=None)
response = client.post(
"/tasks",
headers=auth_headers,
json=sample_task_data
)
# Verify the call was made
assert mock_cursor.execute.called
# Check that config was JSON-encoded
call_args = mock_cursor.execute.call_args
assert 'config' in call_args[0][1]
def test_trigger_task_endpoint(self, client: TestClient, auth_headers: dict):
"""Test manually triggering a task."""
with patch('src.main.get_task_executor') as mock_executor:
mock_conn = MagicMock()
mock_cursor = MagicMock()
# Mock task retrieval
mock_cursor.fetchone.return_value = {
"task_name": "test_task",
"enabled": True,
"priority": 50,
"executor": "example_executor"
}
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_executor.return_value.get_db_connection.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_executor.return_value.get_db_connection.return_value.__exit__ = MagicMock(return_value=None)
with patch('asyncio.create_task'):
response = client.post(
"/tasks/test_task/trigger",
headers=auth_headers
)
# Should return success message
if response.status_code == 200:
data = response.json()
assert data["task_name"] == "test_task"
@pytest.mark.api
@pytest.mark.unit
class TestStatsEndpoint:
"""Tests for /stats endpoint."""
def test_stats_endpoint_requires_auth(self, client: TestClient):
"""Test that stats endpoint requires authentication."""
response = client.get("/stats")
assert response.status_code == 401
def test_stats_endpoint_returns_metrics(self, client: TestClient, auth_headers: dict):
"""Test that stats endpoint returns system metrics."""
with patch('src.main.get_scheduler') as mock_scheduler, \
patch('src.main.get_task_executor') as mock_executor:
mock_scheduler.return_value.running = True
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_cursor.fetchone.side_effect = [
{"count": 3}, # enabled tasks
{"count": 0}, # running tasks
]
mock_cursor.fetchall.return_value = [
{"status": "success", "count": 10},
{"status": "failed", "count": 1}
]
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_executor.return_value.get_db_connection.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_executor.return_value.get_db_connection.return_value.__exit__ = MagicMock(return_value=None)
response = client.get("/stats", headers=auth_headers)
if response.status_code == 200:
data = response.json()
assert "scheduler_running" in data
assert "tasks_enabled" in data
assert "concurrent_limit" in data
@pytest.mark.api
@pytest.mark.unit
class TestExecutionHistoryEndpoint:
"""Tests for /executions endpoint."""
def test_executions_endpoint_returns_history(self, client: TestClient, auth_headers: dict):
"""Test that executions endpoint returns execution history."""
with patch('src.main.get_task_executor') as mock_executor:
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_cursor.fetchall.return_value = [
{
"id": 1,
"task_name": "test_task",
"status": "success",
"duration_seconds": 5
}
]
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_executor.return_value.get_db_connection.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_executor.return_value.get_db_connection.return_value.__exit__ = MagicMock(return_value=None)
response = client.get("/executions", headers=auth_headers)
if response.status_code == 200:
data = response.json()
assert "executions" in data
assert "count" in data
def test_executions_filter_by_task_name(self, client: TestClient, auth_headers: dict):
"""Test filtering executions by task name."""
with patch('src.main.get_task_executor') as mock_executor:
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_cursor.fetchall.return_value = []
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_executor.return_value.get_db_connection.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_executor.return_value.get_db_connection.return_value.__exit__ = MagicMock(return_value=None)
response = client.get(
"/executions?task_name=test_task&limit=10",
headers=auth_headers
)
# Should execute query with filters
assert mock_cursor.execute.called
services/scheduler/tests/test_api_comprehensive.py
-311
View File
@@ -1,311 +0,0 @@
"""
Comprehensive API tests to improve coverage of main.py.
"""
import pytest
from fastapi.testclient import TestClient
from unittest.mock import patch, MagicMock
import json
@pytest.mark.api
@pytest.mark.unit
class TestTaskCRUDOperations:
"""Comprehensive CRUD tests for task endpoints."""
def test_list_tasks_empty(self, client: TestClient, auth_headers: dict):
"""Test listing tasks when none exist."""
with patch('src.main.get_task_executor') as mock_executor:
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_cursor.fetchall.return_value = []
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_executor.return_value.get_db_connection.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_executor.return_value.get_db_connection.return_value.__exit__ = MagicMock(return_value=None)
response = client.get("/tasks", headers=auth_headers)
if response.status_code == 200:
data = response.json()
assert "tasks" in data
assert data["count"] == 0
def test_list_tasks_with_filters(self, client: TestClient, auth_headers: dict):
"""Test listing tasks with enabled and service filters."""
with patch('src.main.get_task_executor') as mock_executor:
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_cursor.fetchall.return_value = []
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_executor.return_value.get_db_connection.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_executor.return_value.get_db_connection.return_value.__exit__ = MagicMock(return_value=None)
response = client.get(
"/tasks?enabled=true&service=scheduler",
headers=auth_headers
)
# Should execute filtered query
assert mock_cursor.execute.called
call_args = str(mock_cursor.execute.call_args)
assert "enabled" in call_args.lower() or response.status_code in [200, 500]
def test_get_task_details_not_found(self, client: TestClient, auth_headers: dict):
"""Test getting details for non-existent task."""
with patch('src.main.get_task_executor') as mock_executor:
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_cursor.fetchone.return_value = None
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_executor.return_value.get_db_connection.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_executor.return_value.get_db_connection.return_value.__exit__ = MagicMock(return_value=None)
response = client.get("/tasks/nonexistent", headers=auth_headers)
assert response.status_code == 404
def test_update_task(self, client: TestClient, auth_headers: dict):
"""Test updating a task."""
with patch('src.main.get_task_executor') as mock_executor:
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_cursor.fetchone.return_value = {
"task_name": "test",
"priority": 60
}
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_executor.return_value.get_db_connection.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_executor.return_value.get_db_connection.return_value.__exit__ = MagicMock(return_value=None)
response = client.put(
"/tasks/test",
headers=auth_headers,
json={"priority": 60}
)
# Should have attempted update
assert mock_cursor.execute.called
def test_update_task_not_found(self, client: TestClient, auth_headers: dict):
"""Test updating non-existent task."""
with patch('src.main.get_task_executor') as mock_executor:
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_cursor.fetchone.return_value = None
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_executor.return_value.get_db_connection.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_executor.return_value.get_db_connection.return_value.__exit__ = MagicMock(return_value=None)
response = client.put(
"/tasks/nonexistent",
headers=auth_headers,
json={"priority": 60}
)
assert response.status_code == 404
def test_update_task_no_fields(self, client: TestClient, auth_headers: dict):
"""Test updating task with no valid fields."""
response = client.put(
"/tasks/test",
headers=auth_headers,
json={"invalid_field": "value"}
)
assert response.status_code == 400
def test_delete_task(self, client: TestClient, auth_headers: dict):
"""Test deleting a task."""
with patch('src.main.get_task_executor') as mock_executor:
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_cursor.fetchone.return_value = ("test_task",)
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_executor.return_value.get_db_connection.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_executor.return_value.get_db_connection.return_value.__exit__ = MagicMock(return_value=None)
response = client.delete("/tasks/test_task", headers=auth_headers)
if response.status_code == 200:
data = response.json()
assert "deleted successfully" in data["message"].lower()
def test_delete_task_not_found(self, client: TestClient, auth_headers: dict):
"""Test deleting non-existent task."""
with patch('src.main.get_task_executor') as mock_executor:
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_cursor.fetchone.return_value = None
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_executor.return_value.get_db_connection.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_executor.return_value.get_db_connection.return_value.__exit__ = MagicMock(return_value=None)
response = client.delete("/tasks/nonexistent", headers=auth_headers)
assert response.status_code == 404
@pytest.mark.api
@pytest.mark.unit
class TestTriggerEndpoint:
"""Tests for task trigger endpoint."""
def test_trigger_disabled_task(self, client: TestClient, auth_headers: dict):
"""Test triggering a disabled task."""
with patch('src.main.get_task_executor') as mock_executor:
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_cursor.fetchone.return_value = {
"task_name": "test",
"enabled": False
}
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_executor.return_value.get_db_connection.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_executor.return_value.get_db_connection.return_value.__exit__ = MagicMock(return_value=None)
response = client.post("/tasks/test/trigger", headers=auth_headers)
assert response.status_code == 400
def test_trigger_nonexistent_task(self, client: TestClient, auth_headers: dict):
"""Test triggering a task that doesn't exist."""
with patch('src.main.get_task_executor') as mock_executor:
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_cursor.fetchone.return_value = None
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_executor.return_value.get_db_connection.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_executor.return_value.get_db_connection.return_value.__exit__ = MagicMock(return_value=None)
response = client.post("/tasks/nonexistent/trigger", headers=auth_headers)
assert response.status_code == 404
@pytest.mark.api
@pytest.mark.unit
class TestLegacyEndpoints:
"""Tests for deprecated/legacy endpoints."""
def test_trigger_backup_endpoint(self, client: TestClient, auth_headers: dict):
"""Test legacy backup trigger endpoint."""
response = client.post("/tasks/backup", headers=auth_headers)
# Should return not implemented status
if response.status_code == 200:
data = response.json()
assert data["status"] == "not_implemented"
def test_trigger_docs_update_endpoint(self, client: TestClient, auth_headers: dict):
"""Test legacy docs update endpoint."""
response = client.post("/tasks/docs/update", headers=auth_headers)
if response.status_code == 200:
data = response.json()
assert data["status"] == "not_implemented"
def test_trigger_docs_update_with_project(self, client: TestClient, auth_headers: dict):
"""Test legacy docs update with project filter."""
response = client.post(
"/tasks/docs/update?project=test",
headers=auth_headers
)
if response.status_code == 200:
data = response.json()
assert "test" in data["message"].lower()
def test_check_doc_versions_endpoint(self, client: TestClient, auth_headers: dict):
"""Test legacy version check endpoint."""
response = client.post("/tasks/docs/check-versions", headers=auth_headers)
if response.status_code == 200:
data = response.json()
assert data["status"] == "not_implemented"
def test_trigger_cleanup_endpoint(self, client: TestClient, auth_headers: dict):
"""Test legacy cleanup endpoint."""
response = client.post("/tasks/cleanup", headers=auth_headers)
if response.status_code == 200:
data = response.json()
assert data["status"] == "not_implemented"
@pytest.mark.api
@pytest.mark.unit
class TestExecutionFiltering:
"""Tests for execution history filtering."""
def test_filter_by_service(self, client: TestClient, auth_headers: dict):
"""Test filtering executions by service."""
with patch('src.main.get_task_executor') as mock_executor:
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_cursor.fetchall.return_value = []
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_executor.return_value.get_db_connection.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_executor.return_value.get_db_connection.return_value.__exit__ = MagicMock(return_value=None)
response = client.get(
"/executions?service=scheduler",
headers=auth_headers
)
assert mock_cursor.execute.called
def test_filter_by_status(self, client: TestClient, auth_headers: dict):
"""Test filtering executions by status."""
with patch('src.main.get_task_executor') as mock_executor:
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_cursor.fetchall.return_value = []
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_executor.return_value.get_db_connection.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_executor.return_value.get_db_connection.return_value.__exit__ = MagicMock(return_value=None)
response = client.get(
"/executions?status=success",
headers=auth_headers
)
assert mock_cursor.execute.called
def test_custom_limit(self, client: TestClient, auth_headers: dict):
"""Test custom limit for executions."""
with patch('src.main.get_task_executor') as mock_executor:
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_cursor.fetchall.return_value = []
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_executor.return_value.get_db_connection.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_executor.return_value.get_db_connection.return_value.__exit__ = MagicMock(return_value=None)
response = client.get("/executions?limit=50", headers=auth_headers)
if response.status_code == 200:
data = response.json()
assert data["limit"] == 50
services/scheduler/tests/test_config_backup_executor.py
-214
View File
@@ -1,214 +0,0 @@
"""
Tests for the config backup executor.
"""
import pytest
from pathlib import Path
from unittest.mock import AsyncMock, patch, MagicMock, mock_open
from src.executors import config_backup_executor
from src.config import Settings
@pytest.mark.executor
@pytest.mark.unit
class TestConfigBackupExecutor:
"""Tests for config_backup_executor module."""
@pytest.mark.asyncio
async def test_executor_requires_config_fields(self, test_settings: Settings):
"""Test that executor validates required config."""
incomplete_config = {
"sources": []
# Missing backup_dir
}
with pytest.raises((ValueError, KeyError)):
await config_backup_executor.execute(incomplete_config, test_settings)
@pytest.mark.asyncio
async def test_executor_with_minimal_config(self, test_settings: Settings, tmp_path: Path):
"""Test executor with minimal valid configuration."""
backup_dir = tmp_path / "backups"
backup_dir.mkdir()
source_dir = tmp_path / "source"
source_dir.mkdir()
(source_dir / "test.txt").write_text("test content")
config = {
"sources": [{
"path": str(source_dir),
"name": "test-source",
"excludes": []
}],
"backup_dir": str(backup_dir),
"compress": True,
"retention_days": 30
}
with patch('src.executors.config_backup_executor.Path') as mock_path_cls:
# Setup path mocking
mock_source = MagicMock()
mock_source.exists.return_value = True
mock_source.is_dir.return_value = True
mock_source.iterdir.return_value = [MagicMock(name="test.txt")]
mock_backup = MagicMock()
mock_backup.mkdir = MagicMock()
def path_side_effect(p):
if str(p) == str(source_dir):
return mock_source
elif str(p) == str(backup_dir):
return mock_backup
return MagicMock()
mock_path_cls.side_effect = path_side_effect
with patch('tarfile.open'), \
patch('src.executors.config_backup_executor._cleanup_old_backups'):
result = await config_backup_executor.execute(config, test_settings)
assert "backed up" in result.lower() or "success" in result.lower()
@pytest.mark.asyncio
async def test_executor_excludes_patterns(self, test_settings: Settings, tmp_path: Path):
"""Test that executor respects exclude patterns."""
backup_dir = tmp_path / "backups"
source_dir = tmp_path / "source"
config = {
"sources": [{
"path": str(source_dir),
"name": "test",
"excludes": ["*.log", "cache/*"]
}],
"backup_dir": str(backup_dir),
"compress": True
}
with patch('src.executors.config_backup_executor.Path'), \
patch('tarfile.open') as mock_tar, \
patch('src.executors.config_backup_executor._cleanup_old_backups'):
# Mock tarfile
mock_tar_obj = MagicMock()
mock_tar.return_value.__enter__ = MagicMock(return_value=mock_tar_obj)
mock_tar.return_value.__exit__ = MagicMock(return_value=None)
try:
await config_backup_executor.execute(config, test_settings)
except Exception:
# May fail due to mocking complexity, but that's ok
pass
# Should have attempted to create tarfile
# assert mock_tar.called # Would check if it was actually called
@pytest.mark.asyncio
async def test_executor_handles_missing_source(self, test_settings: Settings, tmp_path: Path):
"""Test executor handles missing source directory."""
backup_dir = tmp_path / "backups"
backup_dir.mkdir()
config = {
"sources": [{
"path": "/nonexistent/path",
"name": "missing",
"excludes": []
}],
"backup_dir": str(backup_dir),
"compress": True
}
with patch('src.executors.config_backup_executor.Path') as mock_path_cls:
mock_source = MagicMock()
mock_source.exists.return_value = False
mock_path_cls.return_value = mock_source
result = await config_backup_executor.execute(config, test_settings)
# Should skip non-existent sources
assert "skipped" in result.lower() or "not found" in result.lower() or "0" in result
@pytest.mark.asyncio
async def test_cleanup_old_backups(self, tmp_path: Path):
"""Test cleanup of old backup files."""
backup_dir = tmp_path / "backups"
backup_dir.mkdir()
# Create some "old" backup files
old_backup = backup_dir / "backup-2020-01-01.tar.gz"
old_backup.write_text("old")
recent_backup = backup_dir / "backup-2025-12-01.tar.gz"
recent_backup.write_text("recent")
with patch('src.executors.config_backup_executor.Path') as mock_path_cls:
mock_backup_dir = MagicMock()
mock_old_file = MagicMock()
mock_old_file.name = "backup-2020-01-01.tar.gz"
mock_old_file.stat.return_value.st_mtime = 0 # Very old
mock_recent_file = MagicMock()
mock_recent_file.name = "backup-2025-12-01.tar.gz"
mock_recent_file.stat.return_value.st_mtime = 999999999999 # Recent
mock_backup_dir.glob.return_value = [mock_old_file, mock_recent_file]
mock_path_cls.return_value = mock_backup_dir
config_backup_executor._cleanup_old_backups(mock_backup_dir, retention_days=7)
# Old file should be removed
mock_old_file.unlink.assert_called_once()
@pytest.mark.executor
@pytest.mark.unit
class TestConfigBackupHelpers:
"""Tests for helper functions."""
def test_tar_filter_excludes_cache(self):
"""Test that tar filter excludes cache directories."""
excludes = ["*/cache/*", "*.log"]
filter_func = config_backup_executor._create_tar_filter(excludes)
# Mock tarinfo for cache file
cache_tarinfo = MagicMock()
cache_tarinfo.name = "data/cache/temp.txt"
result = filter_func(cache_tarinfo)
assert result is None # Should exclude
def test_tar_filter_includes_normal_files(self):
"""Test that tar filter includes normal files."""
excludes = ["*/cache/*"]
filter_func = config_backup_executor._create_tar_filter(excludes)
# Mock tarinfo for normal file
normal_tarinfo = MagicMock()
normal_tarinfo.name = "data/config.json"
result = filter_func(normal_tarinfo)
assert result == normal_tarinfo # Should include
def test_tar_filter_with_wildcard_patterns(self):
"""Test tar filter with various wildcard patterns."""
excludes = ["*.log", "*.tmp", "temp/*"]
filter_func = config_backup_executor._create_tar_filter(excludes)
# Log file
log_tarinfo = MagicMock()
log_tarinfo.name = "app.log"
assert filter_func(log_tarinfo) is None
# Temp file
tmp_tarinfo = MagicMock()
tmp_tarinfo.name = "cache.tmp"
assert filter_func(tmp_tarinfo) is None
# Normal file
normal_tarinfo = MagicMock()
normal_tarinfo.name = "config.json"
assert filter_func(normal_tarinfo) == normal_tarinfo
services/scheduler/tests/test_database_integration.py
-307
View File
@@ -1,307 +0,0 @@
"""
Database integration tests using real test database.
These tests require the test database to be set up.
"""
import pytest
from fastapi.testclient import TestClient
@pytest.mark.integration
class TestDatabaseTaskOperations:
"""Integration tests for task CRUD with real database."""
def test_create_task_in_database(self, client: TestClient, auth_headers: dict, clean_database):
"""Test creating a task writes to database."""
task_data = {
"task_name": "integration_test_task",
"service": "scheduler",
"executor": "example_executor",
"priority": 40,
"description": "Integration test task",
"config": {"message": "Test", "delay_seconds": 0}
}
response = client.post("/tasks", headers=auth_headers, json=task_data)
assert response.status_code == 200
data = response.json()
assert data["task_name"] == "integration_test_task"
assert data["priority"] == 40
# Verify it's in database
cursor = clean_database.cursor()
cursor.execute("SELECT task_name, priority FROM scheduled_tasks WHERE task_name = %s",
("integration_test_task",))
result = cursor.fetchone()
assert result is not None
assert result[0] == "integration_test_task"
assert result[1] == 40
def test_list_tasks_from_database(self, client: TestClient, auth_headers: dict, sample_task_in_db):
"""Test listing tasks reads from database."""
response = client.get("/tasks", headers=auth_headers)
assert response.status_code == 200
data = response.json()
assert data["count"] >= 1
assert any(task["task_name"] == "test_task" for task in data["tasks"])
def test_get_task_details_from_database(self, client: TestClient, auth_headers: dict, sample_task_in_db):
"""Test getting task details from database."""
response = client.get("/tasks/test_task", headers=auth_headers)
assert response.status_code == 200
data = response.json()
assert data["task_name"] == "test_task"
assert data["executor"] == "example_executor"
assert data["priority"] == 50
def test_update_task_in_database(self, client: TestClient, auth_headers: dict, sample_task_in_db, clean_database):
"""Test updating a task modifies database."""
update_data = {"priority": 99, "description": "Updated description"}
response = client.put("/tasks/test_task", headers=auth_headers, json=update_data)
assert response.status_code == 200
data = response.json()
assert data["priority"] == 99
assert data["description"] == "Updated description"
# Verify in database
cursor = clean_database.cursor()
cursor.execute("SELECT priority, description FROM scheduled_tasks WHERE task_name = %s",
("test_task",))
result = cursor.fetchone()
assert result[0] == 99
assert result[1] == "Updated description"
def test_delete_task_from_database(self, client: TestClient, auth_headers: dict, sample_task_in_db, clean_database):
"""Test deleting a task removes from database."""
response = client.delete("/tasks/test_task", headers=auth_headers)
assert response.status_code == 200
# Verify removed from database
cursor = clean_database.cursor()
cursor.execute("SELECT COUNT(*) FROM scheduled_tasks WHERE task_name = %s",
("test_task",))
count = cursor.fetchone()[0]
assert count == 0
def test_filter_tasks_by_service(self, client: TestClient, auth_headers: dict, clean_database):
"""Test filtering tasks by service."""
# Create tasks with different services
cursor = clean_database.cursor()
cursor.execute("""
INSERT INTO scheduled_tasks (task_name, service, executor, priority)
VALUES ('task1', 'scheduler', 'example_executor', 50),
('task2', 'backup', 'backup_executor', 30)
""")
clean_database.commit()
response = client.get("/tasks?service=scheduler", headers=auth_headers)
assert response.status_code == 200
data = response.json()
assert all(task["service"] == "scheduler" for task in data["tasks"])
def test_filter_tasks_by_enabled(self, client: TestClient, auth_headers: dict, clean_database):
"""Test filtering tasks by enabled status."""
cursor = clean_database.cursor()
cursor.execute("""
INSERT INTO scheduled_tasks (task_name, service, executor, priority, enabled)
VALUES ('enabled_task', 'scheduler', 'example_executor', 50, true),
('disabled_task', 'scheduler', 'example_executor', 50, false)
""")
clean_database.commit()
response = client.get("/tasks?enabled=true", headers=auth_headers)
assert response.status_code == 200
data = response.json()
assert all(task["enabled"] is True for task in data["tasks"])
@pytest.mark.integration
class TestDatabaseTaskExecution:
"""Integration tests for task execution with database tracking."""
@pytest.mark.asyncio
async def test_execute_task_creates_execution_record(self, sample_task_in_db, clean_database):
"""Test that executing a task creates execution record."""
from src.tasks.executor import TaskExecutor
from src.config import get_settings
settings = get_settings()
executor = TaskExecutor(settings)
# Get task from database
cursor = clean_database.cursor()
cursor.execute("""
SELECT id, task_name, service, executor, priority, minute, hour,
day_of_month, month, day_of_week, enabled, description,
config, max_retries, timeout_seconds, retry_count,
last_run, last_status, last_duration_seconds, created_by
FROM scheduled_tasks WHERE id = %s
""", (sample_task_in_db,))
row = cursor.fetchone()
task = {
"id": row[0],
"task_name": row[1],
"service": row[2],
"executor": row[3],
"priority": row[4],
"minute": row[5],
"hour": row[6],
"day_of_month": row[7],
"month": row[8],
"day_of_week": row[9],
"enabled": row[10],
"description": row[11],
"config": row[12],
"max_retries": row[13],
"timeout_seconds": row[14],
"retry_count": row[15],
"last_run": row[16],
"last_status": row[17],
"last_duration_seconds": row[18],
"created_by": row[19]
}
# Execute task
await executor.execute_task(task)
# Verify execution record was created
cursor.execute("""
SELECT status, task_name, executor FROM task_executions
WHERE task_id = %s ORDER BY id DESC LIMIT 1
""", (sample_task_in_db,))
execution = cursor.fetchone()
assert execution is not None
assert execution[0] in ["success", "failed", "timeout"]
assert execution[1] == "test_task"
assert execution[2] == "example_executor"
@pytest.mark.integration
class TestDatabaseExecutionHistory:
"""Integration tests for execution history endpoints."""
def test_get_execution_history(self, client: TestClient, auth_headers: dict, clean_database):
"""Test retrieving execution history from database."""
# Create test execution records
cursor = clean_database.cursor()
# First create a task
cursor.execute("""
INSERT INTO scheduled_tasks (task_name, service, executor, priority)
VALUES ('history_task', 'scheduler', 'example_executor', 50)
RETURNING id
""")
task_id = cursor.fetchone()[0]
# Create execution records
cursor.execute("""
INSERT INTO task_executions
(task_id, task_name, service, executor, priority, status, duration_seconds)
VALUES
(%s, 'history_task', 'scheduler', 'example_executor', 50, 'success', 5),
(%s, 'history_task', 'scheduler', 'example_executor', 50, 'success', 3),
(%s, 'history_task', 'scheduler', 'example_executor', 50, 'failed', 2)
""", (task_id, task_id, task_id))
clean_database.commit()
response = client.get("/executions", headers=auth_headers)
assert response.status_code == 200
data = response.json()
assert data["count"] >= 3
assert any(ex["task_name"] == "history_task" for ex in data["executions"])
def test_filter_executions_by_task_name(self, client: TestClient, auth_headers: dict, clean_database):
"""Test filtering execution history by task name."""
cursor = clean_database.cursor()
# Create tasks
cursor.execute("""
INSERT INTO scheduled_tasks (task_name, service, executor, priority)
VALUES ('task_a', 'scheduler', 'example_executor', 50),
('task_b', 'scheduler', 'example_executor', 50)
RETURNING id
""")
task_ids = [row[0] for row in cursor.fetchall()]
# Create executions
cursor.execute("""
INSERT INTO task_executions (task_id, task_name, service, executor, priority, status)
VALUES (%s, 'task_a', 'scheduler', 'example_executor', 50, 'success'),
(%s, 'task_b', 'scheduler', 'example_executor', 50, 'success')
""", task_ids)
clean_database.commit()
response = client.get("/executions?task_name=task_a", headers=auth_headers)
assert response.status_code == 200
data = response.json()
assert all(ex["task_name"] == "task_a" for ex in data["executions"])
def test_filter_executions_by_status(self, client: TestClient, auth_headers: dict, clean_database):
"""Test filtering execution history by status."""
cursor = clean_database.cursor()
cursor.execute("""
INSERT INTO scheduled_tasks (task_name, service, executor, priority)
VALUES ('status_task', 'scheduler', 'example_executor', 50)
RETURNING id
""")
task_id = cursor.fetchone()[0]
cursor.execute("""
INSERT INTO task_executions (task_id, task_name, service, executor, priority, status)
VALUES (%s, 'status_task', 'scheduler', 'example_executor', 50, 'success'),
(%s, 'status_task', 'scheduler', 'example_executor', 50, 'failed')
""", (task_id, task_id))
clean_database.commit()
response = client.get("/executions?status=success", headers=auth_headers)
assert response.status_code == 200
data = response.json()
success_executions = [ex for ex in data["executions"] if ex["task_name"] == "status_task"]
assert all(ex["status"] == "success" for ex in success_executions)
@pytest.mark.integration
class TestDatabaseStats:
"""Integration tests for stats endpoint with database."""
def test_stats_endpoint_with_database(self, client: TestClient, auth_headers: dict, clean_database):
"""Test stats endpoint returns accurate database counts."""
cursor = clean_database.cursor()
# Create test data
cursor.execute("""
INSERT INTO scheduled_tasks (task_name, service, executor, priority, enabled)
VALUES ('enabled_1', 'scheduler', 'example_executor', 50, true),
('enabled_2', 'scheduler', 'example_executor', 50, true),
('disabled_1', 'scheduler', 'example_executor', 50, false)
RETURNING id
""")
task_ids = [row[0] for row in cursor.fetchall()]
cursor.execute("""
INSERT INTO task_executions (task_id, task_name, service, executor, priority, status)
VALUES (%s, 'enabled_1', 'scheduler', 'example_executor', 50, 'success'),
(%s, 'enabled_2', 'scheduler', 'example_executor', 50, 'success'),
(%s, 'enabled_1', 'scheduler', 'example_executor', 50, 'failed')
""", task_ids)
clean_database.commit()
response = client.get("/stats", headers=auth_headers)
assert response.status_code == 200
data = response.json()
assert data["tasks_enabled"] >= 2
assert "execution_stats_24h" in data
services/scheduler/tests/test_doc_sync_executor.py
-240
View File
@@ -1,240 +0,0 @@
"""
Tests for the documentation sync executor.
"""
import pytest
from pathlib import Path
from unittest.mock import AsyncMock, patch, MagicMock
from src.executors import doc_sync_executor
from src.config import Settings
@pytest.mark.executor
@pytest.mark.unit
class TestDocSyncExecutor:
"""Tests for doc_sync_executor module."""
@pytest.mark.asyncio
async def test_executor_requires_all_config_fields(self, test_settings: Settings):
"""Test that executor validates required config fields."""
incomplete_config = {
"project": "test",
# Missing other required fields
}
with pytest.raises(ValueError, match="Missing required config"):
await doc_sync_executor.execute(incomplete_config, test_settings)
@pytest.mark.asyncio
async def test_executor_validates_project_name(self, test_settings: Settings):
"""Test that executor requires project name."""
config = {
"project": "",
"upstream_repo": "https://github.com/test/repo.git",
"gitea_repo": "library/test"
}
with pytest.raises(ValueError):
await doc_sync_executor.execute(config, test_settings)
@pytest.mark.asyncio
async def test_executor_successful_sync_entire_repo(
self,
test_settings: Settings,
sample_doc_sync_config: dict,
temp_work_dir: Path
):
"""Test successful documentation sync of entire repository."""
# Modify config to sync entire repo
config = {**sample_doc_sync_config, "docs_paths": []}
with patch('src.executors.doc_sync_executor._run_command', new_callable=AsyncMock) as mock_run, \
patch('src.executors.doc_sync_executor._get_git_commit', new_callable=AsyncMock) as mock_get_commit, \
patch('src.executors.doc_sync_executor.shutil') as mock_shutil, \
patch('src.executors.doc_sync_executor.Path') as mock_path:
# Mock git commit hash
mock_get_commit.return_value = "abc123def456"
# Mock path operations
mock_work_dir = MagicMock()
mock_upstream_dir = MagicMock()
mock_gitea_dir = MagicMock()
# Setup directory mocking
mock_upstream_dir.iterdir.return_value = [
MagicMock(name=".git", is_dir=lambda: True),
MagicMock(name="README.md", is_dir=lambda: False),
MagicMock(name="docs", is_dir=lambda: True),
]
mock_gitea_dir.iterdir.return_value = [
MagicMock(name=".git", is_dir=lambda: True)
]
mock_path.return_value = mock_work_dir
mock_work_dir.__truediv__.side_effect = [mock_upstream_dir, mock_gitea_dir]
# Mock git status to show changes
mock_run.side_effect = [
"", # git clone upstream
"", # git rev-parse HEAD
"", # git clone gitea
"", # git add
"M README.md\n", # git status --porcelain (has changes)
"", # git commit
"", # git tag
"", # git push branch
"", # git push tag
]
result = await doc_sync_executor.execute(config, test_settings)
assert "Successfully synced" in result
assert "test-project" in result.lower()
assert "entire repository" in result
@pytest.mark.asyncio
async def test_executor_sync_specific_paths(
self,
test_settings: Settings,
sample_doc_sync_config: dict
):
"""Test syncing only specific paths."""
config = {**sample_doc_sync_config, "docs_paths": ["/docs", "/examples"]}
with patch('src.executors.doc_sync_executor._run_command', new_callable=AsyncMock) as mock_run, \
patch('src.executors.doc_sync_executor._get_git_commit', new_callable=AsyncMock) as mock_get_commit, \
patch('src.executors.doc_sync_executor.shutil') as mock_shutil, \
patch('src.executors.doc_sync_executor.Path') as mock_path:
mock_get_commit.return_value = "abc123"
# Mock path operations
mock_upstream_dir = MagicMock()
mock_gitea_dir = MagicMock()
mock_docs = MagicMock(name="docs")
mock_docs.exists.return_value = True
mock_docs.is_dir.return_value = True
mock_docs.name = "docs"
mock_examples = MagicMock(name="examples")
mock_examples.exists.return_value = True
mock_examples.is_dir.return_value = True
mock_examples.name = "examples"
mock_upstream_dir.__truediv__.side_effect = [mock_docs, mock_examples]
mock_gitea_dir.iterdir.return_value = []
# Mock git status to show changes
async def run_command_side_effect(cmd, **kwargs):
if 'status' in cmd:
return "M docs/README.md\n"
return ""
mock_run.side_effect = run_command_side_effect
result = await doc_sync_executor.execute(config, test_settings)
# Should mention the specific paths
assert "docs" in result.lower() or "Successfully synced" in result
@pytest.mark.asyncio
async def test_executor_handles_no_changes(
self,
test_settings: Settings,
sample_doc_sync_config: dict
):
"""Test that executor handles case where there are no changes."""
with patch('src.executors.doc_sync_executor._run_command', new_callable=AsyncMock) as mock_run, \
patch('src.executors.doc_sync_executor._get_git_commit', new_callable=AsyncMock) as mock_get_commit, \
patch('src.executors.doc_sync_executor.shutil'), \
patch('src.executors.doc_sync_executor.Path') as mock_path:
mock_get_commit.return_value = "abc123"
# Mock empty git status (no changes)
async def run_command_side_effect(cmd, cwd=None, capture=False):
if 'status' in cmd and '--porcelain' in cmd:
return "" # No changes
return ""
mock_run.side_effect = run_command_side_effect
# Setup minimal mocking
mock_work_dir = MagicMock()
mock_upstream_dir = MagicMock()
mock_gitea_dir = MagicMock()
mock_gitea_dir.iterdir.return_value = []
mock_upstream_dir.iterdir.return_value = []
result = await doc_sync_executor.execute(sample_doc_sync_config, test_settings)
assert "already up to date" in result.lower() or "no changes" in result.lower()
@pytest.mark.asyncio
async def test_executor_cleanup_on_error(
self,
test_settings: Settings,
sample_doc_sync_config: dict
):
"""Test that executor cleans up on error."""
with patch('src.executors.doc_sync_executor._run_command', new_callable=AsyncMock) as mock_run, \
patch('src.executors.doc_sync_executor.shutil') as mock_shutil, \
patch('src.executors.doc_sync_executor.Path') as mock_path:
# Make git clone fail
mock_run.side_effect = Exception("Git clone failed")
mock_work_dir = MagicMock()
mock_work_dir.exists.return_value = True
mock_path.return_value = mock_work_dir
with pytest.raises(Exception, match="Git clone failed"):
await doc_sync_executor.execute(sample_doc_sync_config, test_settings)
# Should have attempted cleanup
mock_shutil.rmtree.assert_called()
@pytest.mark.executor
@pytest.mark.unit
class TestDocSyncHelpers:
"""Tests for doc_sync_executor helper functions."""
@pytest.mark.asyncio
async def test_run_command_success(self):
"""Test that _run_command executes successfully."""
with patch('asyncio.create_subprocess_exec', new_callable=AsyncMock) as mock_exec:
mock_proc = AsyncMock()
mock_proc.returncode = 0
mock_proc.communicate.return_value = (b"output", b"")
mock_exec.return_value = mock_proc
result = await doc_sync_executor._run_command(
["echo", "test"],
capture=True
)
assert result == "output"
@pytest.mark.asyncio
async def test_run_command_failure(self):
"""Test that _run_command raises on failure."""
with patch('asyncio.create_subprocess_exec', new_callable=AsyncMock) as mock_exec:
mock_proc = AsyncMock()
mock_proc.returncode = 1
mock_proc.communicate.return_value = (b"", b"error message")
mock_exec.return_value = mock_proc
with pytest.raises(Exception, match="Command failed"):
await doc_sync_executor._run_command(["false"])
@pytest.mark.asyncio
async def test_get_git_commit(self, temp_work_dir: Path):
"""Test getting git commit hash."""
with patch('src.executors.doc_sync_executor._run_command', new_callable=AsyncMock) as mock_run:
mock_run.return_value = "abc123def456\n"
commit = await doc_sync_executor._get_git_commit(temp_work_dir)
assert commit == "abc123def456"
mock_run.assert_called_once()
services/scheduler/tests/test_example_executor.py
-80
View File
@@ -1,80 +0,0 @@
"""
Tests for the example executor.
"""
import pytest
import asyncio
from src.executors import example_executor
from src.config import Settings
@pytest.mark.executor
@pytest.mark.unit
class TestExampleExecutor:
"""Tests for example_executor module."""
@pytest.mark.asyncio
async def test_executor_returns_success_message(self, test_settings: Settings):
"""Test that executor returns success message."""
config = {
"message": "Test message",
"delay_seconds": 0
}
result = await example_executor.execute(config, test_settings)
assert "Test message" in result
assert "took" in result.lower()
@pytest.mark.asyncio
async def test_executor_with_delay(self, test_settings: Settings):
"""Test that executor respects delay_seconds."""
import time
config = {
"message": "Delayed test",
"delay_seconds": 1
}
start = time.time()
result = await example_executor.execute(config, test_settings)
elapsed = time.time() - start
# Should take at least 1 second
assert elapsed >= 1.0
assert "Delayed test" in result
@pytest.mark.asyncio
async def test_executor_with_zero_delay(self, test_settings: Settings):
"""Test that executor works with zero delay."""
config = {
"message": "Instant",
"delay_seconds": 0
}
result = await example_executor.execute(config, test_settings)
assert "Instant" in result
assert "took 0s" in result
@pytest.mark.asyncio
async def test_executor_missing_config_fields(self, test_settings: Settings):
"""Test that executor handles missing config fields gracefully."""
config = {}
# Should use defaults
result = await example_executor.execute(config, test_settings)
assert isinstance(result, str)
assert "took" in result.lower()
@pytest.mark.asyncio
async def test_executor_invalid_delay_type(self, test_settings: Settings):
"""Test that executor handles invalid delay type."""
config = {
"message": "Test",
"delay_seconds": "invalid" # Should be int
}
# Should raise TypeError or handle gracefully
with pytest.raises((TypeError, ValueError)):
await example_executor.execute(config, test_settings)
services/scheduler/tests/test_integration.py
-189
View File
@@ -1,189 +0,0 @@
"""
Integration tests with database.
These tests can be run with a test database or skipped if not available.
"""
import pytest
import os
from fastapi.testclient import TestClient
from unittest.mock import patch
@pytest.mark.integration
@pytest.mark.skipif(
os.getenv("RUN_INTEGRATION_TESTS") != "true",
reason="Integration tests require RUN_INTEGRATION_TESTS=true"
)
class TestDatabaseIntegration:
"""Integration tests that use actual database."""
def test_health_with_real_scheduler(self, client: TestClient):
"""Test health endpoint with real scheduler instance."""
# This test runs against the actual scheduler if it's running
# For true integration testing, we'd set up a test database
pass
@pytest.mark.integration
class TestEndToEndTaskFlow:
"""End-to-end tests for task lifecycle (mocked database)."""
def test_create_list_delete_task_flow(self, client: TestClient, auth_headers: dict, sample_task_data: dict):
"""Test complete task lifecycle: create → list → delete."""
from unittest.mock import MagicMock
# This simulates the full flow with mocked database
with patch('src.main.get_task_executor') as mock_executor:
mock_conn = MagicMock()
mock_cursor = MagicMock()
# Mock create
created_task = {**sample_task_data, "id": 99}
mock_cursor.fetchone.side_effect = [
created_task, # Create task
created_task, # List tasks (as dict)
("test_task",) # Delete task
]
mock_cursor.fetchall.return_value = [created_task]
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_executor.return_value.get_db_connection.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_executor.return_value.get_db_connection.return_value.__exit__ = MagicMock(return_value=None)
# Create
create_response = client.post(
"/tasks",
headers=auth_headers,
json=sample_task_data
)
# List
list_response = client.get("/tasks", headers=auth_headers)
# Delete
delete_response = client.delete(
"/tasks/test_task",
headers=auth_headers
)
# Verify the flow worked
assert create_response.status_code in [200, 500] # May fail on DB issues
assert list_response.status_code in [200, 500]
assert delete_response.status_code in [200, 404, 500]
@pytest.mark.integration
class TestTaskExecutorIntegration:
"""Integration tests for task execution."""
@pytest.mark.asyncio
async def test_execute_example_task_real(self, test_settings):
"""Test executing example executor with real implementation."""
from src.executors import example_executor
config = {
"message": "Integration test",
"delay_seconds": 0
}
result = await example_executor.execute(config, test_settings)
assert "Integration test" in result
assert "took" in result.lower()
@pytest.mark.asyncio
async def test_task_scheduling_logic(self, test_settings):
"""Test task scheduling logic."""
from src.tasks.executor import TaskExecutor
from datetime import datetime
executor = TaskExecutor(test_settings)
# Test various scheduling scenarios
task_every_minute = {
'minute': -1, 'hour': -1, 'day_of_month': -1,
'month': -1, 'day_of_week': -1
}
task_specific_time = {
'minute': 30, 'hour': 14, 'day_of_month': -1,
'month': -1, 'day_of_week': -1
}
now = datetime(2025, 12, 7, 14, 30, 0)
# Every minute task should always run
assert executor._should_run_now(task_every_minute, now) is True
# Specific time task should run at 14:30
assert executor._should_run_now(task_specific_time, now) is True
# But not at 14:31
now_plus_one = datetime(2025, 12, 7, 14, 31, 0)
assert executor._should_run_now(task_specific_time, now_plus_one) is False
@pytest.mark.integration
class TestErrorHandling:
"""Integration tests for error handling."""
@pytest.mark.asyncio
async def test_executor_with_invalid_module(self, test_settings, sample_task_data: dict):
"""Test task execution with invalid executor module."""
from src.tasks.executor import TaskExecutor
executor = TaskExecutor(test_settings)
task = {**sample_task_data, "executor": "nonexistent_executor"}
with patch.object(executor, 'get_db_connection') as mock_get_conn:
from unittest.mock import MagicMock
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_get_conn.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_get_conn.return_value.__exit__ = MagicMock(return_value=None)
# Should handle the error gracefully
await executor.execute_task(task)
# Should have recorded the error
assert mock_cursor.execute.called
@pytest.mark.integration
class TestConfigValidation:
"""Integration tests for configuration validation."""
def test_settings_loads_from_environment(self):
"""Test that settings load correctly from environment."""
from src.config import get_settings
settings = get_settings()
# Should have loaded test environment variables
assert settings.postgres_host == "test-postgres"
assert settings.postgres_db == "test_scheduler"
assert settings.scheduler_api_key == "test-api-key-12345"
def test_settings_provides_database_url(self):
"""Test that settings provides correct database URL."""
from src.config import get_settings
settings = get_settings()
db_url = settings.database_url
assert "postgresql://" in db_url
assert "test_scheduler" in db_url
def test_settings_provides_redis_url(self):
"""Test that settings provides correct Redis URL."""
from src.config import get_settings
settings = get_settings()
redis_url = settings.redis_url
assert "redis://" in redis_url
services/scheduler/tests/test_rest_api_executor.py
-631
View File
@@ -1,631 +0,0 @@
"""
Comprehensive tests for the generic REST API executor.
Tests cover:
- All HTTP methods (GET, POST, PUT, DELETE, PATCH)
- Authentication types (Bearer, Basic, API Key)
- Environment variable substitution
- Error handling
- Response extraction
- Sensitive data redaction
Run with: pytest tests/test_rest_api_executor.py -v -s
"""
import pytest
import asyncio
import os
from unittest.mock import AsyncMock, patch, MagicMock
import httpx
from src.executors import rest_api_executor
from src.config import Settings
@pytest.mark.executor
@pytest.mark.unit
class TestRestApiExecutor:
"""Tests for rest_api_executor module."""
@pytest.fixture
def mock_httpx_client(self):
"""Mock httpx AsyncClient."""
return AsyncMock()
@pytest.fixture
def base_config(self):
"""Base configuration for REST API calls."""
return {
"url": "http://test-service:8080/api/endpoint",
"method": "POST"
}
# Basic HTTP Method Tests
@pytest.mark.asyncio
async def test_post_request_success(self, test_settings: Settings):
"""Test successful POST request."""
config = {
"url": "http://httpbin.org/post",
"method": "POST",
"payload": {"test": "data"},
"timeout": 10
}
# Mock httpx response
with patch('httpx.AsyncClient') as mock_client:
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {"message": "Success"}
mock_response.text = "Success"
mock_client.return_value.__aenter__.return_value.post = AsyncMock(
return_value=mock_response
)
result = await rest_api_executor.execute(config, test_settings)
assert "Success" in result
@pytest.mark.asyncio
async def test_get_request_success(self, test_settings: Settings):
"""Test successful GET request."""
config = {
"url": "http://httpbin.org/get",
"method": "GET",
"timeout": 10
}
with patch('httpx.AsyncClient') as mock_client:
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {"result": "GET success"}
mock_response.text = "Success"
mock_client.return_value.__aenter__.return_value.get = AsyncMock(
return_value=mock_response
)
result = await rest_api_executor.execute(config, test_settings)
assert result is not None
@pytest.mark.asyncio
async def test_put_request_success(self, test_settings: Settings):
"""Test successful PUT request."""
config = {
"url": "http://httpbin.org/put",
"method": "PUT",
"payload": {"update": "data"}
}
with patch('httpx.AsyncClient') as mock_client:
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {"message": "Updated"}
mock_response.text = "Updated"
mock_client.return_value.__aenter__.return_value.put = AsyncMock(
return_value=mock_response
)
result = await rest_api_executor.execute(config, test_settings)
assert "Updated" in result
@pytest.mark.asyncio
async def test_delete_request_success(self, test_settings: Settings):
"""Test successful DELETE request."""
config = {
"url": "http://httpbin.org/delete",
"method": "DELETE"
}
with patch('httpx.AsyncClient') as mock_client:
mock_response = MagicMock()
mock_response.status_code = 204
mock_response.json.side_effect = Exception("No JSON")
mock_response.text = ""
mock_client.return_value.__aenter__.return_value.delete = AsyncMock(
return_value=mock_response
)
result = await rest_api_executor.execute(config, test_settings)
assert "204" in result
@pytest.mark.asyncio
async def test_patch_request_success(self, test_settings: Settings):
"""Test successful PATCH request."""
config = {
"url": "http://httpbin.org/patch",
"method": "PATCH",
"payload": {"field": "value"}
}
with patch('httpx.AsyncClient') as mock_client:
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {"message": "Patched"}
mock_response.text = "Patched"
mock_client.return_value.__aenter__.return_value.patch = AsyncMock(
return_value=mock_response
)
result = await rest_api_executor.execute(config, test_settings)
assert "Patched" in result
# Authentication Tests
@pytest.mark.asyncio
async def test_bearer_auth(self, test_settings: Settings):
"""Test Bearer token authentication."""
config = {
"url": "http://test-service/api",
"method": "GET",
"auth": {
"type": "bearer",
"token": "test-token-123"
}
}
with patch('httpx.AsyncClient') as mock_client:
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {"message": "Authenticated"}
mock_response.text = "Authenticated"
mock_get = AsyncMock(return_value=mock_response)
mock_client.return_value.__aenter__.return_value.get = mock_get
result = await rest_api_executor.execute(config, test_settings)
# Verify Authorization header was set
call_args = mock_get.call_args
headers = call_args.kwargs.get('headers', {})
assert 'Authorization' in headers
assert headers['Authorization'] == 'Bearer test-token-123'
@pytest.mark.asyncio
async def test_basic_auth(self, test_settings: Settings):
"""Test Basic authentication."""
config = {
"url": "http://test-service/api",
"method": "GET",
"auth": {
"type": "basic",
"username": "testuser",
"password": "testpass"
}
}
with patch('httpx.AsyncClient') as mock_client:
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {"message": "Authenticated"}
mock_response.text = "Authenticated"
mock_get = AsyncMock(return_value=mock_response)
mock_client.return_value.__aenter__.return_value.get = mock_get
result = await rest_api_executor.execute(config, test_settings)
# Verify Authorization header was set
call_args = mock_get.call_args
headers = call_args.kwargs.get('headers', {})
assert 'Authorization' in headers
assert headers['Authorization'].startswith('Basic ')
@pytest.mark.asyncio
async def test_api_key_auth(self, test_settings: Settings):
"""Test API Key authentication."""
config = {
"url": "http://test-service/api",
"method": "GET",
"auth": {
"type": "api_key",
"key": "my-api-key-123",
"header": "X-API-Key"
}
}
with patch('httpx.AsyncClient') as mock_client:
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {"message": "Authenticated"}
mock_response.text = "Authenticated"
mock_get = AsyncMock(return_value=mock_response)
mock_client.return_value.__aenter__.return_value.get = mock_get
result = await rest_api_executor.execute(config, test_settings)
# Verify API key header was set
call_args = mock_get.call_args
headers = call_args.kwargs.get('headers', {})
assert 'X-API-Key' in headers
assert headers['X-API-Key'] == 'my-api-key-123'
# Environment Variable Substitution Tests
@pytest.mark.asyncio
async def test_env_var_substitution_in_url(self, test_settings: Settings):
"""Test environment variable substitution in URL."""
os.environ['TEST_SERVICE_URL'] = 'http://my-service:8080'
config = {
"url": "${TEST_SERVICE_URL}/api/endpoint",
"method": "GET"
}
with patch('httpx.AsyncClient') as mock_client:
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {"message": "Success"}
mock_response.text = "Success"
mock_get = AsyncMock(return_value=mock_response)
mock_client.return_value.__aenter__.return_value.get = mock_get
result = await rest_api_executor.execute(config, test_settings)
# Verify URL was substituted
call_args = mock_get.call_args
url = call_args.args[0] if call_args.args else None
assert url == 'http://my-service:8080/api/endpoint'
del os.environ['TEST_SERVICE_URL']
@pytest.mark.asyncio
async def test_env_var_substitution_in_auth_token(self, test_settings: Settings):
"""Test environment variable substitution in auth token."""
os.environ['API_TOKEN'] = 'secret-token-from-env'
config = {
"url": "http://test-service/api",
"method": "GET",
"auth": {
"type": "bearer",
"token": "${API_TOKEN}"
}
}
with patch('httpx.AsyncClient') as mock_client:
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {}
mock_response.text = "Success"
mock_get = AsyncMock(return_value=mock_response)
mock_client.return_value.__aenter__.return_value.get = mock_get
result = await rest_api_executor.execute(config, test_settings)
# Verify token was substituted
call_args = mock_get.call_args
headers = call_args.kwargs.get('headers', {})
assert headers['Authorization'] == 'Bearer secret-token-from-env'
del os.environ['API_TOKEN']
@pytest.mark.asyncio
async def test_env_var_substitution_in_payload(self, test_settings: Settings):
"""Test environment variable substitution in payload."""
os.environ['DATABASE_NAME'] = 'test_db'
config = {
"url": "http://test-service/api",
"method": "POST",
"payload": {
"database": "${DATABASE_NAME}",
"action": "backup"
}
}
with patch('httpx.AsyncClient') as mock_client:
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {}
mock_response.text = "Success"
mock_post = AsyncMock(return_value=mock_response)
mock_client.return_value.__aenter__.return_value.post = mock_post
result = await rest_api_executor.execute(config, test_settings)
# Verify payload was substituted
call_args = mock_post.call_args
payload = call_args.kwargs.get('json', {})
assert payload['database'] == 'test_db'
del os.environ['DATABASE_NAME']
# Response Extraction Tests
@pytest.mark.asyncio
async def test_response_path_extraction(self, test_settings: Settings):
"""Test extracting specific field from response."""
config = {
"url": "http://test-service/api",
"method": "GET",
"response_path": "result.message"
}
with patch('httpx.AsyncClient') as mock_client:
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"result": {
"message": "Extracted message",
"other": "ignored"
}
}
mock_response.text = "Success"
mock_client.return_value.__aenter__.return_value.get = AsyncMock(
return_value=mock_response
)
result = await rest_api_executor.execute(config, test_settings)
assert result == "Extracted message"
@pytest.mark.asyncio
async def test_response_path_not_found(self, test_settings: Settings):
"""Test response path extraction when field doesn't exist."""
config = {
"url": "http://test-service/api",
"method": "GET",
"response_path": "result.missing"
}
with patch('httpx.AsyncClient') as mock_client:
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {"result": {"other": "data"}}
mock_response.text = "Success"
mock_client.return_value.__aenter__.return_value.get = AsyncMock(
return_value=mock_response
)
result = await rest_api_executor.execute(config, test_settings)
# Should fallback to default message
assert result is not None
# Error Handling Tests
@pytest.mark.asyncio
async def test_missing_url_raises_error(self, test_settings: Settings):
"""Test that missing URL raises ValueError."""
config = {
"method": "GET"
}
with pytest.raises(ValueError, match="Missing required config: 'url'"):
await rest_api_executor.execute(config, test_settings)
@pytest.mark.asyncio
async def test_invalid_http_method_raises_error(self, test_settings: Settings):
"""Test that invalid HTTP method raises ValueError."""
config = {
"url": "http://test-service/api",
"method": "INVALID"
}
with pytest.raises(ValueError, match="Invalid HTTP method"):
await rest_api_executor.execute(config, test_settings)
@pytest.mark.asyncio
async def test_http_error_status_code(self, test_settings: Settings):
"""Test handling of HTTP error status codes."""
config = {
"url": "http://test-service/api",
"method": "GET"
}
with patch('httpx.AsyncClient') as mock_client:
mock_response = MagicMock()
mock_response.status_code = 500
mock_response.text = "Internal Server Error"
mock_client.return_value.__aenter__.return_value.get = AsyncMock(
return_value=mock_response
)
with pytest.raises(Exception, match="API call failed with status 500"):
await rest_api_executor.execute(config, test_settings)
@pytest.mark.asyncio
async def test_custom_success_codes(self, test_settings: Settings):
"""Test custom success codes configuration."""
config = {
"url": "http://test-service/api",
"method": "POST",
"success_codes": [201, 202]
}
with patch('httpx.AsyncClient') as mock_client:
mock_response = MagicMock()
mock_response.status_code = 201
mock_response.json.return_value = {"message": "Created"}
mock_response.text = "Created"
mock_client.return_value.__aenter__.return_value.post = AsyncMock(
return_value=mock_response
)
result = await rest_api_executor.execute(config, test_settings)
assert "Created" in result
@pytest.mark.asyncio
async def test_network_error_handling(self, test_settings: Settings):
"""Test handling of network errors."""
config = {
"url": "http://unreachable-service/api",
"method": "GET",
"timeout": 1
}
with patch('httpx.AsyncClient') as mock_client:
mock_client.return_value.__aenter__.return_value.get = AsyncMock(
side_effect=httpx.ConnectError("Connection refused")
)
with pytest.raises(Exception, match="Request failed"):
await rest_api_executor.execute(config, test_settings)
# Sensitive Data Redaction Tests
def test_redact_sensitive_data_in_dict(self):
"""Test redaction of sensitive fields in dictionaries."""
data = {
"username": "user",
"password": "secret123",
"api_key": "key123",
"normal_field": "visible"
}
redacted = rest_api_executor._redact_sensitive(data)
assert redacted["username"] == "user"
assert redacted["password"] == "***REDACTED***"
assert redacted["api_key"] == "***REDACTED***"
assert redacted["normal_field"] == "visible"
def test_redact_sensitive_nested(self):
"""Test redaction in nested structures."""
data = {
"config": {
"database": "mydb",
"password": "secret",
"auth": {
"token": "bearer123"
}
}
}
redacted = rest_api_executor._redact_sensitive(data)
assert redacted["config"]["database"] == "mydb"
assert redacted["config"]["password"] == "***REDACTED***"
assert redacted["config"]["auth"]["token"] == "***REDACTED***"
# Helper Function Tests
def test_extract_json_path_simple(self):
"""Test simple JSONPath extraction."""
data = {"message": "test"}
result = rest_api_executor._extract_json_path(data, "message")
assert result == "test"
def test_extract_json_path_nested(self):
"""Test nested JSONPath extraction."""
data = {"result": {"status": {"message": "success"}}}
result = rest_api_executor._extract_json_path(data, "result.status.message")
assert result == "success"
def test_extract_json_path_not_found(self):
"""Test JSONPath extraction when path doesn't exist."""
data = {"message": "test"}
result = rest_api_executor._extract_json_path(data, "missing.path")
assert result is None
def test_substitute_env_vars(self):
"""Test environment variable substitution."""
os.environ['TEST_VAR'] = 'test_value'
result = rest_api_executor._substitute_env_vars("Prefix ${TEST_VAR} suffix")
assert result == "Prefix test_value suffix"
del os.environ['TEST_VAR']
def test_substitute_env_vars_missing(self):
"""Test substitution with missing environment variable."""
result = rest_api_executor._substitute_env_vars("Prefix ${MISSING_VAR} suffix")
# Should replace with empty string
assert result == "Prefix suffix"
def test_substitute_env_vars_recursive(self):
"""Test recursive environment variable substitution."""
os.environ['HOST'] = 'localhost'
os.environ['PORT'] = '8080'
data = {
"url": "http://${HOST}:${PORT}/api",
"nested": {
"key": "${HOST}"
}
}
result = rest_api_executor._substitute_env_vars_recursive(data)
assert result["url"] == "http://localhost:8080/api"
assert result["nested"]["key"] == "localhost"
del os.environ['HOST']
del os.environ['PORT']
# Configuration Options Tests
@pytest.mark.asyncio
async def test_custom_headers(self, test_settings: Settings):
"""Test adding custom headers."""
config = {
"url": "http://test-service/api",
"method": "GET",
"headers": {
"X-Custom-Header": "custom-value",
"User-Agent": "Test-Agent"
}
}
with patch('httpx.AsyncClient') as mock_client:
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {}
mock_response.text = "Success"
mock_get = AsyncMock(return_value=mock_response)
mock_client.return_value.__aenter__.return_value.get = mock_get
result = await rest_api_executor.execute(config, test_settings)
# Verify custom headers were set
call_args = mock_get.call_args
headers = call_args.kwargs.get('headers', {})
assert headers['X-Custom-Header'] == 'custom-value'
assert headers['User-Agent'] == 'Test-Agent'
@pytest.mark.asyncio
async def test_ssl_verification_disabled(self, test_settings: Settings):
"""Test disabling SSL verification."""
config = {
"url": "https://test-service/api",
"method": "GET",
"verify_ssl": False
}
with patch('httpx.AsyncClient') as mock_client:
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {}
mock_response.text = "Success"
mock_client.return_value.__aenter__.return_value.get = AsyncMock(
return_value=mock_response
)
result = await rest_api_executor.execute(config, test_settings)
# Verify SSL verification was disabled
client_call = mock_client.call_args
assert client_call.kwargs.get('verify') is False
if __name__ == "__main__":
pytest.main([__file__, "-v", "-s"])
services/scheduler/tests/test_task_executor.py
-307
View File
@@ -1,307 +0,0 @@
"""
Tests for the task executor module.
"""
import pytest
from unittest.mock import AsyncMock, MagicMock, patch
from datetime import datetime
from src.tasks.executor import TaskExecutor
from src.config import Settings
@pytest.mark.unit
class TestTaskExecutor:
"""Tests for TaskExecutor class."""
def test_executor_initialization(self, test_settings: Settings):
"""Test that executor initializes correctly."""
executor = TaskExecutor(test_settings)
assert executor.settings == test_settings
assert executor.max_concurrent == 5
@patch('psycopg2.connect')
def test_get_db_connection(self, mock_connect, test_settings: Settings):
"""Test database connection creation."""
mock_conn = MagicMock()
mock_connect.return_value = mock_conn
executor = TaskExecutor(test_settings)
conn = executor.get_db_connection()
assert conn == mock_conn
mock_connect.assert_called_once()
@pytest.mark.asyncio
async def test_process_minute_no_tasks(self, test_settings: Settings):
"""Test process_minute with no scheduled tasks."""
executor = TaskExecutor(test_settings)
with patch.object(executor, 'get_db_connection') as mock_get_conn:
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_cursor.fetchall.return_value = []
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_get_conn.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_get_conn.return_value.__exit__ = MagicMock(return_value=None)
# Should complete without error
await executor.process_minute()
# Should have queried for tasks
assert mock_cursor.execute.called
@pytest.mark.asyncio
async def test_process_minute_with_tasks(self, test_settings: Settings, sample_task_data: dict):
"""Test process_minute executes tasks."""
executor = TaskExecutor(test_settings)
task_from_db = {
**sample_task_data,
'id': 1,
'config': sample_task_data['config'] # Already a dict
}
with patch.object(executor, 'get_db_connection') as mock_get_conn, \
patch.object(executor, 'execute_task', new_callable=AsyncMock) as mock_execute:
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_cursor.fetchall.return_value = [task_from_db]
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_get_conn.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_get_conn.return_value.__exit__ = MagicMock(return_value=None)
await executor.process_minute()
# Should have executed the task
mock_execute.assert_called_once()
@pytest.mark.asyncio
async def test_execute_task_success(self, test_settings: Settings, sample_task_data: dict):
"""Test successful task execution."""
executor = TaskExecutor(test_settings)
task = {**sample_task_data, 'id': 1}
with patch.object(executor, 'get_db_connection') as mock_get_conn, \
patch('src.tasks.executor.importlib.import_module') as mock_import:
# Mock executor module
mock_executor_module = MagicMock()
mock_executor_module.execute = AsyncMock(return_value="Task completed successfully")
mock_import.return_value = mock_executor_module
# Mock database
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_get_conn.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_get_conn.return_value.__exit__ = MagicMock(return_value=None)
await executor.execute_task(task)
# Should have imported executor module
mock_import.assert_called_with('src.executors.example_executor')
# Should have updated task status
assert mock_cursor.execute.call_count >= 2 # Insert execution record + update task
@pytest.mark.asyncio
async def test_execute_task_failure(self, test_settings: Settings, sample_task_data: dict):
"""Test task execution handles failures."""
executor = TaskExecutor(test_settings)
task = {**sample_task_data, 'id': 1}
with patch.object(executor, 'get_db_connection') as mock_get_conn, \
patch('src.tasks.executor.importlib.import_module') as mock_import:
# Mock executor that raises error
mock_executor_module = MagicMock()
mock_executor_module.execute = AsyncMock(side_effect=Exception("Task failed"))
mock_import.return_value = mock_executor_module
# Mock database
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_get_conn.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_get_conn.return_value.__exit__ = MagicMock(return_value=None)
await executor.execute_task(task)
# Should have recorded the failure
assert mock_cursor.execute.called
# Check that error status was recorded
calls = [str(call) for call in mock_cursor.execute.call_args_list]
assert any('failed' in str(call).lower() or 'error' in str(call).lower() for call in calls)
@pytest.mark.asyncio
async def test_execute_task_timeout(self, test_settings: Settings, sample_task_data: dict):
"""Test task execution handles timeouts."""
executor = TaskExecutor(test_settings)
task = {**sample_task_data, 'id': 1, 'timeout_seconds': 1}
with patch.object(executor, 'get_db_connection') as mock_get_conn, \
patch('src.tasks.executor.importlib.import_module') as mock_import:
# Mock executor that takes too long
import asyncio
mock_executor_module = MagicMock()
async def slow_execute(*args, **kwargs):
await asyncio.sleep(10) # Longer than timeout
return "Done"
mock_executor_module.execute = slow_execute
mock_import.return_value = mock_executor_module
# Mock database
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_get_conn.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_get_conn.return_value.__exit__ = MagicMock(return_value=None)
await executor.execute_task(task)
# Should have recorded timeout
calls = [str(call) for call in mock_cursor.execute.call_args_list]
assert any('timeout' in str(call).lower() for call in calls)
def test_should_run_task_wildcard(self, test_settings: Settings):
"""Test task scheduling with wildcards."""
executor = TaskExecutor(test_settings)
# All wildcards should always match
task = {
'minute': -1,
'hour': -1,
'day_of_month': -1,
'month': -1,
'day_of_week': -1
}
now = datetime(2025, 12, 7, 14, 30, 0) # Saturday
assert executor._should_run_now(task, now) is True
def test_should_run_task_specific_time(self, test_settings: Settings):
"""Test task scheduling with specific time."""
executor = TaskExecutor(test_settings)
# Specific time: every day at 14:30
task = {
'minute': 30,
'hour': 14,
'day_of_month': -1,
'month': -1,
'day_of_week': -1
}
# Matching time
now = datetime(2025, 12, 7, 14, 30, 0)
assert executor._should_run_now(task, now) is True
# Non-matching time
now = datetime(2025, 12, 7, 14, 31, 0)
assert executor._should_run_now(task, now) is False
def test_should_run_task_specific_day_of_month(self, test_settings: Settings):
"""Test task scheduling with specific day of month."""
executor = TaskExecutor(test_settings)
# Run on 11th of every month at 04:00
task = {
'minute': 0,
'hour': 4,
'day_of_month': 11,
'month': -1,
'day_of_week': -1
}
# Matching date
now = datetime(2025, 12, 11, 4, 0, 0)
assert executor._should_run_now(task, now) is True
# Wrong day
now = datetime(2025, 12, 12, 4, 0, 0)
assert executor._should_run_now(task, now) is False
def test_should_run_task_specific_month(self, test_settings: Settings):
"""Test task scheduling with specific month."""
executor = TaskExecutor(test_settings)
# Run on January 1st at midnight
task = {
'minute': 0,
'hour': 0,
'day_of_month': 1,
'month': 1,
'day_of_week': -1
}
# Matching date
now = datetime(2025, 1, 1, 0, 0, 0)
assert executor._should_run_now(task, now) is True
# Wrong month
now = datetime(2025, 2, 1, 0, 0, 0)
assert executor._should_run_now(task, now) is False
def test_should_run_task_day_of_week(self, test_settings: Settings):
"""Test task scheduling with day of week."""
executor = TaskExecutor(test_settings)
# Run every Monday at 09:00
task = {
'minute': 0,
'hour': 9,
'day_of_month': -1,
'month': -1,
'day_of_week': 0 # Monday
}
# Monday
now = datetime(2025, 12, 8, 9, 0, 0) # Monday
assert executor._should_run_now(task, now) is True
# Tuesday
now = datetime(2025, 12, 9, 9, 0, 0) # Tuesday
assert executor._should_run_now(task, now) is False
@pytest.mark.asyncio
async def test_concurrent_task_limit(self, test_settings: Settings, sample_task_data: dict):
"""Test that executor respects concurrent task limit."""
executor = TaskExecutor(test_settings)
# Create 10 tasks
tasks = [
{**sample_task_data, 'id': i, 'task_name': f'task_{i}'}
for i in range(10)
]
with patch.object(executor, 'get_db_connection') as mock_get_conn, \
patch.object(executor, 'execute_task', new_callable=AsyncMock) as mock_execute:
mock_conn = MagicMock()
mock_cursor = MagicMock()
mock_cursor.fetchall.return_value = tasks
mock_conn.cursor.return_value.__enter__ = MagicMock(return_value=mock_cursor)
mock_conn.cursor.return_value.__exit__ = MagicMock(return_value=None)
mock_get_conn.return_value.__enter__ = MagicMock(return_value=mock_conn)
mock_get_conn.return_value.__exit__ = MagicMock(return_value=None)
await executor.process_minute()
# Should only execute max_concurrent (5) tasks
assert mock_execute.call_count <= executor.max_concurrent
stacks/gitea.yml
+18
View File
@@ -48,6 +48,24 @@ services:
networks:
- docker-dataplane
gitea-runner:
image: gitea/act_runner:latest
container_name: gitea-runner
restart: unless-stopped
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /home/jpmschweitzer/docker-data/gitea/runner:/data
environment:
- CONFIG_FILE=/data/config.yaml
- GITEA_INSTANCE_URL=http://gitea:3000
- GITEA_RUNNER_REGISTRATION_TOKEN=${GITEA_RUNNER_TOKEN}
- GITEA_RUNNER_NAME=docker-runner
- TZ=Europe/Amsterdam
depends_on:
- gitea
networks:
- docker-dataplane
networks:
docker-dataplane:
external: true
stacks/scheduler.yml
+3 -25
View File
@@ -4,35 +4,14 @@ version: '3.8'
# Purpose: System-wide maintenance orchestration - backups, doc mirroring, cleanup, task automation
# Port: 8090 (API + UI)
# Network: docker-dataplane
# Image: git.schweitz.net/jpmschweitzer/scheduler (private registry)
services:
scheduler:
image: python:3.12-slim
image: git.schweitz.net/jpmschweitzer/scheduler:latest
container_name: scheduler
restart: unless-stopped
command: >
sh -c "
echo 'Installing system dependencies...' &&
apt-get update -qq &&
apt-get install -y --no-install-recommends git ssh postgresql-client curl docker.io >/dev/null 2>&1 &&
rm -rf /var/lib/apt/lists/* &&
echo 'Setting up Python environment...' &&
if [ ! -f /venv/bin/python ]; then
echo 'Initializing venv...' &&
python3 -m venv --clear /venv;
fi &&
echo 'Upgrading pip...' &&
/venv/bin/python -m pip install --upgrade pip --quiet &&
echo 'Installing dependencies...' &&
/venv/bin/python -m pip install -r /app/requirements.txt --quiet &&
echo 'Configuring Git...' &&
git config --global user.name 'The Librarian' &&
git config --global user.email 'librarian@portainer-core.local' &&
echo 'Starting The Scheduler...' &&
/venv/bin/python -m uvicorn src.main:app --host 0.0.0.0 --port 8090 --workers 1
"
ports:
- "8090:8090"
@@ -65,8 +44,7 @@ services:
- PYTHONPATH=/app
volumes:
- /home/jpmschweitzer/Projects/portainer-core/services/scheduler:/app
- /home/jpmschweitzer/docker-data/scheduler/venv:/venv
# Data volumes only - source code is baked into image
- /home/jpmschweitzer/docker-data/scheduler/logs:/app/logs
- /home/jpmschweitzer/docker-data/scheduler/task-data:/app/task-data
- /home/jpmschweitzer/docker-data/scheduler/ssh:/root/.ssh:ro