feat: backup executors report their own outcome to check_history
The health record used to learn about backups by polling the mtime of the newest archive, hourly, against a 48-hour threshold — for a job that runs once a day. Forty-seven of every forty-eight runs could not produce a new answer. Worse, file age cannot distinguish a failed backup from one that has not run yet: when tonight's job dies, yesterday's archive is 24 hours old and still reads healthy, and keeps reading healthy until hour 48. A failure stayed invisible for two days to the check whose only job was noticing it. These executors know at 03:05. Reporting wraps the work rather than living inside it, so the failure path cannot be forgotten — an exception is reported as critical and re-raised, leaving the task's own status untouched. Reporting only success would reproduce exactly the blind spot this replaces. A reporting failure never fails the backup. Everything in health_report is caught and logged, which means the absence of rows is the only symptom a broken reporter produces — so monitor for rows, not for errors. Not yet functional in production: scheduler_user holds DELETE and SELECT on check_history (it prunes the table nightly) but not INSERT. Until that grant is made the report logs a refusal and skips. The ticket predicted this as the step that would fail silently, which is why it is named in the changelog and handled as its own exception rather than folded into a generic catch. Verified against baseline: tests/test_config_backup_executor.py and tests/test_portainer_backup_executor.py report 7 failed / 17 passed both with and without this change, so the pre-existing failures are untouched. Workspace D-33, T-69.
This commit is contained in:
@@ -12,11 +12,12 @@ from pathlib import Path
|
||||
from typing import List, Dict, Any
|
||||
|
||||
from src.config import Settings
|
||||
from src.executors import health_report
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def execute(config: dict, settings: Settings) -> str:
|
||||
async def _run(config: dict, settings: Settings) -> str:
|
||||
"""
|
||||
Execute config backup task.
|
||||
|
||||
@@ -157,3 +158,33 @@ async def cleanup_old_backups(backup_dir: Path, retention_days: int):
|
||||
logger.info(f"Removed {removed_count} old backups, freed {removed_size_mb:.2f}MB")
|
||||
else:
|
||||
logger.info("No old backups to remove")
|
||||
|
||||
|
||||
async def execute(config: dict, settings: Settings) -> str:
|
||||
"""Run the backup and report its own outcome to check_history (D-33, T-69).
|
||||
|
||||
The report wraps the work rather than living inside it, so the failure path
|
||||
cannot be forgotten: an exception is reported as critical and then re-raised,
|
||||
leaving the task's own status untouched. Reporting only success would
|
||||
reproduce exactly the blind spot this replaces — a monitor that cannot tell
|
||||
a failed backup from one that has not run.
|
||||
"""
|
||||
try:
|
||||
output = await _run(config, settings)
|
||||
except Exception as exc:
|
||||
health_report.report(
|
||||
settings,
|
||||
domain="backup",
|
||||
status=health_report.CRITICAL,
|
||||
source="scheduler/config_backup_executor",
|
||||
metrics={"job": "scheduler/config_backup_executor", "error": str(exc)[:400]},
|
||||
)
|
||||
raise
|
||||
health_report.report(
|
||||
settings,
|
||||
domain="backup",
|
||||
status=health_report.OK,
|
||||
source="scheduler/config_backup_executor",
|
||||
metrics={"job": "scheduler/config_backup_executor", "summary": output[:400]},
|
||||
)
|
||||
return output
|
||||
|
||||
Reference in New Issue
Block a user