feat: backup executors report their own outcome to check_history

The health record used to learn about backups by polling the mtime of the
newest archive, hourly, against a 48-hour threshold — for a job that runs once
a day. Forty-seven of every forty-eight runs could not produce a new answer.

Worse, file age cannot distinguish a failed backup from one that has not run
yet: when tonight's job dies, yesterday's archive is 24 hours old and still
reads healthy, and keeps reading healthy until hour 48. A failure stayed
invisible for two days to the check whose only job was noticing it. These
executors know at 03:05.

Reporting wraps the work rather than living inside it, so the failure path
cannot be forgotten — an exception is reported as critical and re-raised,
leaving the task's own status untouched. Reporting only success would reproduce
exactly the blind spot this replaces.

A reporting failure never fails the backup. Everything in health_report is
caught and logged, which means the absence of rows is the only symptom a broken
reporter produces — so monitor for rows, not for errors.

Not yet functional in production: scheduler_user holds DELETE and SELECT on
check_history (it prunes the table nightly) but not INSERT. Until that grant is
made the report logs a refusal and skips. The ticket predicted this as the step
that would fail silently, which is why it is named in the changelog and handled
as its own exception rather than folded into a generic catch.

Verified against baseline: tests/test_config_backup_executor.py and
tests/test_portainer_backup_executor.py report 7 failed / 17 passed both with
and without this change, so the pre-existing failures are untouched.

Workspace D-33, T-69.
This commit is contained in:
2026-08-11 00:11:52 +02:00
parent 0c2199667f
commit d9cbaee1fc
4 changed files with 209 additions and 2 deletions
+32 -1
View File
@@ -12,11 +12,12 @@ from pathlib import Path
from typing import List, Dict, Any
from src.config import Settings
from src.executors import health_report
logger = logging.getLogger(__name__)
async def execute(config: dict, settings: Settings) -> str:
async def _run(config: dict, settings: Settings) -> str:
"""
Execute config backup task.
@@ -157,3 +158,33 @@ async def cleanup_old_backups(backup_dir: Path, retention_days: int):
logger.info(f"Removed {removed_count} old backups, freed {removed_size_mb:.2f}MB")
else:
logger.info("No old backups to remove")
async def execute(config: dict, settings: Settings) -> str:
"""Run the backup and report its own outcome to check_history (D-33, T-69).
The report wraps the work rather than living inside it, so the failure path
cannot be forgotten: an exception is reported as critical and then re-raised,
leaving the task's own status untouched. Reporting only success would
reproduce exactly the blind spot this replaces — a monitor that cannot tell
a failed backup from one that has not run.
"""
try:
output = await _run(config, settings)
except Exception as exc:
health_report.report(
settings,
domain="backup",
status=health_report.CRITICAL,
source="scheduler/config_backup_executor",
metrics={"job": "scheduler/config_backup_executor", "error": str(exc)[:400]},
)
raise
health_report.report(
settings,
domain="backup",
status=health_report.OK,
source="scheduler/config_backup_executor",
metrics={"job": "scheduler/config_backup_executor", "summary": output[:400]},
)
return output