fix(executors): run backups off the event loop
The scheduler serves its own REST API from the same loop that runs executors, and the config backup spends ~21 minutes inside tarfile and zlib. Called inline that starves the loop for the whole window: the service was unreachable 03:05-03:25 every night, and again at 07:39 today when the job was triggered by hand to prove the T-69 report path. asyncio.to_thread is the fix -- zlib releases the GIL while compressing, so the loop is scheduled normally. The outage was invisible for as long as it existed. The hourly health check fires at :35 and the outage runs 03:05-03:25, so no sample ever landed inside it. A fixed-phase hourly probe cannot see a 20-minute event; that is aliasing, not bad luck, and it would have stayed hidden indefinitely. health_report.report_async joins it: psycopg2 is a blocking driver, so reporting from the loop held it for the connect and insert -- up to connect_timeout seconds precisely when the database is unreachable, which is when a report matters most. Both backup executors use it now. Caveat worth knowing: the engine wraps executors in asyncio.wait_for and a thread cannot be cancelled, so on timeout the task is recorded failed while the tar runs to completion. Still strictly better than blocking everything, and the configured 3600s is well clear of the observed 1263s. Seven tests in this file had been red since the initial commit -- they came over with the portainer-core extraction, patched the Path class wholesale, asserted "backed up" against a function returning "Backup completed: ...", and one wrapped its call in except Exception: pass with its only assertion commented out. There is no CI test gate here, so nothing reported it. Replaced with tests that build real archives in tmp_path and assert on their contents. The new loop test is the one that matters and it is mutation-checked: with to_thread reverted it counts 0 heartbeat ticks, with it ~40. Their structural demands (_create_tar_filter, a sync _cleanup_old_backups) were adopted because threading wanted that shape anyway. Their exclude semantics were not. The tests assert fnmatch behaviour and the deployed config is written against substring matching -- it excludes logs as ".log", and paths as unanchored fragments like "ollama/models/*" against members named "docker-data/ollama/...". Under fnmatch neither matches, and the nightly archive would silently gain many GB of model blobs instead of shrinking. That is now pinned by tests naming the consequence, so the "improvement" fails loudly. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -153,7 +153,7 @@ async def execute(config: dict, settings: Settings) -> str:
|
||||
try:
|
||||
output = await _run(config, settings)
|
||||
except Exception as exc:
|
||||
health_report.report(
|
||||
await health_report.report_async(
|
||||
settings,
|
||||
domain="backup",
|
||||
status=health_report.CRITICAL,
|
||||
@@ -161,7 +161,7 @@ async def execute(config: dict, settings: Settings) -> str:
|
||||
metrics={"job": "scheduler/portainer_backup_executor", "error": str(exc)[:400]},
|
||||
)
|
||||
raise
|
||||
health_report.report(
|
||||
await health_report.report_async(
|
||||
settings,
|
||||
domain="backup",
|
||||
status=health_report.OK,
|
||||
|
||||
Reference in New Issue
Block a user