feat(config): T-1280 — job log retention, and the corpse that would never die
Pruning happens at spawn time rather than on a schedule: a retention pass that depends on someone remembering to run it is one that silently never happens. reach jobs prune is the explicit escape hatch for reclaiming space now. The cap was measured rather than guessed, which is why this ticket ran last. A chatty short job writes ~1.8 KB across its three files, so 100 jobs is single-digit megabytes even if a generator emits per-body progress — inside .cache/, where being wrong costs disk and never data. SR_JOB_KEEP overrides it. The interesting part is what "a running job is never pruned" has to mean. Not "the file says running" — a process killed outright never updates its own status, so that reading would make every crashed job immortal. Those are exactly the ones that accumulate, so the naive rule produces the opposite of retention: the only logs that never go away are the ones nobody wants. The check consults the process table instead. Verified both directions. Live, a running 30-second job survived a prune to --keep 1. Pinned with a fixture holding a finished job, a corpse (record says running, pid gone), and a genuinely live one — asserting the live one survives and the corpse does not. Proven to fail by dropping the liveness check. One false alarm worth recording: my first live test looked exactly like the bug, showing a running job pruned. It was not — my commands ran two minutes apart, so the "20-second" job had finished long before. The test was invalid, not the guard. A timing-sensitive check across separate shell turns proves nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -111,9 +111,54 @@ def spawn_detached(argv: list[str]) -> str:
|
||||
"status": "running",
|
||||
},
|
||||
)
|
||||
|
||||
# At write time, not on a schedule. A retention pass that depends on someone
|
||||
# remembering to run it is one that silently never happens.
|
||||
prune()
|
||||
return job_id
|
||||
|
||||
|
||||
# Measured 2026-08-31: a chatty short job (10 progress events) writes ~1.8 KB
|
||||
# across its three files. A generator emitting per-body progress might reach
|
||||
# 100 KB. 100 jobs is therefore single-digit megabytes at worst, inside .cache/
|
||||
# which is gitignored scratch — so the failure mode of this number being wrong
|
||||
# is disk, never data. Raise it freely if a real batch proves it tight.
|
||||
DEFAULT_KEEP = 100
|
||||
|
||||
ENV_KEEP = "SR_JOB_KEEP"
|
||||
|
||||
|
||||
def prune(keep: int | None = None) -> list[str]:
|
||||
"""Delete all but the `keep` most recent jobs. Returns the ids removed.
|
||||
|
||||
**A genuinely running job is never pruned** — but note what that has to
|
||||
mean. Skipping anything whose *file* says "running" would make every
|
||||
crashed job immortal, because a process killed outright never gets to
|
||||
update its own status. Those are precisely the jobs that accumulate, so the
|
||||
check is against the process table, not the record.
|
||||
|
||||
Called at spawn time rather than by a sweep: a cleanup nothing invokes is a
|
||||
cleanup that does not happen.
|
||||
"""
|
||||
if keep is None:
|
||||
raw = os.environ.get(ENV_KEEP)
|
||||
keep = int(raw) if raw and raw.isdigit() else DEFAULT_KEEP
|
||||
|
||||
directory = jobs_dir()
|
||||
# Ids sort chronologically because they are timestamp-first (T-1276).
|
||||
ids = sorted((path.stem for path in directory.glob("*.json")), reverse=True)
|
||||
|
||||
removed: list[str] = []
|
||||
for job_id in ids[keep:]:
|
||||
meta = read_meta(job_id)
|
||||
if meta and meta.get("status") == "running" and is_alive(int(meta.get("pid", -1))):
|
||||
continue # actually running — a long generator may outlive the window
|
||||
for path in (log_path(job_id), output_path(job_id), meta_path(job_id)):
|
||||
path.unlink(missing_ok=True)
|
||||
removed.append(job_id)
|
||||
return removed
|
||||
|
||||
|
||||
def finish_if_detached(exit_code: int) -> None:
|
||||
"""Record completion — called by the CHILD, from the outermost decorator.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user