feat(config): T-1279 — a detached failure reaches its caller
The non-negotiable from D-263, pointed at its worst hiding place: a foreground command that swallows a failure at least does it in front of someone, while a background runner that reports "started" and loses the failure does it where nothing is watching. Testing the two timing cases the ticket names — fails before the parent exits, fails long after — needs a command slow enough to tell them apart, and every verb in reach finishes in milliseconds. So `reach dev selftest` exists: emits progress for N seconds, then optionally fails with a chosen code. A genuine diagnostic rather than a test hook, in the dev domain the map already planned, and the only way to answer "does streaming work here, can I tail it, does a failure survive detach" by observation instead of argument. The slow case is the one that proves the design. --detach returned in 75ms while the child ran six seconds, so the parent was demonstrably gone long before the child failed — and wait still relayed exit 7. That is the half of the recording path only this case reaches, and why T-1277 moved completion recording into the child. Also pinned: --detach exits 0 for starting and SAYS "not succeeded" in words, which the test asserts on rather than trusting the code to be read correctly; a failed job nobody waited on shows as failed in jobs list; and every event a detached job emits carries its job id. Closed T-1278's open gap in passing — jobs log --follow had never run against a genuinely long job because none existed. It now has: attached mid-flight, streamed the remaining steps live, and caught the final verdict after the job ended. Proven to fail by making effective_exit_code always return 0 — the trap itself. Both timing cases failed by name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -1961,3 +1961,25 @@ SERVICE-LEVEL TESTS — tooling/test_jobs.py, and these close the gap T-1257 nam
|
||||
read_events was split into read_events(job_id) and read_events_from(path) so a test can drive it against a temp file — better shape regardless.
|
||||
|
||||
NOT YET EXERCISED, and honest about it: `jobs log --follow` against a genuinely LONG-running job. Nothing in reach currently runs long enough to tail. The offset mechanics underneath it are tested directly, and the live loop gets its first real workout when a slow domain lands (planet or db, T-1250).', NULL, '2026-08-31 15:02:20', '2026-08-31 15:02:20.709', '2026-08-31 15:02:20.709', NULL, 'becbc04f4ea86ae9f73e0c9541992300', 2) ON CONFLICT(hash) DO NOTHING;
|
||||
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5VJ89C7Q4EPR6Q73FSS74', 'status', 'in_progress', 'done', NULL, '2026-08-31 15:02:40', '2026-08-31 15:02:40.198', '2026-08-31 15:02:40.198', NULL, '005d8c2b4c8e970ae5d4b33ab9bb15b5', 2) ON CONFLICT(hash) DO NOTHING;
|
||||
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5WNGDMENP7AY9VMB5G3XM', 'status', 'backlog', 'in_progress', NULL, '2026-08-31 15:06:54', '2026-08-31 15:06:54.261', '2026-08-31 15:06:54.261', NULL, '85a6301b9102a7cd71655fa52a02963f', 2) ON CONFLICT(hash) DO NOTHING;
|
||||
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5WNGDMENP7AY9VMB5G3XM', 'status', 'in_progress', 'in_progress', NULL, '2026-08-31 15:07:19', '2026-08-31 15:07:19.025', '2026-08-31 15:07:19.025', NULL, '8aa6e2e845e5dc18acf8c627dd14c52d', 2) ON CONFLICT(hash) DO NOTHING;
|
||||
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5WNGDMENP7AY9VMB5G3XM', 'description', 'A runner that reports started and then loses the failure is the exit-0 trap from the top of D-263 relocated somewhere nothing is watching, which is strictly worse than the original: at least a foreground exit 0 is in front of someone. Requirements: reach jobs wait <id> EXITS WITH THE JOB EXIT CODE, so a Makefile or hook can gate on a detached run exactly as it would on a foreground one; a failed job that nobody waited on is visible as failed in reach jobs list rather than merely absent; and reach --detach itself exits 0 for SUCCESSFULLY STARTING, which is a different claim from the job succeeding, so the output must say so in words rather than leaving a reader to infer it. TEST THE FAILING PATH FIRST and give it a deliberately failing command, since a job runner that has only ever run successful jobs has never been tested — this is the same discipline that caught the gate tests. Also test the case where the job fails BEFORE the parent has exited, and where it fails long AFTER, since those exercise different halves of the recording path.', 'A runner that reports started and then loses the failure is the exit-0 trap from the top of D-263 relocated somewhere nothing is watching, which is strictly worse than the original: at least a foreground exit 0 is in front of someone. Requirements: reach jobs wait <id> EXITS WITH THE JOB EXIT CODE, so a Makefile or hook can gate on a detached run exactly as it would on a foreground one; a failed job that nobody waited on is visible as failed in reach jobs list rather than merely absent; and reach --detach itself exits 0 for SUCCESSFULLY STARTING, which is a different claim from the job succeeding, so the output must say so in words rather than leaving a reader to infer it. TEST THE FAILING PATH FIRST and give it a deliberately failing command, since a job runner that has only ever run successful jobs has never been tested — this is the same discipline that caught the gate tests. Also test the case where the job fails BEFORE the parent has exited, and where it fails long AFTER, since those exercise different halves of the recording path.
|
||||
|
||||
DONE 2026-08-31. All four properties verified and pinned in tooling/test_job_exit_codes.py, wired into make test-tooling.
|
||||
|
||||
BUILT A REAL VERB TO TEST WITH, rather than a test hook. The two timing cases this ticket names — fails BEFORE the parent exits, fails long AFTER — cannot be distinguished when every command in reach finishes in milliseconds. So `reach dev selftest --seconds N --fail --exit-code C` now exists: it emits progress events over N seconds then optionally fails. It is a genuine diagnostic ("does streaming work end to end on this machine, can I tail it, does a failure survive detach?"), it lives in the `dev` domain the map already planned, and it is the only command slow enough to answer those by observation instead of argument. A test-only hook would have been the same code with less value.
|
||||
|
||||
VERIFIED, and the slow case is the one that proves the design:
|
||||
- FAST failure (0s): wait relays exit 3.
|
||||
- SLOW failure (6s real / 2s in the test): --detach returned in 75 ms while the child ran 6 seconds, so the parent was demonstrably gone long before the child failed — and wait still relayed exit 7. That is the half of the recording path only this case reaches, and it is why T-1277 moved completion recording to the child.
|
||||
- SUCCESS: wait relays 0.
|
||||
- A failed job nobody waited on shows as `failed` in jobs list with its full command.
|
||||
- --detach exits 0 AND says "this exit status means STARTED, not succeeded" in words. The test asserts on that wording, not just the code, because a bare 0 invites exactly the wrong reading.
|
||||
- Every event in a detached job''s log carries the job id.
|
||||
|
||||
CLOSED T-1278''s OPEN GAP as a side effect. `jobs log --follow` had never been exercised against a genuinely long job because none existed. With selftest it now has been: attached to a running 6-second job mid-flight, streamed the remaining steps live as they happened, and caught the final failure verdict with its remedy after the job ended. The follow loop and its final post-completion read both work.
|
||||
|
||||
PROVEN TO FAIL: made effective_exit_code always return 0 — the exit-0 trap itself — and both the fast and slow cases failed by name, reporting the expected code against the 0 they got. Restored, green.
|
||||
|
||||
NOTE ON WHAT --follow RETURNS: `jobs log --follow` exits 0 even when the job it tailed failed. That is correct and deliberate — log succeeded at logging. `wait` is the verb that relays an outcome, and conflating the two would mean you could not read a failed job''s log without your shell treating the read as a failure.', NULL, '2026-08-31 15:14:09', '2026-08-31 15:14:09.188', '2026-08-31 15:14:09.188', NULL, 'a48443a46dc1cc27d6089e81695a0b5c', 2) ON CONFLICT(hash) DO NOTHING;
|
||||
|
||||
@@ -2287,3 +2287,47 @@ SERVICE-LEVEL TESTS — tooling/test_jobs.py, and these close the gap T-1257 nam
|
||||
read_events was split into read_events(job_id) and read_events_from(path) so a test can drive it against a temp file — better shape regardless.
|
||||
|
||||
NOT YET EXERCISED, and honest about it: `jobs log --follow` against a genuinely LONG-running job. Nothing in reach currently runs long enough to tail. The offset mechanics underneath it are tested directly, and the live loop gets its first real workout when a slow domain lands (planet or db, T-1250).', 'in_progress', 'high', NULL, NULL, 'D-263', '2026-08-31 13:52:06.722', '2026-08-31 15:02:20.709', NULL, '1ce1eb66a71783ab5218d27ae143b1aa', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
|
||||
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5VJ89C7Q4EPR6Q73FSS74', 'task', '06G1S7NVJR0GT9KWS9QVYNNFMM', 'The jobs domain — list, status, log --follow, wait', 'The user-facing verbs, and a DOMAIN rather than core/ because they carry logic and state of their own — the first real test of the D-263 core bound, which it passes. Deliverables: reach jobs list (recent jobs with status, command and duration), status <id>, log <id> with --follow to tail, and wait <id>. REATTACH IS A BYTE OFFSET into an append-only file, which is the entire reason no daemon is needed: a caller can attach, drop off, and come back without losing anything, and there is no lifecycle to get wrong, nothing to orphan, and no stale state to reconcile. log --follow is therefore a poll on file length, not a subscription. Render the JSONL through the same path a live terminal uses, so a tailed log and a live run are the same artefact in two presentations rather than two renderers that drift. Note for the port: this domain is the first one written from scratch under the full contract rather than ported, so it doubles as the worked example the reach skill (T-1254) should show.
|
||||
|
||||
FROM T-1277 (2026-08-31) — the reconciliation requirement is yours and it is not optional. A child killed outright (SIGKILL, OOM, an interpreter crash) never gets to record its own completion, so its metadata file stays status=running with the process long gone. jobs list and jobs status must therefore RECONCILE against process.is_alive(pid) rather than trusting the file: a job whose pid is dead and whose status still reads running is not running, it died. Report it as such — ''died without recording an exit'' is honest and actionable, whereas showing it as running is the exit-0 trap in a place nobody is watching, and a caller polling for completion would wait forever on something that failed in milliseconds. Note the ordinary failure paths are already covered: T-1277 moved completion recording to the process''s exit (tooling/main.py main()), so bad arguments, unknown verbs, ReachErrors and unhandled exceptions all record correctly. What remains is only the case where the process cannot run code at all.
|
||||
|
||||
DONE 2026-08-31. reach jobs list / status / log --follow / wait, all four working, plus service-level tests.
|
||||
|
||||
A LATENT BUG FOUND BEFORE BUILDING ON IT, and it was already committed. typer.Exit is a RuntimeError, NOT a SystemExit — so @handle_errors caught it like any other unexpected exception. Verified: `raise typer.Exit(3)` inside a decorated command printed "unexpected Exit: 3" and exited 1, SILENTLY DISCARDING the requested code. Nothing hit it today because T-1267 had converted the check router to ReachError, but `jobs wait` needs exactly this and it is the natural thing anyone would write.
|
||||
FIX: core/errors.ReachExit(code) as the sanctioned control-flow exit, passed straight through by handle_errors as SystemExit with no verdict. ReachError would have been wrong twice over for wait — it prints a failure verdict for a command that worked, and demands a fix= for a situation with no remedy. Kept typer out of core/, which the conformance test enforces.
|
||||
|
||||
RECONCILIATION WORKS, AND WAS PROVEN ON A REAL CORPSE rather than a simulated one. The job left stranded by the T-1277 bug — status "running", process long gone — now reports as `died` with its true elapsed time. Status DIED is derived, never recorded: a process killed outright cannot write its own ending.
|
||||
|
||||
EXIT-CODE RELAY, all three cases: done -> 0, failed -> 2 (the job''s own code), died -> 137. A died job must NEVER relay 0; it has no code of its own and borrowing success is the exit-0 trap pointed at whatever gated on the run.
|
||||
|
||||
BYTE-OFFSET REATTACH verified directly: first read consumed one event to offset 169; resuming from 169 returned nothing and left the offset unmoved. That is the whole reason no daemon is needed.
|
||||
|
||||
SECOND UTC BUG, same class as T-1276''s. jobs list reported a job started minutes earlier as running for 133m — _parse used time.mktime, which reads a UTC stamp as local time and silently adds the offset to every duration. Now calendar.timegm. Both directions of this conversion have now bitten once each; both have the reason in a comment.
|
||||
|
||||
RENDERING: console.render() is now public and jobs log replays stored events through it. A second renderer for stored events would drift from the live one, and the divergence would surface exactly when someone is reading a log to work out what went wrong.
|
||||
|
||||
SERVICE-LEVEL TESTS — tooling/test_jobs.py, and these close the gap T-1257 named. D-263 claims a service is transport-agnostic and callable without a CLI round trip; nothing had ever demonstrated it, which made the layering unverified decoration. Every test here imports service and calls a function directly. Covers the partial trailing line (a reader arriving mid-append must leave the incomplete line for next time, not discard the one event being written when it looked), offset stability, dead-pid reconciliation, and died-never-relays-0. Proven to fail: making a died job return 0 tripped the exit-0 assertion by name.
|
||||
read_events was split into read_events(job_id) and read_events_from(path) so a test can drive it against a temp file — better shape regardless.
|
||||
|
||||
NOT YET EXERCISED, and honest about it: `jobs log --follow` against a genuinely LONG-running job. Nothing in reach currently runs long enough to tail. The offset mechanics underneath it are tested directly, and the live loop gets its first real workout when a slow domain lands (planet or db, T-1250).', 'done', 'high', NULL, NULL, 'D-263', '2026-08-31 13:52:06.722', '2026-08-31 15:02:40.198', NULL, '9516817a097607051c57cedd37210d22', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
|
||||
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5WNGDMENP7AY9VMB5G3XM', 'task', '06G1S7NVJR0GT9KWS9QVYNNFMM', 'Exit codes survive a detached run — the non-negotiable', 'A runner that reports started and then loses the failure is the exit-0 trap from the top of D-263 relocated somewhere nothing is watching, which is strictly worse than the original: at least a foreground exit 0 is in front of someone. Requirements: reach jobs wait <id> EXITS WITH THE JOB EXIT CODE, so a Makefile or hook can gate on a detached run exactly as it would on a foreground one; a failed job that nobody waited on is visible as failed in reach jobs list rather than merely absent; and reach --detach itself exits 0 for SUCCESSFULLY STARTING, which is a different claim from the job succeeding, so the output must say so in words rather than leaving a reader to infer it. TEST THE FAILING PATH FIRST and give it a deliberately failing command, since a job runner that has only ever run successful jobs has never been tested — this is the same discipline that caught the gate tests. Also test the case where the job fails BEFORE the parent has exited, and where it fails long AFTER, since those exercise different halves of the recording path.', 'in_progress', 'high', NULL, NULL, 'D-263', '2026-08-31 13:52:15.747', '2026-08-31 15:06:54.261', NULL, '4ca7d9770f698bf0ecd7e925b624d916', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
|
||||
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5WNGDMENP7AY9VMB5G3XM', 'task', '06G1S7NVJR0GT9KWS9QVYNNFMM', 'Exit codes survive a detached run — the non-negotiable', 'A runner that reports started and then loses the failure is the exit-0 trap from the top of D-263 relocated somewhere nothing is watching, which is strictly worse than the original: at least a foreground exit 0 is in front of someone. Requirements: reach jobs wait <id> EXITS WITH THE JOB EXIT CODE, so a Makefile or hook can gate on a detached run exactly as it would on a foreground one; a failed job that nobody waited on is visible as failed in reach jobs list rather than merely absent; and reach --detach itself exits 0 for SUCCESSFULLY STARTING, which is a different claim from the job succeeding, so the output must say so in words rather than leaving a reader to infer it. TEST THE FAILING PATH FIRST and give it a deliberately failing command, since a job runner that has only ever run successful jobs has never been tested — this is the same discipline that caught the gate tests. Also test the case where the job fails BEFORE the parent has exited, and where it fails long AFTER, since those exercise different halves of the recording path.', 'in_progress', 'high', NULL, NULL, 'D-263', '2026-08-31 13:52:15.747', '2026-08-31 15:07:19.024', NULL, '3621fd2c6aa8fb8791b303ea74584f08', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
|
||||
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5WNGDMENP7AY9VMB5G3XM', 'task', '06G1S7NVJR0GT9KWS9QVYNNFMM', 'Exit codes survive a detached run — the non-negotiable', 'A runner that reports started and then loses the failure is the exit-0 trap from the top of D-263 relocated somewhere nothing is watching, which is strictly worse than the original: at least a foreground exit 0 is in front of someone. Requirements: reach jobs wait <id> EXITS WITH THE JOB EXIT CODE, so a Makefile or hook can gate on a detached run exactly as it would on a foreground one; a failed job that nobody waited on is visible as failed in reach jobs list rather than merely absent; and reach --detach itself exits 0 for SUCCESSFULLY STARTING, which is a different claim from the job succeeding, so the output must say so in words rather than leaving a reader to infer it. TEST THE FAILING PATH FIRST and give it a deliberately failing command, since a job runner that has only ever run successful jobs has never been tested — this is the same discipline that caught the gate tests. Also test the case where the job fails BEFORE the parent has exited, and where it fails long AFTER, since those exercise different halves of the recording path.
|
||||
|
||||
DONE 2026-08-31. All four properties verified and pinned in tooling/test_job_exit_codes.py, wired into make test-tooling.
|
||||
|
||||
BUILT A REAL VERB TO TEST WITH, rather than a test hook. The two timing cases this ticket names — fails BEFORE the parent exits, fails long AFTER — cannot be distinguished when every command in reach finishes in milliseconds. So `reach dev selftest --seconds N --fail --exit-code C` now exists: it emits progress events over N seconds then optionally fails. It is a genuine diagnostic ("does streaming work end to end on this machine, can I tail it, does a failure survive detach?"), it lives in the `dev` domain the map already planned, and it is the only command slow enough to answer those by observation instead of argument. A test-only hook would have been the same code with less value.
|
||||
|
||||
VERIFIED, and the slow case is the one that proves the design:
|
||||
- FAST failure (0s): wait relays exit 3.
|
||||
- SLOW failure (6s real / 2s in the test): --detach returned in 75 ms while the child ran 6 seconds, so the parent was demonstrably gone long before the child failed — and wait still relayed exit 7. That is the half of the recording path only this case reaches, and it is why T-1277 moved completion recording to the child.
|
||||
- SUCCESS: wait relays 0.
|
||||
- A failed job nobody waited on shows as `failed` in jobs list with its full command.
|
||||
- --detach exits 0 AND says "this exit status means STARTED, not succeeded" in words. The test asserts on that wording, not just the code, because a bare 0 invites exactly the wrong reading.
|
||||
- Every event in a detached job''s log carries the job id.
|
||||
|
||||
CLOSED T-1278''s OPEN GAP as a side effect. `jobs log --follow` had never been exercised against a genuinely long job because none existed. With selftest it now has been: attached to a running 6-second job mid-flight, streamed the remaining steps live as they happened, and caught the final failure verdict with its remedy after the job ended. The follow loop and its final post-completion read both work.
|
||||
|
||||
PROVEN TO FAIL: made effective_exit_code always return 0 — the exit-0 trap itself — and both the fast and slow cases failed by name, reporting the expected code against the 0 they got. Restored, green.
|
||||
|
||||
NOTE ON WHAT --follow RETURNS: `jobs log --follow` exits 0 even when the job it tailed failed. That is correct and deliberate — log succeeded at logging. `wait` is the verb that relays an outcome, and conflating the two would mean you could not read a failed job''s log without your shell treating the read as a failure.', 'in_progress', 'high', NULL, NULL, 'D-263', '2026-08-31 13:52:15.747', '2026-08-31 15:14:09.188', NULL, '3bd99b15e59df3b447ab404740ee63fa', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
|
||||
|
||||
Reference in New Issue
Block a user