feat(config): T-1278 — the jobs domain, and typer.Exit is not a SystemExit

reach jobs list / status / log --follow / wait. A domain rather than core/,
because these verbs carry logic and state: they reconcile recorded status
against process liveness, tail a file from an offset, and relay an exit code.

Found a latent bug in already-committed code before building on it. typer.Exit
is a RuntimeError, not a SystemExit, so @handle_errors caught it like any other
unexpected exception: `raise typer.Exit(3)` inside a decorated command printed
"unexpected Exit: 3" and exited 1, silently discarding the requested code.
Nothing hit it because the check router had been converted to ReachError — but
jobs wait needs exactly this and it is what anyone would naturally write. Added
core/errors.ReachExit as the sanctioned control-flow exit, passed straight
through with no verdict. ReachError would have been wrong twice: a failure
verdict for a command that worked, and a demand for a fix= where there is no
remedy.

Reconciliation proved out on a real corpse rather than a simulated one — the
job stranded by the T-1277 bug, status "running" with its process long gone,
now reports as died. DIED is derived, never recorded, because a process killed
outright cannot write its own ending. It relays 137, never 0: a died job has no
exit code of its own and borrowing success points the exit-0 trap straight at
whatever gated on the run.

Second UTC bug of the same family as T-1276's: jobs list reported a job started
minutes earlier as running for 133m, because _parse used mktime on a UTC stamp
and silently added the offset to every duration.

console.render() is public now, so jobs log replays stored events through the
same path a live run prints them — a second renderer would drift, and the
divergence would surface exactly when someone is reading a log to find out what
went wrong.

test_jobs.py closes the gap T-1257 named: D-263 claims services are callable
without a CLI round trip, and nothing had ever demonstrated it, which left the
layering as unverified decoration. Every test here calls the service directly.

Not yet exercised, and said plainly: log --follow against a genuinely
long-running job. Nothing in reach runs long enough to tail yet. The offset
mechanics underneath are tested; the live loop waits for a slow domain.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-31 17:02:36 +02:00
co-authored by Claude Opus 5
parent c924b0934e
commit 6f08cc9156
11 changed files with 582 additions and 0 deletions
+28
View File
@@ -1933,3 +1933,31 @@ reach --help is still 73 ms, so the entry-point wrapper costs nothing on the fas
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5VJ89C7Q4EPR6Q73FSS74', 'description', 'The user-facing verbs, and a DOMAIN rather than core/ because they carry logic and state of their own — the first real test of the D-263 core bound, which it passes. Deliverables: reach jobs list (recent jobs with status, command and duration), status <id>, log <id> with --follow to tail, and wait <id>. REATTACH IS A BYTE OFFSET into an append-only file, which is the entire reason no daemon is needed: a caller can attach, drop off, and come back without losing anything, and there is no lifecycle to get wrong, nothing to orphan, and no stale state to reconcile. log --follow is therefore a poll on file length, not a subscription. Render the JSONL through the same path a live terminal uses, so a tailed log and a live run are the same artefact in two presentations rather than two renderers that drift. Note for the port: this domain is the first one written from scratch under the full contract rather than ported, so it doubles as the worked example the reach skill (T-1254) should show.', 'The user-facing verbs, and a DOMAIN rather than core/ because they carry logic and state of their own — the first real test of the D-263 core bound, which it passes. Deliverables: reach jobs list (recent jobs with status, command and duration), status <id>, log <id> with --follow to tail, and wait <id>. REATTACH IS A BYTE OFFSET into an append-only file, which is the entire reason no daemon is needed: a caller can attach, drop off, and come back without losing anything, and there is no lifecycle to get wrong, nothing to orphan, and no stale state to reconcile. log --follow is therefore a poll on file length, not a subscription. Render the JSONL through the same path a live terminal uses, so a tailed log and a live run are the same artefact in two presentations rather than two renderers that drift. Note for the port: this domain is the first one written from scratch under the full contract rather than ported, so it doubles as the worked example the reach skill (T-1254) should show. INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5VJ89C7Q4EPR6Q73FSS74', 'description', 'The user-facing verbs, and a DOMAIN rather than core/ because they carry logic and state of their own — the first real test of the D-263 core bound, which it passes. Deliverables: reach jobs list (recent jobs with status, command and duration), status <id>, log <id> with --follow to tail, and wait <id>. REATTACH IS A BYTE OFFSET into an append-only file, which is the entire reason no daemon is needed: a caller can attach, drop off, and come back without losing anything, and there is no lifecycle to get wrong, nothing to orphan, and no stale state to reconcile. log --follow is therefore a poll on file length, not a subscription. Render the JSONL through the same path a live terminal uses, so a tailed log and a live run are the same artefact in two presentations rather than two renderers that drift. Note for the port: this domain is the first one written from scratch under the full contract rather than ported, so it doubles as the worked example the reach skill (T-1254) should show.', 'The user-facing verbs, and a DOMAIN rather than core/ because they carry logic and state of their own — the first real test of the D-263 core bound, which it passes. Deliverables: reach jobs list (recent jobs with status, command and duration), status <id>, log <id> with --follow to tail, and wait <id>. REATTACH IS A BYTE OFFSET into an append-only file, which is the entire reason no daemon is needed: a caller can attach, drop off, and come back without losing anything, and there is no lifecycle to get wrong, nothing to orphan, and no stale state to reconcile. log --follow is therefore a poll on file length, not a subscription. Render the JSONL through the same path a live terminal uses, so a tailed log and a live run are the same artefact in two presentations rather than two renderers that drift. Note for the port: this domain is the first one written from scratch under the full contract rather than ported, so it doubles as the worked example the reach skill (T-1254) should show.
FROM T-1277 (2026-08-31) the reconciliation requirement is yours and it is not optional. A child killed outright (SIGKILL, OOM, an interpreter crash) never gets to record its own completion, so its metadata file stays status=running with the process long gone. jobs list and jobs status must therefore RECONCILE against process.is_alive(pid) rather than trusting the file: a job whose pid is dead and whose status still reads running is not running, it died. Report it as such ''died without recording an exit'' is honest and actionable, whereas showing it as running is the exit-0 trap in a place nobody is watching, and a caller polling for completion would wait forever on something that failed in milliseconds. Note the ordinary failure paths are already covered: T-1277 moved completion recording to the process''s exit (tooling/main.py main()), so bad arguments, unknown verbs, ReachErrors and unhandled exceptions all record correctly. What remains is only the case where the process cannot run code at all.', NULL, '2026-08-31 14:46:03', '2026-08-31 14:46:03.939', '2026-08-31 14:46:03.939', NULL, 'a02f3f35330e271011c615aef8f9a141', 2) ON CONFLICT(hash) DO NOTHING; FROM T-1277 (2026-08-31) the reconciliation requirement is yours and it is not optional. A child killed outright (SIGKILL, OOM, an interpreter crash) never gets to record its own completion, so its metadata file stays status=running with the process long gone. jobs list and jobs status must therefore RECONCILE against process.is_alive(pid) rather than trusting the file: a job whose pid is dead and whose status still reads running is not running, it died. Report it as such ''died without recording an exit'' is honest and actionable, whereas showing it as running is the exit-0 trap in a place nobody is watching, and a caller polling for completion would wait forever on something that failed in milliseconds. Note the ordinary failure paths are already covered: T-1277 moved completion recording to the process''s exit (tooling/main.py main()), so bad arguments, unknown verbs, ReachErrors and unhandled exceptions all record correctly. What remains is only the case where the process cannot run code at all.', NULL, '2026-08-31 14:46:03', '2026-08-31 14:46:03.939', '2026-08-31 14:46:03.939', NULL, 'a02f3f35330e271011c615aef8f9a141', 2) ON CONFLICT(hash) DO NOTHING;
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5TWD0VVBM4F2WZYAQFET0', 'status', 'in_progress', 'done', NULL, '2026-08-31 14:46:22', '2026-08-31 14:46:22.896', '2026-08-31 14:46:22.896', NULL, '36f5973de73c625bf83f3a4b7c6550c9', 2) ON CONFLICT(hash) DO NOTHING;
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5VJ89C7Q4EPR6Q73FSS74', 'status', 'backlog', 'in_progress', NULL, '2026-08-31 14:52:17', '2026-08-31 14:52:17.707', '2026-08-31 14:52:17.707', NULL, '30c48c89065bcea3bdec59fc056c2358', 2) ON CONFLICT(hash) DO NOTHING;
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5VJ89C7Q4EPR6Q73FSS74', 'status', 'in_progress', 'in_progress', NULL, '2026-08-31 14:52:36', '2026-08-31 14:52:36.710', '2026-08-31 14:52:36.710', NULL, 'b5e4dd4502af98fa5342e25449f740f5', 2) ON CONFLICT(hash) DO NOTHING;
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5VJ89C7Q4EPR6Q73FSS74', 'description', 'The user-facing verbs, and a DOMAIN rather than core/ because they carry logic and state of their own — the first real test of the D-263 core bound, which it passes. Deliverables: reach jobs list (recent jobs with status, command and duration), status <id>, log <id> with --follow to tail, and wait <id>. REATTACH IS A BYTE OFFSET into an append-only file, which is the entire reason no daemon is needed: a caller can attach, drop off, and come back without losing anything, and there is no lifecycle to get wrong, nothing to orphan, and no stale state to reconcile. log --follow is therefore a poll on file length, not a subscription. Render the JSONL through the same path a live terminal uses, so a tailed log and a live run are the same artefact in two presentations rather than two renderers that drift. Note for the port: this domain is the first one written from scratch under the full contract rather than ported, so it doubles as the worked example the reach skill (T-1254) should show.
FROM T-1277 (2026-08-31) the reconciliation requirement is yours and it is not optional. A child killed outright (SIGKILL, OOM, an interpreter crash) never gets to record its own completion, so its metadata file stays status=running with the process long gone. jobs list and jobs status must therefore RECONCILE against process.is_alive(pid) rather than trusting the file: a job whose pid is dead and whose status still reads running is not running, it died. Report it as such ''died without recording an exit'' is honest and actionable, whereas showing it as running is the exit-0 trap in a place nobody is watching, and a caller polling for completion would wait forever on something that failed in milliseconds. Note the ordinary failure paths are already covered: T-1277 moved completion recording to the process''s exit (tooling/main.py main()), so bad arguments, unknown verbs, ReachErrors and unhandled exceptions all record correctly. What remains is only the case where the process cannot run code at all.', 'The user-facing verbs, and a DOMAIN rather than core/ because they carry logic and state of their own the first real test of the D-263 core bound, which it passes. Deliverables: reach jobs list (recent jobs with status, command and duration), status <id>, log <id> with --follow to tail, and wait <id>. REATTACH IS A BYTE OFFSET into an append-only file, which is the entire reason no daemon is needed: a caller can attach, drop off, and come back without losing anything, and there is no lifecycle to get wrong, nothing to orphan, and no stale state to reconcile. log --follow is therefore a poll on file length, not a subscription. Render the JSONL through the same path a live terminal uses, so a tailed log and a live run are the same artefact in two presentations rather than two renderers that drift. Note for the port: this domain is the first one written from scratch under the full contract rather than ported, so it doubles as the worked example the reach skill (T-1254) should show.
FROM T-1277 (2026-08-31) the reconciliation requirement is yours and it is not optional. A child killed outright (SIGKILL, OOM, an interpreter crash) never gets to record its own completion, so its metadata file stays status=running with the process long gone. jobs list and jobs status must therefore RECONCILE against process.is_alive(pid) rather than trusting the file: a job whose pid is dead and whose status still reads running is not running, it died. Report it as such ''died without recording an exit'' is honest and actionable, whereas showing it as running is the exit-0 trap in a place nobody is watching, and a caller polling for completion would wait forever on something that failed in milliseconds. Note the ordinary failure paths are already covered: T-1277 moved completion recording to the process''s exit (tooling/main.py main()), so bad arguments, unknown verbs, ReachErrors and unhandled exceptions all record correctly. What remains is only the case where the process cannot run code at all.
DONE 2026-08-31. reach jobs list / status / log --follow / wait, all four working, plus service-level tests.
A LATENT BUG FOUND BEFORE BUILDING ON IT, and it was already committed. typer.Exit is a RuntimeError, NOT a SystemExit so @handle_errors caught it like any other unexpected exception. Verified: `raise typer.Exit(3)` inside a decorated command printed "unexpected Exit: 3" and exited 1, SILENTLY DISCARDING the requested code. Nothing hit it today because T-1267 had converted the check router to ReachError, but `jobs wait` needs exactly this and it is the natural thing anyone would write.
FIX: core/errors.ReachExit(code) as the sanctioned control-flow exit, passed straight through by handle_errors as SystemExit with no verdict. ReachError would have been wrong twice over for wait it prints a failure verdict for a command that worked, and demands a fix= for a situation with no remedy. Kept typer out of core/, which the conformance test enforces.
RECONCILIATION WORKS, AND WAS PROVEN ON A REAL CORPSE rather than a simulated one. The job left stranded by the T-1277 bug status "running", process long gone now reports as `died` with its true elapsed time. Status DIED is derived, never recorded: a process killed outright cannot write its own ending.
EXIT-CODE RELAY, all three cases: done -> 0, failed -> 2 (the job''s own code), died -> 137. A died job must NEVER relay 0; it has no code of its own and borrowing success is the exit-0 trap pointed at whatever gated on the run.
BYTE-OFFSET REATTACH verified directly: first read consumed one event to offset 169; resuming from 169 returned nothing and left the offset unmoved. That is the whole reason no daemon is needed.
SECOND UTC BUG, same class as T-1276''s. jobs list reported a job started minutes earlier as running for 133m _parse used time.mktime, which reads a UTC stamp as local time and silently adds the offset to every duration. Now calendar.timegm. Both directions of this conversion have now bitten once each; both have the reason in a comment.
RENDERING: console.render() is now public and jobs log replays stored events through it. A second renderer for stored events would drift from the live one, and the divergence would surface exactly when someone is reading a log to work out what went wrong.
SERVICE-LEVEL TESTS tooling/test_jobs.py, and these close the gap T-1257 named. D-263 claims a service is transport-agnostic and callable without a CLI round trip; nothing had ever demonstrated it, which made the layering unverified decoration. Every test here imports service and calls a function directly. Covers the partial trailing line (a reader arriving mid-append must leave the incomplete line for next time, not discard the one event being written when it looked), offset stability, dead-pid reconciliation, and died-never-relays-0. Proven to fail: making a died job return 0 tripped the exit-0 assertion by name.
read_events was split into read_events(job_id) and read_events_from(path) so a test can drive it against a temp file better shape regardless.
NOT YET EXERCISED, and honest about it: `jobs log --follow` against a genuinely LONG-running job. Nothing in reach currently runs long enough to tail. The offset mechanics underneath it are tested directly, and the live loop gets its first real workout when a slow domain lands (planet or db, T-1250).', NULL, '2026-08-31 15:02:20', '2026-08-31 15:02:20.709', '2026-08-31 15:02:20.709', NULL, 'becbc04f4ea86ae9f73e0c9541992300', 2) ON CONFLICT(hash) DO NOTHING;
+51
View File
@@ -2236,3 +2236,54 @@ reach --help is still 73 ms, so the entry-point wrapper costs nothing on the fas
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5VJ89C7Q4EPR6Q73FSS74', 'task', '06G1S7NVJR0GT9KWS9QVYNNFMM', 'The jobs domain — list, status, log --follow, wait', 'The user-facing verbs, and a DOMAIN rather than core/ because they carry logic and state of their own — the first real test of the D-263 core bound, which it passes. Deliverables: reach jobs list (recent jobs with status, command and duration), status <id>, log <id> with --follow to tail, and wait <id>. REATTACH IS A BYTE OFFSET into an append-only file, which is the entire reason no daemon is needed: a caller can attach, drop off, and come back without losing anything, and there is no lifecycle to get wrong, nothing to orphan, and no stale state to reconcile. log --follow is therefore a poll on file length, not a subscription. Render the JSONL through the same path a live terminal uses, so a tailed log and a live run are the same artefact in two presentations rather than two renderers that drift. Note for the port: this domain is the first one written from scratch under the full contract rather than ported, so it doubles as the worked example the reach skill (T-1254) should show. INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5VJ89C7Q4EPR6Q73FSS74', 'task', '06G1S7NVJR0GT9KWS9QVYNNFMM', 'The jobs domain — list, status, log --follow, wait', 'The user-facing verbs, and a DOMAIN rather than core/ because they carry logic and state of their own — the first real test of the D-263 core bound, which it passes. Deliverables: reach jobs list (recent jobs with status, command and duration), status <id>, log <id> with --follow to tail, and wait <id>. REATTACH IS A BYTE OFFSET into an append-only file, which is the entire reason no daemon is needed: a caller can attach, drop off, and come back without losing anything, and there is no lifecycle to get wrong, nothing to orphan, and no stale state to reconcile. log --follow is therefore a poll on file length, not a subscription. Render the JSONL through the same path a live terminal uses, so a tailed log and a live run are the same artefact in two presentations rather than two renderers that drift. Note for the port: this domain is the first one written from scratch under the full contract rather than ported, so it doubles as the worked example the reach skill (T-1254) should show.
FROM T-1277 (2026-08-31) the reconciliation requirement is yours and it is not optional. A child killed outright (SIGKILL, OOM, an interpreter crash) never gets to record its own completion, so its metadata file stays status=running with the process long gone. jobs list and jobs status must therefore RECONCILE against process.is_alive(pid) rather than trusting the file: a job whose pid is dead and whose status still reads running is not running, it died. Report it as such ''died without recording an exit'' is honest and actionable, whereas showing it as running is the exit-0 trap in a place nobody is watching, and a caller polling for completion would wait forever on something that failed in milliseconds. Note the ordinary failure paths are already covered: T-1277 moved completion recording to the process''s exit (tooling/main.py main()), so bad arguments, unknown verbs, ReachErrors and unhandled exceptions all record correctly. What remains is only the case where the process cannot run code at all.', 'backlog', 'high', NULL, NULL, 'D-263', '2026-08-31 13:52:06.722', '2026-08-31 14:46:03.939', NULL, 'f9aac99fd6449919ab2d48455a81b6e7', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at; FROM T-1277 (2026-08-31) the reconciliation requirement is yours and it is not optional. A child killed outright (SIGKILL, OOM, an interpreter crash) never gets to record its own completion, so its metadata file stays status=running with the process long gone. jobs list and jobs status must therefore RECONCILE against process.is_alive(pid) rather than trusting the file: a job whose pid is dead and whose status still reads running is not running, it died. Report it as such ''died without recording an exit'' is honest and actionable, whereas showing it as running is the exit-0 trap in a place nobody is watching, and a caller polling for completion would wait forever on something that failed in milliseconds. Note the ordinary failure paths are already covered: T-1277 moved completion recording to the process''s exit (tooling/main.py main()), so bad arguments, unknown verbs, ReachErrors and unhandled exceptions all record correctly. What remains is only the case where the process cannot run code at all.', 'backlog', 'high', NULL, NULL, 'D-263', '2026-08-31 13:52:06.722', '2026-08-31 14:46:03.939', NULL, 'f9aac99fd6449919ab2d48455a81b6e7', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5TWD0VVBM4F2WZYAQFET0', 'task', '06G1S7NVJR0GT9KWS9QVYNNFMM', 'core/process.py — detach so the child outlives the parent', 'The spawn primitive, and it is substrate rather than a domain because it has no verbs of its own. Deliverables: spawn a detached child that survives the parent exiting (setsid or equivalent, not just a background shell job, since a killed parent must not take the work with it); redirect the child''s event stream to .cache/reach/jobs/<id>.jsonl and its stdout to a sibling file, keeping the two channels separate exactly as they are in the foreground; write a metadata record carrying command, argv, start time, pid and — on completion — end time and exit code. The metadata file is what makes a finished job readable without re-reading a possibly enormous log. .cache/ is already gitignored. WATCH: the child must re-exec the same reach that was invoked, resolved by bare name per T-1261''s negative criterion, never by an interpreter path or a .venv path — an absolute path here would break the moment the tool is re-pointed at another checkout, and would be a silent wrong-source failure of exactly the kind make reach-repoint exists to fix. Also watch the completion race: the exit code must be recorded by the CHILD as its last act, not polled by a parent that may already be gone.
DONE 2026-08-31. Detached execution works end to end, and testing it found a real hole that the design as written would have shipped.
DELIVERED: tooling/core/process.py (spawn, metadata, liveness), the --detach flag on the root callback, and completion recording. Verified against a real spawn — parent returns the job id and exits 0, child runs on and writes .cache/reach/jobs/<id>.jsonl tagged with that id, plus a .out sibling and a .json metadata record.
THE THREE THINGS THE TICKET FLAGGED, each handled and each verified:
- start_new_session=True, so the child gets its own session and process group and a signal to the parent''s group does not take the work with it.
- The child re-execs `reach` by BARE NAME. An absolute path would freeze it to whichever checkout was current at spawn time, so after make reach-repoint a detached job would silently run the wrong source the exact failure that command exists to fix.
- The child records its own exit code. Confirmed on the success path (status done, exit_code 0) and on a real failure path (status failed, exit_code 1) using a drift fixture pointed at by SR_REPO_ROOT.
THE HOLE, found only because I tested a THIRD case the ticket did not name.
Recording completion inside @command looked right and was subtly wrong. A child that fails BEFORE any command runs bad arguments, an unknown verb, an import error never reaches that decorator. Verified: `reach --detach check bogus` left its metadata reading status "running" FOREVER, with the process long gone.
That is the exit-0 trap wearing a new disguise, and worse than the original: a failed job that looks busy, in a place nobody is watching. A caller polling for completion would wait indefinitely on a job that failed in milliseconds.
FIX: completion is now recorded at the PROCESS''s exit rather than a command''s. tooling/main.py gains main(), which wraps cli() in one try/finally, and [project.scripts] points at main:main instead of main:cli. Every exit path success, ReachError, usage error, unhandled exception now passes through a single finally. Re-verified: `reach --detach check bogus` records status failed, exit_code 2.
The recording was REMOVED from @command rather than left in both places; two writers of the same field is how they drift.
CONFORMANCE EXEMPTION ADDED, deliberately narrow. The new no-domain-imports-core.jobs invariant fired on main.py, correctly by its letter and wrongly by its purpose. main.py is not a command it is the entry point, and it already owns --detach, --verbose and --no-input. Reading a job-id constant there is far less coupled than the --detach flag it already carries. Exempted main.py explicitly, with the reason inline so it does not read as an oversight.
STILL OPEN, and correctly belongs to T-1278: a child killed outright (SIGKILL, OOM, interpreter crash) still cannot record anything, so its metadata stays "running". process.is_alive(pid) exists for exactly this, and jobs list/status must reconcile against it rather than trusting the file.
reach --help is still 73 ms, so the entry-point wrapper costs nothing on the fast path.', 'done', 'high', NULL, NULL, 'D-263', '2026-08-31 13:52:01.128', '2026-08-31 14:46:22.895', NULL, '1d5d86a8353e16e113d1c77588a90e55', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5VJ89C7Q4EPR6Q73FSS74', 'task', '06G1S7NVJR0GT9KWS9QVYNNFMM', 'The jobs domain — list, status, log --follow, wait', 'The user-facing verbs, and a DOMAIN rather than core/ because they carry logic and state of their own — the first real test of the D-263 core bound, which it passes. Deliverables: reach jobs list (recent jobs with status, command and duration), status <id>, log <id> with --follow to tail, and wait <id>. REATTACH IS A BYTE OFFSET into an append-only file, which is the entire reason no daemon is needed: a caller can attach, drop off, and come back without losing anything, and there is no lifecycle to get wrong, nothing to orphan, and no stale state to reconcile. log --follow is therefore a poll on file length, not a subscription. Render the JSONL through the same path a live terminal uses, so a tailed log and a live run are the same artefact in two presentations rather than two renderers that drift. Note for the port: this domain is the first one written from scratch under the full contract rather than ported, so it doubles as the worked example the reach skill (T-1254) should show.
FROM T-1277 (2026-08-31) the reconciliation requirement is yours and it is not optional. A child killed outright (SIGKILL, OOM, an interpreter crash) never gets to record its own completion, so its metadata file stays status=running with the process long gone. jobs list and jobs status must therefore RECONCILE against process.is_alive(pid) rather than trusting the file: a job whose pid is dead and whose status still reads running is not running, it died. Report it as such ''died without recording an exit'' is honest and actionable, whereas showing it as running is the exit-0 trap in a place nobody is watching, and a caller polling for completion would wait forever on something that failed in milliseconds. Note the ordinary failure paths are already covered: T-1277 moved completion recording to the process''s exit (tooling/main.py main()), so bad arguments, unknown verbs, ReachErrors and unhandled exceptions all record correctly. What remains is only the case where the process cannot run code at all.', 'in_progress', 'high', NULL, NULL, 'D-263', '2026-08-31 13:52:06.722', '2026-08-31 14:52:17.707', NULL, '5c00db52e693d19da200416e4d3901a5', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5VJ89C7Q4EPR6Q73FSS74', 'task', '06G1S7NVJR0GT9KWS9QVYNNFMM', 'The jobs domain — list, status, log --follow, wait', 'The user-facing verbs, and a DOMAIN rather than core/ because they carry logic and state of their own — the first real test of the D-263 core bound, which it passes. Deliverables: reach jobs list (recent jobs with status, command and duration), status <id>, log <id> with --follow to tail, and wait <id>. REATTACH IS A BYTE OFFSET into an append-only file, which is the entire reason no daemon is needed: a caller can attach, drop off, and come back without losing anything, and there is no lifecycle to get wrong, nothing to orphan, and no stale state to reconcile. log --follow is therefore a poll on file length, not a subscription. Render the JSONL through the same path a live terminal uses, so a tailed log and a live run are the same artefact in two presentations rather than two renderers that drift. Note for the port: this domain is the first one written from scratch under the full contract rather than ported, so it doubles as the worked example the reach skill (T-1254) should show.
FROM T-1277 (2026-08-31) the reconciliation requirement is yours and it is not optional. A child killed outright (SIGKILL, OOM, an interpreter crash) never gets to record its own completion, so its metadata file stays status=running with the process long gone. jobs list and jobs status must therefore RECONCILE against process.is_alive(pid) rather than trusting the file: a job whose pid is dead and whose status still reads running is not running, it died. Report it as such ''died without recording an exit'' is honest and actionable, whereas showing it as running is the exit-0 trap in a place nobody is watching, and a caller polling for completion would wait forever on something that failed in milliseconds. Note the ordinary failure paths are already covered: T-1277 moved completion recording to the process''s exit (tooling/main.py main()), so bad arguments, unknown verbs, ReachErrors and unhandled exceptions all record correctly. What remains is only the case where the process cannot run code at all.', 'in_progress', 'high', NULL, NULL, 'D-263', '2026-08-31 13:52:06.722', '2026-08-31 14:52:36.710', NULL, '0a348ae5a631dcd868a7830e0be0de6c', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G5G5VJ89C7Q4EPR6Q73FSS74', 'task', '06G1S7NVJR0GT9KWS9QVYNNFMM', 'The jobs domain — list, status, log --follow, wait', 'The user-facing verbs, and a DOMAIN rather than core/ because they carry logic and state of their own — the first real test of the D-263 core bound, which it passes. Deliverables: reach jobs list (recent jobs with status, command and duration), status <id>, log <id> with --follow to tail, and wait <id>. REATTACH IS A BYTE OFFSET into an append-only file, which is the entire reason no daemon is needed: a caller can attach, drop off, and come back without losing anything, and there is no lifecycle to get wrong, nothing to orphan, and no stale state to reconcile. log --follow is therefore a poll on file length, not a subscription. Render the JSONL through the same path a live terminal uses, so a tailed log and a live run are the same artefact in two presentations rather than two renderers that drift. Note for the port: this domain is the first one written from scratch under the full contract rather than ported, so it doubles as the worked example the reach skill (T-1254) should show.
FROM T-1277 (2026-08-31) the reconciliation requirement is yours and it is not optional. A child killed outright (SIGKILL, OOM, an interpreter crash) never gets to record its own completion, so its metadata file stays status=running with the process long gone. jobs list and jobs status must therefore RECONCILE against process.is_alive(pid) rather than trusting the file: a job whose pid is dead and whose status still reads running is not running, it died. Report it as such ''died without recording an exit'' is honest and actionable, whereas showing it as running is the exit-0 trap in a place nobody is watching, and a caller polling for completion would wait forever on something that failed in milliseconds. Note the ordinary failure paths are already covered: T-1277 moved completion recording to the process''s exit (tooling/main.py main()), so bad arguments, unknown verbs, ReachErrors and unhandled exceptions all record correctly. What remains is only the case where the process cannot run code at all.
DONE 2026-08-31. reach jobs list / status / log --follow / wait, all four working, plus service-level tests.
A LATENT BUG FOUND BEFORE BUILDING ON IT, and it was already committed. typer.Exit is a RuntimeError, NOT a SystemExit so @handle_errors caught it like any other unexpected exception. Verified: `raise typer.Exit(3)` inside a decorated command printed "unexpected Exit: 3" and exited 1, SILENTLY DISCARDING the requested code. Nothing hit it today because T-1267 had converted the check router to ReachError, but `jobs wait` needs exactly this and it is the natural thing anyone would write.
FIX: core/errors.ReachExit(code) as the sanctioned control-flow exit, passed straight through by handle_errors as SystemExit with no verdict. ReachError would have been wrong twice over for wait it prints a failure verdict for a command that worked, and demands a fix= for a situation with no remedy. Kept typer out of core/, which the conformance test enforces.
RECONCILIATION WORKS, AND WAS PROVEN ON A REAL CORPSE rather than a simulated one. The job left stranded by the T-1277 bug status "running", process long gone now reports as `died` with its true elapsed time. Status DIED is derived, never recorded: a process killed outright cannot write its own ending.
EXIT-CODE RELAY, all three cases: done -> 0, failed -> 2 (the job''s own code), died -> 137. A died job must NEVER relay 0; it has no code of its own and borrowing success is the exit-0 trap pointed at whatever gated on the run.
BYTE-OFFSET REATTACH verified directly: first read consumed one event to offset 169; resuming from 169 returned nothing and left the offset unmoved. That is the whole reason no daemon is needed.
SECOND UTC BUG, same class as T-1276''s. jobs list reported a job started minutes earlier as running for 133m _parse used time.mktime, which reads a UTC stamp as local time and silently adds the offset to every duration. Now calendar.timegm. Both directions of this conversion have now bitten once each; both have the reason in a comment.
RENDERING: console.render() is now public and jobs log replays stored events through it. A second renderer for stored events would drift from the live one, and the divergence would surface exactly when someone is reading a log to work out what went wrong.
SERVICE-LEVEL TESTS tooling/test_jobs.py, and these close the gap T-1257 named. D-263 claims a service is transport-agnostic and callable without a CLI round trip; nothing had ever demonstrated it, which made the layering unverified decoration. Every test here imports service and calls a function directly. Covers the partial trailing line (a reader arriving mid-append must leave the incomplete line for next time, not discard the one event being written when it looked), offset stability, dead-pid reconciliation, and died-never-relays-0. Proven to fail: making a died job return 0 tripped the exit-0 assertion by name.
read_events was split into read_events(job_id) and read_events_from(path) so a test can drive it against a temp file better shape regardless.
NOT YET EXERCISED, and honest about it: `jobs log --follow` against a genuinely LONG-running job. Nothing in reach currently runs long enough to tail. The offset mechanics underneath it are tested directly, and the live loop gets its first real workout when a slow domain lands (planet or db, T-1250).', 'in_progress', 'high', NULL, NULL, 'D-263', '2026-08-31 13:52:06.722', '2026-08-31 15:02:20.709', NULL, '1ce1eb66a71783ab5218d27ae143b1aa', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
+4
View File
@@ -307,6 +307,10 @@ test-tooling:
@mkdir -p .cache @mkdir -p .cache
@$(VENV_PY) tooling/test_lazy_domains.py 2> .cache/test-tooling-lazy-domains.log || \ @$(VENV_PY) tooling/test_lazy_domains.py 2> .cache/test-tooling-lazy-domains.log || \
{ echo " FAIL: reach lazy registration — log follows:"; cat .cache/test-tooling-lazy-domains.log; exit 1; } { echo " FAIL: reach lazy registration — log follows:"; cat .cache/test-tooling-lazy-domains.log; exit 1; }
@echo " [test-tooling] reach jobs service units (T-1278)..."
@mkdir -p .cache
@$(VENV_PY) tooling/test_jobs.py 2> .cache/test-tooling-jobs.log || \
{ echo " FAIL: jobs service — log follows:"; cat .cache/test-tooling-jobs.log; exit 1; }
@echo " [test-tooling] reach D-263 conformance (T-1270)..." @echo " [test-tooling] reach D-263 conformance (T-1270)..."
@mkdir -p .cache @mkdir -p .cache
@$(VENV_PY) tooling/test_conformance.py 2> .cache/test-tooling-conformance.log || \ @$(VENV_PY) tooling/test_conformance.py 2> .cache/test-tooling-conformance.log || \
+12
View File
@@ -150,6 +150,18 @@ def _render_as_text(stream: TextIO) -> str | bool:
return False return False
def render(payload: dict[str, Any]) -> str:
"""Render one event as a human would see it live.
Public because `reach jobs log` replays a stored event stream through it.
A second renderer for stored events would drift from this one, and the
divergence would show up exactly when someone is reading a log to work out
what went wrong — the worst moment to be looking at output that does not
match what the live run printed.
"""
return _render(payload)
def _render(payload: dict[str, Any]) -> str: def _render(payload: dict[str, Any]) -> str:
message = payload.get("message", "") message = payload.get("message", "")
if payload.get("kind") == "verdict": if payload.get("kind") == "verdict":
+24
View File
@@ -42,6 +42,27 @@ class ReachError(Exception):
self.exit_code = exit_code if exit_code != 0 else 1 self.exit_code = exit_code if exit_code != 0 else 1
class ReachExit(Exception): # noqa: N818 control flow, not an error
"""Exit with a specific code, quietly. Not a failure.
For a command that must RELAY an exit code rather than report one — most
obviously `reach jobs wait`, which exits with the code of the job it waited
on. That is not `jobs wait` failing, so a `ReachError` would be wrong twice
over: it would print a failure verdict for a command that worked, and
demand a `fix=` for a situation with no remedy.
**Use this, never `typer.Exit`, inside a decorated command.** `typer.Exit`
is a `RuntimeError`, not a `SystemExit`, so `handle_errors` catches it like
any other unexpected exception — reporting "unexpected Exit: 3" and exiting
**1**, silently discarding the code that was asked for. The conformance
suite forbids it in routers so the trap cannot be re-entered.
"""
def __init__(self, exit_code: int = 0) -> None:
super().__init__(f"exit {exit_code}")
self.exit_code = exit_code
def unknown_choice(kind: str, given: str, accepted: Iterable[str]) -> ReachError: def unknown_choice(kind: str, given: str, accepted: Iterable[str]) -> ReachError:
"""Reject a value from a known finite set, naming the whole set. """Reject a value from a known finite set, naming the whole set.
@@ -70,6 +91,9 @@ def handle_errors(func: F) -> F:
def wrapper(*args: Any, **kwargs: Any) -> Any: def wrapper(*args: Any, **kwargs: Any) -> Any:
try: try:
return func(*args, **kwargs) return func(*args, **kwargs)
except ReachExit as exc:
# Control flow, not a failure — no verdict, just the code.
raise SystemExit(exc.exit_code) from None
except ReachError as exc: except ReachError as exc:
# The verdict prints ONCE, LAST, after whatever the command streamed. # The verdict prints ONCE, LAST, after whatever the command streamed.
# A remedy emitted mid-stream at line 400 of 900 is technically # A remedy emitted mid-stream at line 400 of 900 is technically
+12
View File
@@ -0,0 +1,12 @@
"""The `jobs` domain — detached runs, their logs, and their outcomes.
A domain rather than part of `core/`, and this is the first real test of that
bound (D-263). The *primitives* — spawn, detach, record — are substrate and live
in `core/process.py`. These verbs have logic and state of their own: they
reconcile recorded status against process liveness, tail a file from an offset,
and relay an exit code. A job store in `core/` would be exactly the drift the
record warns about.
Also the first domain written from scratch under the full contract rather than
ported, which makes it the worked example for the reach skill (T-1254).
"""
+116
View File
@@ -0,0 +1,116 @@
"""Transport for the `jobs` domain — args in, delegate, format out.
Zero logic. Reconciliation, offsets and polling all live in `service.py`; what
happens here is turning a `Job` into lines and an exit code.
"""
from __future__ import annotations
import time
import typer
from tooling.core import cli, console
from tooling.core.command import command
from tooling.core.errors import ReachExit
from tooling.domains.jobs import service
from tooling.domains.jobs.schemas import Status
app = cli.domain("jobs", "Detached runs — what is running, what it printed, how it ended.")
@app.callback()
def _domain() -> None:
"""Keeps `jobs` a group (Typer collapses a single-command app)."""
@app.command("list")
@command
def list_jobs(
limit: int = typer.Option(20, "--limit", "-n", help="How many recent jobs to show."),
) -> None:
"""Recent detached runs, newest first."""
jobs = service.list_jobs(limit)
if not jobs:
console.out("no jobs recorded — start one with: reach --detach <command>")
return
for job in jobs:
console.out(
f"{job.job} {job.status.value:<8} {service.duration(job):>7} {job.command}"
)
@app.command("status")
@command
def status(job_id: str = typer.Argument(..., help="Job id, as printed by --detach.")) -> None:
"""One job's outcome, reconciled against whether its process is alive."""
job = service.get(job_id)
console.out(f"job {job.job}")
console.out(f"command {job.command}")
console.out(f"status {job.status.value}")
console.out(f"started {job.started_at}")
console.out(f"elapsed {service.duration(job)}")
if job.exit_code is not None:
console.out(f"exit {job.exit_code}")
if job.status is Status.DIED:
# Said in words, because "died" alone reads like a synonym for "failed"
# and the distinction matters: nothing recorded an outcome here.
console.out("")
console.out("This job's process is gone but it never recorded an ending —")
console.out("killed outright (SIGKILL, OOM, or a crash). Its log holds")
console.out("whatever it managed to emit before that.")
@app.command("log")
@command
def log(
job_id: str = typer.Argument(..., help="Job id, as printed by --detach."),
follow: bool = typer.Option(False, "--follow", "-f", help="Keep printing as it runs."),
) -> None:
"""Replay a job's event stream, rendered as it appeared live."""
job = service.get(job_id)
events, offset = service.read_events(job_id)
for event in events:
# Rendered through console, not a local formatter, so a stored log and a
# live run are one artefact in two presentations rather than two
# renderers that drift apart precisely when someone is debugging.
console.out(console.render(event).rstrip("\n"))
if not follow:
return
while not job.status.finished:
time.sleep(service.POLL_SECONDS)
events, offset = service.read_events(job_id, offset)
for event in events:
console.out(console.render(event).rstrip("\n"))
job = service.get(job_id)
# One last read: the job may have written its final events between the last
# poll and its exit, and stopping at the status flip would drop them.
events, offset = service.read_events(job_id, offset)
for event in events:
console.out(console.render(event).rstrip("\n"))
@app.command("wait")
@command
def wait(
job_id: str = typer.Argument(..., help="Job id, as printed by --detach."),
timeout: float = typer.Option(None, "--timeout", help="Give up after N seconds."),
) -> None:
"""Block until a job finishes, then exit with ITS exit code.
That relay is the point: a Makefile or a hook can gate on a detached run
exactly as it would on a foreground one. `ReachExit` rather than
`ReachError` because waiting successfully for a job that failed is not a
failure of `wait`.
"""
job = service.wait(job_id, timeout)
console.verdict(
f"job {job.job} {job.status.value} after {service.duration(job)}"
+ (f" (exit {job.exit_code})" if job.exit_code is not None else ""),
ok=job.status is Status.DONE,
fix=None if job.status is Status.DONE else f"reach jobs log {job.job}",
)
raise ReachExit(job.effective_exit_code)
+55
View File
@@ -0,0 +1,55 @@
"""Data shapes for the `jobs` domain."""
from __future__ import annotations
from enum import Enum
from pydantic import BaseModel, ConfigDict
class Status(str, Enum):
"""What a job is doing, after reconciliation.
`RUNNING`, `DONE` and `FAILED` are recorded by the job itself. **`DIED` is
never recorded** — it is derived, when a job's file still says running but
its pid is gone. A process killed outright (SIGKILL, OOM, an interpreter
crash) cannot write its own ending, so trusting the file would leave a
corpse looking busy forever, and anything polling for completion would wait
on it indefinitely.
"""
RUNNING = "running"
DONE = "done"
FAILED = "failed"
DIED = "died"
@property
def finished(self) -> bool:
return self is not Status.RUNNING
class Job(BaseModel):
"""One detached run."""
model_config = ConfigDict(frozen=True)
job: str
command: str
argv: list[str] = []
pid: int
started_at: str
status: Status
exit_code: int | None = None
ended_at: str | None = None
@property
def effective_exit_code(self) -> int:
"""The code a caller should adopt when relaying this job's outcome.
A job that died without recording anything has no code of its own. It
must not be reported as 0 — that is the exit-0 trap, and the whole
reason `DIED` is distinguished from `DONE`.
"""
if self.status is Status.DIED:
return 137 # conventional 128+SIGKILL: killed, not completed
return self.exit_code if self.exit_code is not None else 0
+146
View File
@@ -0,0 +1,146 @@
"""Logic for the `jobs` domain. Transport-agnostic (D-263).
Nothing here prints, exits, or imports typer.
"""
from __future__ import annotations
import calendar
import json
import time
from pathlib import Path
from typing import Any
from tooling.core import process
from tooling.core.errors import ReachError
from tooling.domains.jobs.schemas import Job, Status
# How often a follow/wait loop re-checks. Chosen for a caller that is a program
# rather than an eye: fast enough that `wait` does not add noticeable latency to
# a short job, slow enough not to spin a core on a long one.
POLL_SECONDS = 0.25
def list_jobs(limit: int = 20) -> list[Job]:
"""Recent jobs, newest first, each reconciled against process liveness."""
directory = process.jobs_dir()
files = sorted(directory.glob("*.json"), reverse=True)
jobs = [_load(path.stem) for path in files[:limit]]
return [job for job in jobs if job is not None]
def get(job_id: str) -> Job:
"""One job by id, reconciled. Raises if it does not exist."""
job = _load(job_id)
if job is None:
known = [path.stem for path in sorted(process.jobs_dir().glob("*.json"), reverse=True)]
recent = ", ".join(known[:5]) if known else "(no jobs recorded yet)"
raise ReachError(
f"no such job: {job_id}",
fix=f"reach jobs list — most recent are: {recent}",
exit_code=2,
)
return job
def read_events(job_id: str, offset: int = 0) -> tuple[list[dict[str, Any]], int]:
"""Events from `offset`, plus the new offset.
A byte offset into an append-only file is the entire reason no daemon is
needed: a caller can read, drop off, and come back with the offset it kept,
and nothing has to have been holding a subscription open on its behalf.
"""
return read_events_from(process.log_path(job_id), offset)
def read_events_from(path: Path, offset: int = 0) -> tuple[list[dict[str, Any]], int]:
"""The same, given a path — so a test can drive it without a real job."""
if not path.is_file():
return [], offset
with path.open("rb") as handle:
handle.seek(offset)
raw = handle.read()
new_offset = handle.tell()
events: list[dict[str, Any]] = []
consumed = offset
for line in raw.split(b"\n"):
if not line.strip():
consumed += len(line) + 1
continue
try:
events.append(json.loads(line))
except json.JSONDecodeError:
# A partial trailing line: the writer is mid-append. Leave the
# offset before it so the next read picks it up whole rather than
# discarding an event because we looked a millisecond too early.
return events, consumed
consumed += len(line) + 1
return events, new_offset
def wait(job_id: str, timeout: float | None = None) -> Job:
"""Block until the job finishes; return it. Never returns while running."""
deadline = None if timeout is None else time.monotonic() + timeout
while True:
job = get(job_id)
if job.status.finished:
return job
if deadline is not None and time.monotonic() >= deadline:
raise ReachError(
f"timed out after {timeout:g}s waiting for job {job_id}",
fix=f"reach jobs status {job_id} — the job is still running, not lost",
exit_code=2,
)
time.sleep(POLL_SECONDS)
def duration(job: Job) -> str:
"""Human-readable elapsed time, or how long it has been running so far."""
start = _parse(job.started_at)
end = _parse(job.ended_at) if job.ended_at else time.time()
if start is None or end is None:
return "?"
seconds = max(0.0, end - start)
if seconds < 60:
return f"{seconds:.1f}s"
minutes, rest = divmod(int(seconds), 60)
return f"{minutes}m{rest:02d}s"
def _load(job_id: str) -> Job | None:
meta = process.read_meta(job_id)
if meta is None:
return None
return _reconcile(Job.model_validate(meta))
def _reconcile(job: Job) -> Job:
"""Correct a recorded status against reality.
A job whose file says running but whose pid is gone did not keep running —
it died without being able to record anything. Reporting it as running
would be the exit-0 trap somewhere nobody is watching, and would hang any
caller polling for it to finish.
"""
if job.status is Status.RUNNING and not process.is_alive(job.pid):
return job.model_copy(update={"status": Status.DIED})
return job
def _parse(stamp: str | None) -> float | None:
"""Parse a recorded timestamp as UTC.
`calendar.timegm`, NOT `time.mktime`: the stamps are written in UTC, and
mktime would read them as local time. That silently adds the UTC offset to
every duration — a job started seconds ago reported as having run for over
two hours. The same mismatch bit the job id in T-1276; both directions of
this conversion need saying out loud.
"""
if not stamp:
return None
try:
return calendar.timegm(time.strptime(stamp, "%Y-%m-%dT%H:%M:%S"))
except (ValueError, TypeError):
return None
+4
View File
@@ -51,6 +51,10 @@ DOMAINS: dict[str, tuple[str, str]] = {
"tooling.domains.check.router:app", "tooling.domains.check.router:app",
"Consistency gates — the checks the push hook runs", "Consistency gates — the checks the push hook runs",
), ),
"jobs": (
"tooling.domains.jobs.router:app",
"Detached runs — status, logs and outcomes",
),
} }
+130
View File
@@ -0,0 +1,130 @@
#!/usr/bin/env python3
"""Units for the jobs service (T-1278), called DIRECTLY — no CLI round trip.
That is half the point of these tests. D-263 says a service must be
transport-agnostic so it can be called by a test, by another service, or by a
future second front end. If nothing ever exercises that, the layering is
unverified decoration — a claim in a decision record with no evidence behind it.
Every test here imports `service` and calls a function.
The other half is the two behaviours that are easy to get wrong and impossible
to notice when they are:
1. A partial trailing line. The log is appended to by a live process, so a
reader can arrive mid-write. Parsing greedily would either crash or, worse,
silently discard the event and advance past it — losing exactly one line,
the one being written when someone looked.
2. Reconciliation. A job killed outright cannot record its ending, so its file
says `running` forever. Trusting the file leaves a corpse looking busy and
hangs anything waiting on it.
Run: python3 tooling/test_jobs.py
"""
import json
import sys
import tempfile
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from tooling.domains.jobs import service # noqa: E402
from tooling.domains.jobs.schemas import Job, Status # noqa: E402
def _job(**overrides) -> Job:
base = {
"job": "20260101T000000-test",
"command": "reach check client-version",
"argv": ["check", "client-version"],
"pid": 1,
"started_at": "2026-01-01T00:00:00",
"status": Status.RUNNING,
}
return Job(**{**base, **overrides})
def test_partial_trailing_line(failures: list[str]) -> None:
"""A half-written final line is left for the next read, not dropped."""
with tempfile.TemporaryDirectory() as tmp:
log = Path(tmp) / "j.jsonl"
complete = json.dumps({"message": "one"}) + "\n"
partial = '{"message": "tw'
log.write_text(complete + partial, encoding="utf-8")
events, offset = service.read_events_from(log, 0)
if len(events) != 1:
failures.append(f"partial line: expected 1 complete event, got {len(events)}")
if offset != len(complete):
failures.append(
f"partial line: offset {offset} should stop at {len(complete)}, "
"before the incomplete line — otherwise that event is lost forever"
)
# Now the writer finishes the line. The event must appear.
log.write_text(complete + json.dumps({"message": "two"}) + "\n", encoding="utf-8")
events, _ = service.read_events_from(log, offset)
if [e.get("message") for e in events] != ["two"]:
failures.append(f"partial line: resumed read lost the completed event: {events}")
def test_offset_resume_is_stable(failures: list[str]) -> None:
"""Reading from the end returns nothing and does not move the offset."""
with tempfile.TemporaryDirectory() as tmp:
log = Path(tmp) / "j.jsonl"
log.write_text(json.dumps({"message": "one"}) + "\n", encoding="utf-8")
_, first = service.read_events_from(log, 0)
events, second = service.read_events_from(log, first)
if events or first != second:
failures.append(
f"offset resume: re-reading returned {len(events)} events and moved "
f"{first}->{second}; a follow loop would replay forever"
)
def test_dead_pid_is_reconciled(failures: list[str]) -> None:
"""running + a pid that is gone == died, never running."""
# PID 1 exists; a very high pid almost certainly does not.
alive = service._reconcile(_job(pid=1))
if alive.status is not Status.RUNNING:
failures.append("reconcile: a live pid was reported as not running")
dead = service._reconcile(_job(pid=4_000_000))
if dead.status is not Status.DIED:
failures.append(
f"reconcile: a dead pid stayed {dead.status.value} — a corpse that looks "
"busy hangs every caller waiting on it"
)
def test_died_never_relays_success(failures: list[str]) -> None:
"""A job that died has no exit code of its own, and must not borrow 0."""
died = _job(status=Status.DIED)
if died.effective_exit_code == 0:
failures.append(
"died job relayed exit 0 — that is the exit-0 trap: a killed job "
"reported as success to whatever gated on it"
)
finished = _job(status=Status.FAILED, exit_code=2)
if finished.effective_exit_code != 2:
failures.append("failed job did not relay its own exit code")
def main() -> int:
failures: list[str] = []
test_partial_trailing_line(failures)
test_offset_resume_is_stable(failures)
test_dead_pid_is_reconciled(failures)
test_died_never_relays_success(failures)
if failures:
print("test_jobs: FAIL", file=sys.stderr)
for failure in failures:
print(f" - {failure}", file=sys.stderr)
return 1
print("test_jobs: OK — offsets resume cleanly, dead pids reconcile, died never relays 0")
return 0
if __name__ == "__main__":
sys.exit(main())