mirror of
https://github.com/pewdiepie-archdaemon/odysseus.git
synced 2026-11-04 04:02:21 +01:00
fix(runtime): verify process identity before any teardown signal
A recorded pid is a claim, not a handle. The containment grant store, the background-job store and the Cookbook task list all outlive the process that wrote them — deliberately, so a restart keeps a job and its result — and the kernel reuses pids. Any teardown driven off one of those records can therefore land on a process we never started. ODY-86 was exactly this, and the Cookbook survivor sweep still terminated any process whose full command line matched a tracked one, which is the same mistake spelled differently. Identity is (pid, start token). The token comes from /proc/<pid>/stat on Linux, ps -o lstart= on macOS and the BSDs, and GetProcessTimes on Windows; the kernel will not hand a pid to a process that started earlier, so comparing the token recorded at launch against the token read now answers "is this still ours" without a handle or a supervisor. verify() returns owned, gone, foreign or unverifiable, and only owned permits a signal. Keeping "unverifiable" out of the other two is the point. Process inspection has broken off Linux four times here — ODY-70, -86, -94, -99 — every time because an absent mechanism read as a successful answer. Folding it into "ours" signals strangers; folding it into "gone" abandons live processes. It is a containment failure and every caller treats it as one. Wired into the three places that signal: - containment.release() gates a grant recovered from the durable store, and leaves an in-process teardown alone, where the caller holds the child and no identity question arises. The verdict lands on the record, so "why is this grant still here" is answerable afterwards. - A startup reaper. Nothing read either store before, so a crashed run left every grant permanently active and every job permanently running, and the first thing to touch such a record was a teardown aimed at a reassigned pid. The two stores get opposite treatment: an orphaned grant has no caller left and is torn down, while a detached job is documented to survive a restart and is only corrected, never killed. - The Cookbook sweep takes its ownership from the tmux pane's process tree, captured before the kill destroys the only link between a surviving model server and the session that started it. A process that merely matches the tracked command line is now reported rather than killed: the Cookbook composed that command line, so an identical one is just as likely to be a server the user started by hand. The sweep also runs on hosts with no procfs instead of silently skipping, and says so when it could not look at all.
This commit is contained in:
+62
-1
@@ -38,6 +38,7 @@ from core.platform_compat import (
|
||||
pid_alive,
|
||||
)
|
||||
|
||||
from src import process_ownership
|
||||
from src.constants import BG_JOBS_DIR, BG_JOBS_FILE
|
||||
|
||||
_JOBS_DIR = Path(BG_JOBS_DIR)
|
||||
@@ -152,6 +153,11 @@ def launch(command: str, session_id: str, cwd: Optional[str] = None,
|
||||
"followed_up": False, # has the agent been re-invoked with the result?
|
||||
"log_path": str(log_path),
|
||||
"exit_path": str(exit_path),
|
||||
# Identity, not just a slot. The pid above is reused by the kernel, and
|
||||
# this record outlives the process and the server; the token is what a
|
||||
# later run compares before it signals anything. See
|
||||
# src/process_ownership.py.
|
||||
"start_token": process_ownership.capture(proc.pid)["start_token"],
|
||||
}
|
||||
jobs = _load()
|
||||
jobs[job_id] = rec
|
||||
@@ -283,10 +289,65 @@ def kill(job_id: str) -> Optional[Dict[str, Any]]:
|
||||
return rec
|
||||
|
||||
|
||||
def disown_unverified() -> Dict[str, Any]:
|
||||
"""Stop tracking running jobs whose process can no longer be proven ours.
|
||||
|
||||
Called once at startup by :mod:`src.process_reaper`, never from the poll
|
||||
loop — every record it sees was written by an earlier run, which is what
|
||||
makes "unidentifiable" a statement about a previous run's child rather than
|
||||
about a job this run just launched.
|
||||
|
||||
Signals nothing. A detached job is meant to survive a restart, so a job that
|
||||
verifies as ours is left alone and its result is still collected. What is
|
||||
corrected is the record that would otherwise be signalled later on a pid the
|
||||
kernel has reassigned: the max-runtime branch of :func:`refresh` sends
|
||||
SIGTERM then SIGKILL to ``rec["pid"]`` an hour in, and on a reused pid that
|
||||
lands on a bystander.
|
||||
|
||||
Fail closed: a job that cannot be verified is retired too, not kept.
|
||||
Retiring loses a result, which is visible; keeping it leaves a pid this
|
||||
server will eventually signal without knowing what it is pointing at, which
|
||||
is not.
|
||||
"""
|
||||
jobs = _load()
|
||||
report = {"seen": 0, "retired": 0, "kept": 0}
|
||||
changed = False
|
||||
now = time.time()
|
||||
for rec in jobs.values():
|
||||
if rec.get("status") != "running":
|
||||
continue
|
||||
report["seen"] += 1
|
||||
verdict = process_ownership.verify(rec.get("pid"), rec.get("start_token"))
|
||||
if verdict in (process_ownership.OWNED, process_ownership.GONE):
|
||||
# OWNED: still ours, still running, still watched. GONE: refresh()
|
||||
# already turns an absent process into a "died" record, and it may
|
||||
# yet find an exit-code file the job wrote before it went.
|
||||
report["kept"] += 1
|
||||
continue
|
||||
rec["status"] = "failed"
|
||||
rec["exit_code"] = -1
|
||||
rec["ended_at"] = now
|
||||
rec["ownership_lost"] = verdict
|
||||
# followed_up stays False: the agent asked for this job and is owed an
|
||||
# answer, even when the answer is that we lost track of it.
|
||||
report["retired"] += 1
|
||||
changed = True
|
||||
if changed:
|
||||
_save(jobs)
|
||||
return report
|
||||
|
||||
|
||||
def result_text(rec: Dict[str, Any]) -> str:
|
||||
"""Human/agent-readable summary of a finished job, for the follow-up."""
|
||||
out = _read_output(rec)
|
||||
if rec.get("killed"):
|
||||
if rec.get("ownership_lost"):
|
||||
head = (
|
||||
"Background job was abandoned across a server restart: its process "
|
||||
f"could not be identified as ours ({rec.get('ownership_lost')}), so it was "
|
||||
"neither waited on nor signalled. Any output below is what it had "
|
||||
"written by then; if the work matters, re-run it."
|
||||
)
|
||||
elif rec.get("killed"):
|
||||
head = "Background job was killed."
|
||||
elif rec.get("timed_out"):
|
||||
head = f"Background job timed out after {rec.get('max_runtime_s')}s."
|
||||
|
||||
Reference in New Issue
Block a user