292c7de2bf3aa1c8a3194e856d09d1889d7d54e2
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jpmschweitzerandClaude
|
4e68767771 |
refactor(health-report): put summary where the other writer puts it
check_history has two producers. sysmon-go writes `summary` at the top level beside `status`; this module wrote it under `metrics`. So a reader had to know which producer wrote a row before it could find out what the row said, and a query written the obvious way found one and silently missed the other. That is the T-36 failure repeating. There, per-domain queries returned rows from August and looked like a system that had stopped reporting, because the data was nested under a composite row nobody had mentioned. Nothing was missing; the query was asking the wrong shape. verify.sh had already grown a coalesce over both spellings, which is the tell: a compatibility shim that hides a schema disagreement rather than resolving it. D-33 made this table a contract between producers, and a contract needs one spelling. Summary is now a required parameter with no default. sysmon-go enforces the same thing through Domain.Run's signature, and the reason is identical: a row whose substance is missing looks exactly like a row whose check found nothing to say. Both call sites pass it; the failure path passes the exception rather than leaving the field to the metrics blob. Old rows keep the nested spelling and verify.sh keeps reading both, because rewriting history to match a new convention is a worse trade than a fallback with a reason attached. Also drops "Three consequences" from the module docstring, which by then listed five. A hardcoded count beside the thing it counts is the same defect as install.sh printing "wrote 8 keys" while writing ten — this morning's bug, in prose instead of code. Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
jpmschweitzer
|
c34db66f51 |
fix(api): refuse to delete a task's history by accident, with 409 and ?purge
DELETE /tasks/{name} issued a bare DELETE against scheduled_tasks. Any task
that had ever run owns rows in task_executions, so the foreign key rejected it
and the caller got:
psycopg2.errors.ForeignKeyViolation: update or delete on table
"scheduled_tasks" violates foreign key constraint
"task_executions_task_id_fkey" on table "task_executions"
surfaced as a bare 500 with nothing naming history as the obstacle. It read as
the service being broken rather than the request being refusable, and since
every task that has ever fired has history, the endpoint effectively worked
only for tasks that had never run. Found while removing a temporary probe task,
which then had to be deleted with hand-written SQL across two tables.
Refusing rather than cascading, because the outcomes are not equally
recoverable: a task definition can be recreated from the API in one call, its
execution history cannot be recreated at all. Defaulting to the destructive
reading of an ambiguous request is how audit trails disappear quietly.
The 409 carries what the caller needs to act -- how many records are at stake,
the flag that proceeds anyway, and PUT enabled=false, which is usually what was
actually wanted: it stops the task running and keeps the record. A bare
"conflict" would be little better than the 500 it replaces.
Purge deletes history and task in one transaction. Split across two, a failure
between them leaves the audit trail gone and the task alive -- the worst of both.
Mutation-checked: removing the guard fails the refusal tests. A test also pins
that a refused delete issues no DELETE at all, and that ?purge=true on a missing
task is still 404 rather than a success.
|
||
|
jpmschweitzerandClaude
|
538bfe5944 |
feat(health-report): name the task that produced each check_history row
`source` names the code that wrote a row. It cannot name the schedule that
invoked it, and two tasks may share one executor -- so a row could not answer
the question a health record mostly exists to answer: which job broke?
Concretely, on 2026-08-11 a 425 MB probe task and the 5 GB nightly backup both
ran through config_backup_executor. The probe failed and wrote
source: scheduler/config_backup_executor
status: critical
error: Backup file was not created
which is byte-for-byte what a nightly backup failure would have written. The
row was true and unattributable, and the reflex it invited -- delete the
inconvenient row -- was correctly refused. Attribution is the actual fix: the
record stays intact and starts saying who it is about.
Carried in a ContextVar rather than an argument. Executors are invoked as
execute(config, settings) and there are ten of them, several dormant -- existing
only as a string in a database row and becoming live the moment someone inserts
a task naming them. A signature change would leave those broken in a way nothing
imports, greps or tests would reveal. Injecting the name into `config` was the
other option and is worse: `config` is what a human wrote in the task
definition, and an executor is entitled to reject keys it does not recognise.
Two properties make the ContextVar safe, both verified in the deployed runtime
rather than reasoned about:
- asyncio.to_thread propagates the context, so reporting still sees the task
after T-74 moved executor bodies into worker threads. Had it not, every row
from a real executor would have quietly lost its task while unit tests kept
passing -- so there is a test that specifically goes through report_async.
- Each asyncio Task gets its own copy, so the five concurrent executions
MAX_CONCURRENT_TASKS permits cannot read each other's value. The isolation
test yields mid-execution to force interleaving; without that it would pass
even against a shared global.
A plain await does NOT get its own copy and leaks the value to the caller, which
the runtime check showed. Both real entry points go through create_task, but
task_scope resets via token rather than depending on that.
The field is omitted, not nulled, when there is no task: report() is callable
from a script, and a null would claim a task existed with no name.
Mutation-checked: removing the scope from the engine fails both isolation tests.
Co-Authored-By: Claude <noreply@anthropic.com>
|
||
|
jpmschweitzerandClaude
|
e45de4fec7 |
fix(orchestrator): record the terminal states that were never written
Two defects with one shape: an execution reaches a terminal condition and the
orchestrator fails to write it down, so the system's own record disagrees with
what happened. Neither produced an error. Both produced silence.
T-2 -- a restart mid-task unscheduled that task forever.
get_tasks_for_minute excludes any task holding a task_executions row with
status='running'. The row is written before the executor runs and updated
after, so a process dying in between left it 'running' permanently, and the
task was then excluded from every future minute with no error, no alarm and no
log line. It did not fail; it went quiet.
test_example_task had held such a row since 2025-12-07 -- 5916 hours. It is
disabled, so nothing was broken by that instance; the mechanism is the point,
and the exposure is daily, because Watchtower restarts this container at 4 AM
while the config backup starts at 03:05 and runs ~21 minutes.
Startup now reconciles them, where the reasoning is sound by construction: this
process has just begun, so nothing it can see is genuinely running.
Marked 'orphaned', not 'failed'. When the process dies mid-task the work may
well have completed -- a backup that finished and never got to update its row
is indistinguishable from one that died halfway -- and 'failed' would assert an
outcome nobody observed. Same error as the health-report diagnostic fixed in
|
||
|
jpmschweitzerandClaude
|
55fe0f9083 |
chore: adopt the workspace agent-config baseline
Commits a .claude/settings.json rather than leaving permissions to per-developer local state, and initialises a pql vault for this repo's tickets and internal decisions. Every git deny rule appears in both the `git <verb>` and `git * <verb>` forms. Only the second catches `git -C <path>`, and without it the whole deny list is decorative -- it looks like a policy and stops nothing. The allow list carries pql's absolute path alongside the bare name. pql is installed to ~/.local/bin, which is on the login PATH but not the one a non-interactive shell gets, so the bare-name rules match nothing on their own and every call would prompt anyway. .gitignore now covers .claude/settings.local.json, which is machine-local and must never be shared. `pql init` contributed the .pql/* rules with an exception for the changelog, which is the replication log of record and has to be committed for tickets to travel with a clone. Co-Authored-By: Claude <noreply@anthropic.com> |