feat(log): crash-survivable FileLogSink + dev/prod verbosity toggle (T-432)

First increment of the observability epic (T-425), the productive pivot after
the ConPTY freeze refused to reproduce on CI: if we can't reproduce it, make
the next occurrence leave evidence.

- FileLogSink (lib/kernel/src/file_log_sink.dart): synchronous, crash-survivable
  LogSink. Appends each record as one JSON line to a size-rotated file; fsyncs
  warn/error + risky-source (pty/ffi/conpty/watchdog) records immediately so the
  last breadcrumb is on disk before a hard death, batches the rest on a timer.
  Never throws. Flutter-free → unit-tested under dart test against a temp dir.
- logDirectory() (paths.dart): persistent per-platform log dir (LOCALAPPDATA /
  ~/Library/Logs / $XDG_STATE_HOME) — durable across reboot, unlike the
  ephemeral socketDirectory.
- resolveLogLevel() (log.dart): the requested dev/prod toggle. CLIDE_LOG
  dart-define → CLIDE_LOG env → app.log.level setting → warn(release)/info(debug).
  Lenient parse; an invalid source falls through.
- Boot wiring (facade.boot + main.dart): FileLogSink leads the sink chain (so a
  crash records before the volatile stderr/ring sinks) and the resolved level
  sets Logger.minLevel.

Tests: FileLogSink (JSON shape, error/stack, rotation cap, append-across-restart,
timer-cancel), resolveLogLevel precedence + fall-through, logDirectory per-OS.
Coverage gate 95.11%.

Follow-ups under T-425: live toggle CLI/command/chip (T-433), FFI breadcrumbs
(T-434), watchdog isolate (T-435), CI artifact wiring (T-436).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-15 09:29:41 +02:00
co-authored by Claude Opus 4.8
parent b3acb8a34c
commit 85cc34e09c
12 changed files with 494 additions and 2 deletions
+37
View File
@@ -5034,3 +5034,40 @@ All the raw syscalls and anything whose output is a syscall return or that has n
windows.yml runs the real ConPTY suite (start/write/resize/kill/errors) on windows-latest but collects NO coverage (no --coverage flag). So the FFI spawn path has functional validation on Windows + the VM soak (tools/windows-verify/) but no line-coverage metric anywhere. Decide whether to (a) accept functional-only validation explicitly, or (b) collect coverage on the Windows runner and merge it so the FFI path is measured. Cross-platform lcov merge is non-trivial (the gate reads one file) — may warrant a Q-record.
Audit detail: full per-fragment findings + adversarial verdicts in the workflow result for run wf_a3cacb2c-2c7.', NULL, '2026-06-14 22:38:01', '2026-06-14 22:38:01', '2026-06-14 22:38:01', NULL, 'd8cafbc05c3a83c1041c29ba25e4ed83', 2) ON CONFLICT(hash) DO NOTHING;
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06FCENXW1VF7VXQX64982X171R', 'description', 'Rank-1 culprit from the Windows test-freeze analysis (2026-06-14). On Windows, each WindowsPty.start() (lib/src/pty/windows_pty.dart) pairs the child with its own conhost.exe/OpenConsole.exe. TerminateProcess + ClosePseudoConsole do NOT reliably reap that conhost — it lingers until the PARENT process exits (microsoft/terminal#4050) — and the child is not placed in a Windows Job Object, so dart:io children survive parent death (dart-lang/sdk#49234). Across repeated test runs the orphaned conhost/cmd processes accumulate at the session level and are a credible bridge to the whole-OS resource starvation that ends in a hard power-cycle (cf. anthropics/claude-code#63043: ~1088 leaked conhost.exe).
Fix: create a Windows Job Object per WindowsPty (CreateJobObject) with JOBOBJECT_EXTENDED_LIMIT_INFORMATION.BasicLimitInformation.LimitFlags = JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE; AssignProcessToJobObject(job, hProcess) immediately after CreateProcessW; close the job handle on teardown so the child AND its conhost die when the session (or the test process) exits. Add a tearDownAll backstop in test/pty/windows_pty_test.dart that sweeps stray cmd.exe/conhost.exe children of the test pid.
Verify with tools/windows-verify/soak-conpty.ps1 the orphaned ConPTY-host count must stop climbing across iterations.
Sibling ConPTY-teardown fixes surfaced by the same analysis (fold in here or file separately): CancelIoEx/overlapped reader instead of Isolate.kill (the reader isolate blocks forever in ReadFile and Isolate.kill cannot interrupt FFI dart-lang/sdk#46680); clamp cols/rows >= 2 before CreatePseudoConsole/ResizePseudoConsole (microsoft/terminal#19922 narrow-terminal CRLF spin); fix close()/_closeConsole() teardown order (terminate -> drain to EOF -> ClosePseudoConsole -> release); add --timeout to the dart-test pty line in ci/test.sh.
Progress (commit 606d3df, pre-VM hardening): two sibling quick-wins landed on the branch cols/rows clamped to >= 2 in both PTY backends (lib/src/pty/pty_size.dart; microsoft/terminal#19922) and --timeout 60s on the dart-test pty line in ci/test.sh. Also made windows_pty.dart''s pure helpers (quoteArg / composeEnvironmentBlock / resolveExecutable) public + unit-tested off-Windows.
Still open and VM-gated (new/changed FFI, can''t validate off-Windows): the Job Object reaping (this ticket''s core), CancelIoEx/overlapped reader, and the close()/_closeConsole() teardown reorder. Do these in the Windows VM session and validate each with tools/windows-verify/soak-conpty.ps1 (orphan host count must go flat).', 'Rank-1 culprit from the Windows test-freeze analysis (2026-06-14). On Windows, each WindowsPty.start() (lib/src/pty/windows_pty.dart) pairs the child with its own conhost.exe/OpenConsole.exe. TerminateProcess + ClosePseudoConsole do NOT reliably reap that conhost it lingers until the PARENT process exits (microsoft/terminal#4050) and the child is not placed in a Windows Job Object, so dart:io children survive parent death (dart-lang/sdk#49234). Across repeated test runs the orphaned conhost/cmd processes accumulate at the session level and are a credible bridge to the whole-OS resource starvation that ends in a hard power-cycle (cf. anthropics/claude-code#63043: ~1088 leaked conhost.exe).
Fix: create a Windows Job Object per WindowsPty (CreateJobObject) with JOBOBJECT_EXTENDED_LIMIT_INFORMATION.BasicLimitInformation.LimitFlags = JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE; AssignProcessToJobObject(job, hProcess) immediately after CreateProcessW; close the job handle on teardown so the child AND its conhost die when the session (or the test process) exits. Add a tearDownAll backstop in test/pty/windows_pty_test.dart that sweeps stray cmd.exe/conhost.exe children of the test pid.
Verify with tools/windows-verify/soak-conpty.ps1 the orphaned ConPTY-host count must stop climbing across iterations.
Sibling ConPTY-teardown fixes surfaced by the same analysis (fold in here or file separately): CancelIoEx/overlapped reader instead of Isolate.kill (the reader isolate blocks forever in ReadFile and Isolate.kill cannot interrupt FFI dart-lang/sdk#46680); clamp cols/rows >= 2 before CreatePseudoConsole/ResizePseudoConsole (microsoft/terminal#19922 narrow-terminal CRLF spin); fix close()/_closeConsole() teardown order (terminate -> drain to EOF -> ClosePseudoConsole -> release); add --timeout to the dart-test pty line in ci/test.sh.
Progress (commit 606d3df, pre-VM hardening): two sibling quick-wins landed on the branch cols/rows clamped to >= 2 in both PTY backends (lib/src/pty/pty_size.dart; microsoft/terminal#19922) and --timeout 60s on the dart-test pty line in ci/test.sh. Also made windows_pty.dart''s pure helpers (quoteArg / composeEnvironmentBlock / resolveExecutable) public + unit-tested off-Windows.
Still open and VM-gated (new/changed FFI, can''t validate off-Windows): the Job Object reaping (this ticket''s core), CancelIoEx/overlapped reader, and the close()/_closeConsole() teardown reorder. Do these in the Windows VM session and validate each with tools/windows-verify/soak-conpty.ps1 (orphan host count must go flat).
## Soak results on GitHub windows-latest (Server 2022) orphan-accumulation NOT reproduced (2026-06-14)
Ran both halves of the windows-verify soak on GitHub-hosted Windows (no VM needed windows-latest runs the ConPTY suite green, so the soak just wraps it):
1. Clean-path soak (soak-conpty.ps1, 25 iters): orphans stayed at 0, dart handles flat ~152, threads flat at 7. Orderly close() reaps everything. NOT REPRODUCED.
2. Abrupt-death probe (soak-conpty-kill.ps1 + conpty_orphan_probe.dart, 15 iters x 2 PTYs): start real WindowsPty sessions on long-lived children, block WITHOUT close(), then taskkill /F the parent dart.exe (no /T). Every cycle reaped to baseline survivors=0, cum=0. When the parent dies the OS breaks the pipes and conhost exits on its own. NOT REPRODUCED.
**Implication:** the conhost-orphan-accumulation mechanism this ticket is premised on does NOT hold on Server 2022, under clean OR abrupt teardown. The Job Object fix may still be worthwhile as defense-in-depth, but its justification (a reproduced leak) is not confirmed.
**Caveats / what''s still untested:**
- OS mismatch: the real crashes were on desktop Win10/11; this is headless Server 2022. terminal#4050 was a desktop report. A desktop-specific behavior may be unreproducible on CI.
- Both probes let the process DIE, so within-process accumulation (culprit #2: reader isolates blocked forever in ReadFile, threads/handles climbing within one long-lived process) is reclaimed at exit and never measured. A long-lived-process probe (one dart.exe spawning + abandoning PTYs, watching its OWN handle/thread count climb) would test that the more likely freeze mode for a long-running app. Not yet built.
Diagnostics live in tools/windows-verify/ and run via .github/workflows/windows-soak.yml (workflow_dispatch). The same kill-probe will validate the fix if/when it lands (survivors should stay 0 though they already do, which is the problem).', NULL, '2026-06-15 07:11:41', '2026-06-15 07:11:41', '2026-06-15 07:11:41', NULL, 'dadb5e43a8a9b44b2c4be4b7d4528beb', 2) ON CONFLICT(hash) DO NOTHING;
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06FCM9ER04JVFW8CN3JW1AWYA8', 'status', 'backlog', 'in_progress', NULL, '2026-06-15 07:19:13', '2026-06-15 07:19:13', '2026-06-15 07:19:13', NULL, '55a53ab84e2a10fd52fe6853bd32dc55', 2) ON CONFLICT(hash) DO NOTHING;
+32
View File
@@ -5635,3 +5635,35 @@ All the raw syscalls and anything whose output is a syscall return or that has n
windows.yml runs the real ConPTY suite (start/write/resize/kill/errors) on windows-latest but collects NO coverage (no --coverage flag). So the FFI spawn path has functional validation on Windows + the VM soak (tools/windows-verify/) but no line-coverage metric anywhere. Decide whether to (a) accept functional-only validation explicitly, or (b) collect coverage on the Windows runner and merge it so the FFI path is measured. Cross-platform lcov merge is non-trivial (the gate reads one file) — may warrant a Q-record.
Audit detail: full per-fragment findings + adversarial verdicts in the workflow result for run wf_a3cacb2c-2c7.', 'backlog', 'medium', NULL, NULL, NULL, '2026-06-14 22:37:27', '2026-06-14 22:38:01', NULL, '43890d3049d818cea0acd681a191bc94', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at > tickets.updated_at OR (excluded.updated_at = tickets.updated_at AND excluded.hash > tickets.hash);
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06FCENXW1VF7VXQX64982X171R', 'bug', NULL, 'ConPTY children leak: place each WindowsPty child in a kill-on-close Job Object', 'Rank-1 culprit from the Windows test-freeze analysis (2026-06-14). On Windows, each WindowsPty.start() (lib/src/pty/windows_pty.dart) pairs the child with its own conhost.exe/OpenConsole.exe. TerminateProcess + ClosePseudoConsole do NOT reliably reap that conhost — it lingers until the PARENT process exits (microsoft/terminal#4050) — and the child is not placed in a Windows Job Object, so dart:io children survive parent death (dart-lang/sdk#49234). Across repeated test runs the orphaned conhost/cmd processes accumulate at the session level and are a credible bridge to the whole-OS resource starvation that ends in a hard power-cycle (cf. anthropics/claude-code#63043: ~1088 leaked conhost.exe).
Fix: create a Windows Job Object per WindowsPty (CreateJobObject) with JOBOBJECT_EXTENDED_LIMIT_INFORMATION.BasicLimitInformation.LimitFlags = JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE; AssignProcessToJobObject(job, hProcess) immediately after CreateProcessW; close the job handle on teardown so the child AND its conhost die when the session (or the test process) exits. Add a tearDownAll backstop in test/pty/windows_pty_test.dart that sweeps stray cmd.exe/conhost.exe children of the test pid.
Verify with tools/windows-verify/soak-conpty.ps1 the orphaned ConPTY-host count must stop climbing across iterations.
Sibling ConPTY-teardown fixes surfaced by the same analysis (fold in here or file separately): CancelIoEx/overlapped reader instead of Isolate.kill (the reader isolate blocks forever in ReadFile and Isolate.kill cannot interrupt FFI dart-lang/sdk#46680); clamp cols/rows >= 2 before CreatePseudoConsole/ResizePseudoConsole (microsoft/terminal#19922 narrow-terminal CRLF spin); fix close()/_closeConsole() teardown order (terminate -> drain to EOF -> ClosePseudoConsole -> release); add --timeout to the dart-test pty line in ci/test.sh.
Progress (commit 606d3df, pre-VM hardening): two sibling quick-wins landed on the branch cols/rows clamped to >= 2 in both PTY backends (lib/src/pty/pty_size.dart; microsoft/terminal#19922) and --timeout 60s on the dart-test pty line in ci/test.sh. Also made windows_pty.dart''s pure helpers (quoteArg / composeEnvironmentBlock / resolveExecutable) public + unit-tested off-Windows.
Still open and VM-gated (new/changed FFI, can''t validate off-Windows): the Job Object reaping (this ticket''s core), CancelIoEx/overlapped reader, and the close()/_closeConsole() teardown reorder. Do these in the Windows VM session and validate each with tools/windows-verify/soak-conpty.ps1 (orphan host count must go flat).
## Soak results on GitHub windows-latest (Server 2022) orphan-accumulation NOT reproduced (2026-06-14)
Ran both halves of the windows-verify soak on GitHub-hosted Windows (no VM needed windows-latest runs the ConPTY suite green, so the soak just wraps it):
1. Clean-path soak (soak-conpty.ps1, 25 iters): orphans stayed at 0, dart handles flat ~152, threads flat at 7. Orderly close() reaps everything. NOT REPRODUCED.
2. Abrupt-death probe (soak-conpty-kill.ps1 + conpty_orphan_probe.dart, 15 iters x 2 PTYs): start real WindowsPty sessions on long-lived children, block WITHOUT close(), then taskkill /F the parent dart.exe (no /T). Every cycle reaped to baseline survivors=0, cum=0. When the parent dies the OS breaks the pipes and conhost exits on its own. NOT REPRODUCED.
**Implication:** the conhost-orphan-accumulation mechanism this ticket is premised on does NOT hold on Server 2022, under clean OR abrupt teardown. The Job Object fix may still be worthwhile as defense-in-depth, but its justification (a reproduced leak) is not confirmed.
**Caveats / what''s still untested:**
- OS mismatch: the real crashes were on desktop Win10/11; this is headless Server 2022. terminal#4050 was a desktop report. A desktop-specific behavior may be unreproducible on CI.
- Both probes let the process DIE, so within-process accumulation (culprit #2: reader isolates blocked forever in ReadFile, threads/handles climbing within one long-lived process) is reclaimed at exit and never measured. A long-lived-process probe (one dart.exe spawning + abandoning PTYs, watching its OWN handle/thread count climb) would test that the more likely freeze mode for a long-running app. Not yet built.
Diagnostics live in tools/windows-verify/ and run via .github/workflows/windows-soak.yml (workflow_dispatch). The same kill-probe will validate the fix if/when it lands (survivors should stay 0 though they already do, which is the problem).', 'backlog', 'high', NULL, NULL, NULL, '2026-06-14 18:14:36', '2026-06-15 07:11:41', NULL, '7103394bfdac9685026b652023a7569e', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at > tickets.updated_at OR (excluded.updated_at = tickets.updated_at AND excluded.hash > tickets.hash);
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06FCM9ER04JVFW8CN3JW1AWYA8', 'task', '06FCENXXZFBZ0HVD1VCW4ZASCC', 'FileLogSink + logDirectory + boot-time verbosity resolver (CLIDE_LOG → env → setting → release/debug default)', NULL, 'backlog', 'high', NULL, NULL, NULL, '2026-06-15 07:18:58', '2026-06-15 07:18:58', NULL, '7f08c492c5eb3f6642719f0b6990f7cd', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at > tickets.updated_at OR (excluded.updated_at = tickets.updated_at AND excluded.hash > tickets.hash);
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06FCM9F446MZFXVHH65Q6CKTPM', 'task', '06FCENXXZFBZ0HVD1VCW4ZASCC', 'Live verbosity toggle: clide log level CLI + /loglevel command + sync output-dock Level chip to kernel Logger + persist app.log.level', NULL, 'backlog', 'medium', NULL, NULL, NULL, '2026-06-15 07:19:01', '2026-06-15 07:19:01', NULL, '7801e4323815cf95bdc78fa005611cf0', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at > tickets.updated_at OR (excluded.updated_at = tickets.updated_at AND excluded.hash > tickets.hash);
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06FCM9FHC8VX50759X35VNER1R', 'task', '06FCENXXZFBZ0HVD1VCW4ZASCC', 'FFI breadcrumbs in windows_pty.dart (+native_pty): injectable log callback, before/after each risky syscall with return + GetLastError; reader/waiter isolates flushSync their own append handle', NULL, 'backlog', 'high', NULL, NULL, NULL, '2026-06-15 07:19:04', '2026-06-15 07:19:04', NULL, 'd796d50acb2751cc8732a20d3805a77f', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at > tickets.updated_at OR (excluded.updated_at = tickets.updated_at AND excluded.hash > tickets.hash);
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06FCM9FYDEXCM15FXTER032K84', 'task', '06FCENXXZFBZ0HVD1VCW4ZASCC', 'Watchdog heartbeat + resource sampler in a dedicated isolate (heartbeat ~500ms; sample ConPTY child / handle / thread / memory ~2s)', NULL, 'backlog', 'medium', NULL, NULL, NULL, '2026-06-15 07:19:08', '2026-06-15 07:19:08', NULL, 'a51aac4a65ec7589c8181655aeeb73e9', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at > tickets.updated_at OR (excluded.updated_at = tickets.updated_at AND excluded.hash > tickets.hash);
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06FCM9GAQ2G0KCVMZS67SK3324', 'task', '06FCENXXZFBZ0HVD1VCW4ZASCC', 'Wire FileLogSink into test harness + ci/test.sh (CLIDE_LOG=debug, log dir outside build tree, upload as CI artifact in always() step)', NULL, 'backlog', 'medium', NULL, NULL, NULL, '2026-06-15 07:19:11', '2026-06-15 07:19:11', NULL, '1c35b3eea276c8d7d442e8b09f2c7ad8', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at > tickets.updated_at OR (excluded.updated_at = tickets.updated_at AND excluded.hash > tickets.hash);
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06FCM9ER04JVFW8CN3JW1AWYA8', 'task', '06FCENXXZFBZ0HVD1VCW4ZASCC', 'FileLogSink + logDirectory + boot-time verbosity resolver (CLIDE_LOG → env → setting → release/debug default)', NULL, 'in_progress', 'high', NULL, NULL, NULL, '2026-06-15 07:18:58', '2026-06-15 07:19:13', NULL, 'f0ac1b130b28ff2e01db483a5fa47ab7', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at > tickets.updated_at OR (excluded.updated_at = tickets.updated_at AND excluded.hash > tickets.hash);