ci(soak): abrupt-death ConPTY orphan probe (the failure-mode the clean soak misses)
The clean-path soak found no leak on windows-latest — because orderly close() reaps every host. That never exercises the freeze hypothesis (T-424), which is the parent dying WITHOUT teardown while a child is live. This probe does. conpty_orphan_probe.dart starts N real WindowsPty sessions on long-lived children and blocks WITHOUT ever calling close(); soak-conpty-kill.ps1 force-kills only the dart.exe parent (taskkill /F, no /T) once the hosts are up and counts the conhost/OpenConsole/cmd processes that SURVIVE. Absent a kill-on-close Job Object, abrupt parent death should orphan them — a survivor count that climbs across cycles is the leak signature. The same probe will prove the T-424 fix: with the job, survivors should drop to ~0. Wired as a second job in windows-soak.yml (workflow_dispatch + when the kit changes). Diagnostic only, never a gate, always exits 0; publishes the verdict to the job summary and uploads the CSV. Not part of the regular test suite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.8
@@ -19,8 +19,14 @@ on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
iterations:
|
||||
description: How many times to run the ConPTY suite
|
||||
description: How many times to run the ConPTY suite (clean-path soak)
|
||||
default: "25"
|
||||
kill_iterations:
|
||||
description: Spawn+force-kill cycles (abrupt-death orphan probe)
|
||||
default: "15"
|
||||
ptys_per_iter:
|
||||
description: WindowsPty sessions spawned per kill cycle
|
||||
default: "2"
|
||||
push:
|
||||
branches: [windows-support]
|
||||
paths:
|
||||
@@ -57,3 +63,39 @@ jobs:
|
||||
name: conpty-soak
|
||||
path: soak-out
|
||||
if-no-files-found: warn
|
||||
|
||||
conpty-kill-probe:
|
||||
# Abrupt-death half: force-kill the parent dart.exe mid-life (no close(),
|
||||
# no Job Object) and count the ConPTY hosts that survive. This is the path
|
||||
# the freeze hypothesis (T-424) actually implicates — the clean-path soak
|
||||
# above never exercises it. Diagnostic only; always succeeds.
|
||||
runs-on: windows-latest
|
||||
timeout-minutes: 30
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: subosito/flutter-action@v2
|
||||
with:
|
||||
channel: stable
|
||||
- run: flutter --version
|
||||
- run: flutter pub get
|
||||
- name: ConPTY abrupt-death orphan probe
|
||||
shell: pwsh
|
||||
run: |
|
||||
$iters = "${{ github.event.inputs.kill_iterations }}"
|
||||
if (-not $iters) { $iters = "15" }
|
||||
$ptys = "${{ github.event.inputs.ptys_per_iter }}"
|
||||
if (-not $ptys) { $ptys = "2" }
|
||||
tools/windows-verify/soak-conpty-kill.ps1 -Iterations ([int]$iters) -PtysPerIter ([int]$ptys) -OutDir "$env:GITHUB_WORKSPACE/kill-out"
|
||||
- name: Publish verdict to job summary
|
||||
if: always()
|
||||
shell: pwsh
|
||||
run: |
|
||||
$s = Get-ChildItem "$env:GITHUB_WORKSPACE/kill-out/*.summary.txt" -ErrorAction SilentlyContinue | Select-Object -First 1
|
||||
if ($s) { Get-Content $s.FullName | Add-Content $env:GITHUB_STEP_SUMMARY }
|
||||
- name: Upload kill-probe CSV + summary
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: conpty-kill-probe
|
||||
path: kill-out
|
||||
if-no-files-found: warn
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
/// ConPTY orphan probe — the ABRUPT-DEATH half of the windows-verify kit.
|
||||
///
|
||||
/// `soak-conpty.ps1` measures the CLEAN path: dart exits normally, the ConPTY
|
||||
/// teardown (`close()` → ClosePseudoConsole → handle release) runs, and the
|
||||
/// hosts are reaped. On GitHub's windows-latest that path showed NO leak —
|
||||
/// orderly shutdown reclaims everything. But the freeze hypothesis (T-424) is
|
||||
/// not about orderly shutdown; it is about the parent dying WITHOUT teardown
|
||||
/// (a crash, a Ctrl-C, a wedged reader isolate) while the child is still live.
|
||||
///
|
||||
/// This probe exercises exactly that. It starts [count] real [WindowsPty]
|
||||
/// sessions — the production backend, no mocks — each running a long-lived
|
||||
/// child, prints a READY line carrying the dart pid + each child pid, then
|
||||
/// blocks forever and NEVER calls `close()`. Its companion driver
|
||||
/// (`soak-conpty-kill.ps1`) force-kills this dart.exe (`taskkill /F`, NOT
|
||||
/// `/T`, so only the parent dies) once the children are up, then counts the
|
||||
/// conhost / OpenConsole / cmd processes that SURVIVE the parent's death.
|
||||
///
|
||||
/// Because the children are not placed in a kill-on-close Job Object, abrupt
|
||||
/// parent death is expected to orphan them — that is the leak this probe is
|
||||
/// built to expose. The same probe will later PROVE the T-424 fix: once each
|
||||
/// child lives in a JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE job, killing the
|
||||
/// parent should take the job — and every host — down with it, and the
|
||||
/// survivor count should drop to zero.
|
||||
library;
|
||||
|
||||
import 'dart:io';
|
||||
|
||||
import 'package:clide/src/pty/windows_pty.dart';
|
||||
|
||||
Future<void> main(List<String> args) async {
|
||||
if (!Platform.isWindows) {
|
||||
stderr.writeln('conpty_orphan_probe: Windows only (ConPTY).');
|
||||
exit(2);
|
||||
}
|
||||
final count = args.isNotEmpty ? (int.tryParse(args.first) ?? 1) : 1;
|
||||
|
||||
final sessions = <WindowsPty>[];
|
||||
for (var i = 0; i < count; i++) {
|
||||
final s = WindowsPty.start(
|
||||
executable: 'cmd.exe',
|
||||
// A long-lived child so the ConPTY host stays alive across the whole
|
||||
// kill window. `ping -n 600` loops for ~10 min with no extra deps.
|
||||
arguments: ['/c', 'ping -n 600 127.0.0.1'],
|
||||
columns: 80,
|
||||
rows: 24,
|
||||
environment: {...Platform.environment, 'TERM': 'xterm-256color'},
|
||||
);
|
||||
// Drain output so the reader isolate is actively pumping, closest to a
|
||||
// real live pane. Discard the bytes.
|
||||
s.output.listen((_) {}, onError: (_) {});
|
||||
sessions.add(s);
|
||||
}
|
||||
|
||||
// Signal the driver that the children are up. It waits for the host count
|
||||
// to rise, then kills us.
|
||||
stdout.writeln('PROBE READY dart_pid=$pid child_pids=${sessions.map((s) => s.pid).join(',')}');
|
||||
await stdout.flush();
|
||||
|
||||
// Block WITHOUT ever calling close() — the driver kills us mid-sleep. That
|
||||
// missing teardown is the entire point; do not add a finally/close here.
|
||||
await Future<void>.delayed(const Duration(minutes: 10));
|
||||
exit(0);
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Abrupt-death ConPTY orphan probe — the failure-mode half of the soak.
|
||||
|
||||
.DESCRIPTION
|
||||
soak-conpty.ps1 measures the CLEAN path (dart exits normally, ConPTY
|
||||
teardown runs) and found NO leak on windows-latest: orderly close() reaps
|
||||
every host. That does not exercise the freeze hypothesis (T-424), which is
|
||||
about the parent dying WITHOUT teardown.
|
||||
|
||||
This driver does. Per iteration it launches conpty_orphan_probe.dart — which
|
||||
starts $PtysPerIter real WindowsPty sessions on long-lived children and then
|
||||
blocks WITHOUT ever calling close() — waits for the ConPTY hosts to come up,
|
||||
then force-kills ONLY the dart.exe parent (taskkill /F, no /T) and counts the
|
||||
conhost / OpenConsole / cmd processes that SURVIVE. Because the children are
|
||||
not in a kill-on-close Job Object, abrupt parent death is expected to orphan
|
||||
them; a survivor count that climbs across iterations and never returns to
|
||||
baseline is the leak signature this probe is built to expose.
|
||||
|
||||
By default it does NOT clean up between iterations, so accumulation is
|
||||
visible (the freeze is a cumulative end-state). A final sweep reclaims any
|
||||
strays. Pass -CleanEachIter to isolate the per-kill measurement instead.
|
||||
|
||||
This is a DIAGNOSTIC, not a gate. It writes a CSV + verdict and always
|
||||
succeeds. Re-run it unchanged to validate the T-424 Job Object fix: with the
|
||||
job, survivors should drop to ~0.
|
||||
|
||||
.PARAMETER Iterations How many spawn+kill cycles (default 15).
|
||||
.PARAMETER PtysPerIter WindowsPty sessions spawned per cycle (default 1).
|
||||
.PARAMETER RepoDir clide checkout (default: two levels up).
|
||||
.PARAMETER OutDir Where the CSV + summary land (default LOCALAPPDATA).
|
||||
.PARAMETER SpawnWaitSec Max wait for the hosts to appear before killing.
|
||||
.PARAMETER SettleMs Pause after the kill before counting survivors.
|
||||
.PARAMETER CleanEachIter Reclaim strays after each cycle (isolate per-kill).
|
||||
|
||||
.EXAMPLE
|
||||
pwsh -File soak-conpty-kill.ps1 -Iterations 20 -PtysPerIter 2
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[int] $Iterations = 15,
|
||||
[int] $PtysPerIter = 1,
|
||||
[string] $RepoDir = (Resolve-Path "$PSScriptRoot\..\..").Path,
|
||||
[string] $OutDir = "$env:LOCALAPPDATA\clide\windows-verify",
|
||||
[int] $SpawnWaitSec = 25,
|
||||
[int] $SettleMs = 2000,
|
||||
[switch] $CleanEachIter
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$hostNames = @('conhost', 'OpenConsole', 'cmd')
|
||||
$probe = Join-Path $PSScriptRoot 'conpty_orphan_probe.dart'
|
||||
|
||||
function Get-HostCount {
|
||||
(Get-Process -Name $hostNames -ErrorAction SilentlyContinue | Measure-Object).Count
|
||||
}
|
||||
|
||||
function Clear-StrayHosts([int]$keep) {
|
||||
# Reclaim hosts above the baseline so the runner does not fill with orphans.
|
||||
$alive = Get-Process -Name $hostNames -ErrorAction SilentlyContinue |
|
||||
Sort-Object StartTime -Descending
|
||||
$over = ($alive | Measure-Object).Count - $keep
|
||||
if ($over -gt 0) { $alive | Select-Object -First $over | Stop-Process -Force -ErrorAction SilentlyContinue }
|
||||
}
|
||||
|
||||
if (-not (Get-Command dart -ErrorAction SilentlyContinue)) {
|
||||
throw "dart not on PATH. Run bootstrap-windows.ps1 first (installs Flutter/Dart)."
|
||||
}
|
||||
if (-not (Test-Path (Join-Path $RepoDir 'pubspec.yaml'))) {
|
||||
throw "RepoDir '$RepoDir' does not look like the clide checkout (no pubspec.yaml)."
|
||||
}
|
||||
if (-not (Test-Path $probe)) {
|
||||
throw "probe not found: $probe"
|
||||
}
|
||||
|
||||
New-Item -ItemType Directory -Force -Path $OutDir | Out-Null
|
||||
$stamp = Get-Date -Format 'yyyyMMdd-HHmmss'
|
||||
$csv = Join-Path $OutDir "soak-kill-$stamp.csv"
|
||||
$summary = Join-Path $OutDir "soak-kill-$stamp.summary.txt"
|
||||
|
||||
$os = Get-CimInstance Win32_OperatingSystem
|
||||
"# clide ConPTY ABRUPT-DEATH orphan probe — $(Get-Date -Format o)" | Out-File $summary
|
||||
"# OS build: $($os.Version) ($($os.Caption)) cores: $env:NUMBER_OF_PROCESSORS" | Out-File $summary -Append
|
||||
"# repo: $RepoDir iterations: $Iterations ptys/iter: $PtysPerIter cleanEach: $CleanEachIter" | Out-File $summary -Append
|
||||
"iter,ts,hosts_before,hosts_after_spawn,hosts_after_kill,iter_survivors,cumulative_vs_baseline,spawned_ok,probe_pid" | Out-File $csv
|
||||
|
||||
$baseline = Get-HostCount
|
||||
"baseline,$(Get-Date -Format o),$baseline,,,,0,," | Out-File $csv -Append
|
||||
Write-Host "baseline ConPTY hosts: $baseline" -ForegroundColor Cyan
|
||||
|
||||
$iterSurvivors = @()
|
||||
for ($i = 1; $i -le $Iterations; $i++) {
|
||||
$before = Get-HostCount
|
||||
|
||||
# Launch the probe: it starts $PtysPerIter WindowsPty sessions and blocks
|
||||
# without close(). -PassThru gives us the dart.exe pid to kill.
|
||||
Push-Location $RepoDir
|
||||
$p = Start-Process -FilePath 'dart' `
|
||||
-ArgumentList "run `"$probe`" $PtysPerIter" `
|
||||
-NoNewWindow -PassThru
|
||||
Pop-Location
|
||||
|
||||
# Wait for the ConPTY hosts to actually come up (count rises above $before).
|
||||
$sw = [System.Diagnostics.Stopwatch]::StartNew()
|
||||
$spawned = $false
|
||||
while ($sw.Elapsed.TotalSeconds -lt $SpawnWaitSec) {
|
||||
if ($p.HasExited) { break } # probe died early — unexpected
|
||||
if ((Get-HostCount) -gt $before) { $spawned = $true; Start-Sleep -Milliseconds 600; break }
|
||||
Start-Sleep -Milliseconds 300
|
||||
}
|
||||
$afterSpawn = Get-HostCount
|
||||
|
||||
# ABRUPT parent death: kill ONLY dart.exe. No /T — we are testing whether the
|
||||
# ConPTY children survive their parent (they will, absent a kill-on-close
|
||||
# job; that survival IS the leak).
|
||||
if (-not $p.HasExited) {
|
||||
Start-Process taskkill -ArgumentList "/F /PID $($p.Id)" -NoNewWindow -Wait -ErrorAction SilentlyContinue
|
||||
}
|
||||
|
||||
Start-Sleep -Milliseconds $SettleMs
|
||||
$afterKill = Get-HostCount
|
||||
$survivors = $afterKill - $before # net hosts this cycle left behind
|
||||
$cumulative = $afterKill - $baseline # running accumulation vs baseline
|
||||
$iterSurvivors += $survivors
|
||||
|
||||
$row = "{0},{1},{2},{3},{4},{5},{6},{7},{8}" -f `
|
||||
$i, (Get-Date -Format o), $before, $afterSpawn, $afterKill, $survivors, $cumulative, $spawned, $p.Id
|
||||
$row | Out-File $csv -Append
|
||||
|
||||
$tag = if (-not $spawned) { 'NO-SPAWN' } elseif ($survivors -gt 0) { 'ORPHANED' } else { 'reaped' }
|
||||
$col = if (-not $spawned) { 'DarkYellow' } elseif ($survivors -gt 0) { 'Yellow' } else { 'Green' }
|
||||
Write-Host ("iter {0,3}/{1}: before={2} spawn={3} afterkill={4} survivors={5,3} cum={6,3} {7}" -f `
|
||||
$i, $Iterations, $before, $afterSpawn, $afterKill, $survivors, $cumulative, $tag) -ForegroundColor $col
|
||||
|
||||
if ($CleanEachIter) { Clear-StrayHosts -keep $baseline; Start-Sleep -Milliseconds 500 }
|
||||
}
|
||||
|
||||
# Verdict: did abrupt parent death orphan the ConPTY hosts?
|
||||
$totalSurv = ($iterSurvivors | Measure-Object -Sum).Sum
|
||||
$leakIters = ($iterSurvivors | Where-Object { $_ -gt 0 } | Measure-Object).Count
|
||||
$endCum = (Get-HostCount) - $baseline
|
||||
$verdict = if ($leakIters -ge [math]::Max(2, [int]($Iterations * 0.5))) {
|
||||
"LEAK CONFIRMED (culprit #1 / T-424): abrupt parent death orphaned ConPTY hosts in $leakIters/$Iterations cycles (total survivors $totalSurv). Without a kill-on-close Job Object the children outlive dart.exe."
|
||||
} elseif ($totalSurv -gt 0) {
|
||||
"PARTIAL: orphans appeared in $leakIters/$Iterations cycles (total $totalSurv) but not consistently — re-run with more -Iterations / -PtysPerIter to confirm the slope."
|
||||
} else {
|
||||
"NOT REPRODUCED: every cycle's hosts were reaped even on abrupt kill (the OS broke the pipes and conhost exited). The Job Object may be unnecessary on this OS build, or the leak needs a different trigger."
|
||||
}
|
||||
|
||||
"" | Out-File $summary -Append
|
||||
"iter survivors: $($iterSurvivors -join ',')" | Out-File $summary -Append
|
||||
"cycles with orphans: $leakIters/$Iterations total survivors: $totalSurv end cumulative: $endCum" | Out-File $summary -Append
|
||||
$verdict | Out-File $summary -Append
|
||||
Write-Host "`n$verdict" -ForegroundColor Magenta
|
||||
Write-Host "CSV: $csv"
|
||||
Write-Host "summary: $summary"
|
||||
|
||||
# Always sweep strays at the end so we never leave the box (or a re-run's
|
||||
# baseline) polluted, regardless of -CleanEachIter.
|
||||
Clear-StrayHosts -keep $baseline
|
||||
Reference in New Issue
Block a user