add windows-verify kit: provision + bootstrap + instrumented ConPTY soak

Verification tooling for the Windows freeze assessment (T-424). provision-vm.sh stands up a Windows KVM guest (dry-run unless --go); bootstrap-windows.ps1 installs Flutter + VS C++ Build Tools and checks out the branch; soak-conpty.ps1 runs the pty suite in a fresh dart.exe per iteration and measures the orphaned conhost/cmd count that survives each exit (the leak signature), with a per-iteration timeout so a wedged test can't stall the run. Verifies the ConPTY leak (#1-#4); the GPU/TDR hypothesis (#5) needs passthrough/bare metal (README appendix). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 20:54:46 +02:00
parent 154c2c8e43
commit 9c288a618b
4 changed files with 419 additions and 0 deletions
@@ -0,0 +1,86 @@
+# windows-verify — ConPTY leak verification kit
+
+Tooling to **verify (or refute) the Windows test-freeze assessment** for the
+`windows-support` branch on a real Windows VM. See the full analysis in
+`docs/windows-freeze-analysis.md` (or the report shared in the session).
+
+> **Status: nothing here runs automatically.** These scripts are inert until
+> you invoke them. `provision-vm.sh` is a **dry run** unless you pass `--go`.
+
+## What this verifies — and what it can't
+
+| Hypothesis | In scope here? | Why |
+|---|---|---|
+| **#1** orphaned `conhost.exe`/`cmd.exe` accumulation (no Job Object) | **✅ yes** | `soak-conpty.ps1` measures the orphan count directly — this is the headline result. |
+| **#2** unkillable blocked-FFI isolate threads | ✅ partial | Per-run `dart.exe` peak handle/thread footprint is sampled; the leak is reclaimed at process exit, so it shows as a *within-run* spike, not cross-run growth. |
+| **#3** narrow-terminal CRLF conhost spin | ⚠️ manual | Surfaces as a host that pegs a CPU core; watch Task Manager during a soak. The real fix is clamping `cols/rows ≥ 2` + a unit test. |
+| **#4** `ClosePseudoConsole` hang (pre-24H2) | ⚠️ partial | Run on a **pre-24H2** image (build < 26100) *and* a 24H2+ image to see the version-gated intermittency. |
+| **#5** GPU/display-driver TDR (`0x116`) — the real black-screen | **❌ no** | A stock VM uses a software (WARP) adapter; `make run` cannot trigger a hardware TDR. Needs GPU passthrough or bare metal — see the appendix. |
+
+The leak (#1) is the **test-path** explanation and the actionable one. A VM is
+the right instrument for it; it is the wrong instrument for #5.
+
+## The three steps
+
+1. **Provision the VM** (on the Fedora/KVM host — "danoontje"):
+   ```bash
+   WIN_ISO=~/iso/Win11.iso VIRTIO_ISO=~/iso/virtio-win.iso \
+     tools/windows-verify/provision-vm.sh          # dry run — prints the plan
+   WIN_ISO=... VIRTIO_ISO=... tools/windows-verify/provision-vm.sh --go   # execute
+   ```
+   Finish the interactive Windows install in `virt-viewer` (load the virtio
+   disk driver from the second CD during setup).
+
+2. **Bootstrap the toolchain** (inside Windows, elevated PowerShell):
+   ```powershell
+   pwsh -File tools\windows-verify\bootstrap-windows.ps1
+   ```
+   Installs Git, Flutter/Dart, and VS 2022 Build Tools (C++ workload), then
+   clones + checks out `windows-support` and builds the C CLI.
+
+3. **Run the soak** (new shell, so PATH is fresh):
+   ```powershell
+   pwsh -File tools\windows-verify\soak-conpty.ps1 -Iterations 40
+   ```
+
+## Reading the result
+
+`soak-conpty.ps1` runs the ConPTY suite in a **fresh `dart.exe` per
+iteration** and, after each one exits, counts the `conhost`/`OpenConsole`/`cmd`
+processes that **survived** (baseline-subtracted). It writes a per-iteration
+CSV + a summary to `%LOCALAPPDATA%\clide\windows-verify\` (flushed each line,
+so the data survives even if a later run wedges the box).
+
+- **Orphan count climbs and stays up** (e.g. +1 per iteration, never reclaimed)
+  → **leak confirmed (#1)**: ConPTY hosts outlive the test process. This is the
+  cumulative starvation that, across many runs, thrashes the session to a
+  power-cycle.
+- **Orphan count hovers at ~0** → not reproduced in this config (more
+  iterations may be needed, or the Job-Object fix is already in place).
+- **`dart_peak_handles`/`threads` ratchet up *within* a run** → corroborates #2
+  (blocked reader/waiter isolates), reclaimed when `dart.exe` exits.
+
+The script never tries to crash the machine — it proves the *mechanism* (an
+unreclaimed, monotonically growing host population), which is the safe and
+sufficient verification.
+
+## Safety notes
+
+- Snapshot the VM before soaking (`virsh snapshot-create-as clide-win-verify clean`)
+  so you can roll back instead of reinstalling.
+- If hosts strand after a run: `Get-Process conhost,OpenConsole,cmd | Stop-Process -Force`.
+- Do this in a VM, not a machine you care about — the whole point is to provoke
+  a resource leak.
+
+## Appendix — chasing the GPU/TDR hypothesis (#5)
+
+A software-rendered VM can't reproduce a real display-driver TDR. To test #5
+you need **GPU passthrough** (bind the GPU to `vfio-pci`, pass it with
+`--hostdev`, install the vendor WDDM driver in the guest) or, more simply, run
+`make run` / `make run-testmode` on the **bare-metal Windows box** that
+actually froze. Then, as a *diagnostic only*, raise `TdrDelay` (or set
+`TdrLevel=0`) under
+`HKLM\System\CurrentControlSet\Control\GraphicsDrivers` and see whether a
+previously-rebooting `make run` now only stutters/recovers — and read Event
+Viewer for **Display 4101** / **BugCheck 0x116** after any freeze. Revert the
+registry change afterward.
@@ -0,0 +1,81 @@
+<#
+.SYNOPSIS
+  Provision a Windows VM (or bare machine) with everything needed to build and
+  test clide's windows-support branch, then run the ConPTY soak.
+
+.DESCRIPTION
+  Installs (via winget): Git, the Flutter SDK (brings Dart), and Visual Studio
+  2022 Build Tools with the C++ desktop workload — required for both the
+  ConPTY FFI path and the `clide.c` AF_UNIX CLI (ws2_32 / afunix). Then clones
+  the repo, checks out the branch, runs `flutter pub get`, and builds the C
+  client. Nothing here freezes the box; it just gets you to a runnable state.
+
+  Run from an ELEVATED PowerShell (winget package installs need admin). After
+  it finishes, open a NEW shell so PATH updates take effect, then run
+  soak-conpty.ps1.
+
+.PARAMETER RepoUrl   Git remote (default: the GitHub origin).
+.PARAMETER Branch    Branch to check out (default: windows-support).
+.PARAMETER Dest      Checkout directory (default: %USERPROFILE%\src\clide).
+#>
+[CmdletBinding()]
+param(
+  [string] $RepoUrl = 'https://github.com/postmeridiem/clide.git',
+  [string] $Branch  = 'windows-support',
+  [string] $Dest    = "$env:USERPROFILE\src\clide"
+)
+
+$ErrorActionPreference = 'Stop'
+
+if (-not (Get-Command winget -ErrorAction SilentlyContinue)) {
+  throw "winget not found. Install 'App Installer' from the Microsoft Store (or use a Win11/Server 2022+ image)."
+}
+
+function Install-Pkg([string]$id, [string]$override = $null) {
+  Write-Host "==> winget install $id" -ForegroundColor Cyan
+  $args = @('install', '--id', $id, '-e', '--accept-source-agreements', '--accept-package-agreements')
+  if ($override) { $args += @('--override', $override) }
+  winget @args
+  if ($LASTEXITCODE -ne 0 -and $LASTEXITCODE -ne -1978335189) {  # -1978335189 = already installed
+    throw "winget install $id failed ($LASTEXITCODE)"
+  }
+}
+
+Install-Pkg 'Git.Git'
+Install-Pkg 'Flutter.Flutter'   # provides flutter + bundled dart
+# VS 2022 Build Tools with the native C++ desktop workload (cl.exe, ws2_32, afunix.h).
+Install-Pkg 'Microsoft.VisualStudio.2022.BuildTools' `
+  '--quiet --wait --norestart --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended'
+
+# Re-resolve PATH for this session so the freshly installed tools are visible.
+$env:Path = [System.Environment]::GetEnvironmentVariable('Path','Machine') + ';' +
+            [System.Environment]::GetEnvironmentVariable('Path','User')
+
+if (-not (Get-Command git -ErrorAction SilentlyContinue))     { throw "git not on PATH after install — open a new shell and re-run from the clone step." }
+if (-not (Get-Command flutter -ErrorAction SilentlyContinue)) { throw "flutter not on PATH after install — open a new shell and re-run from the clone step." }
+
+if (-not (Test-Path $Dest)) {
+  Write-Host "==> git clone $RepoUrl -> $Dest" -ForegroundColor Cyan
+  git clone $RepoUrl $Dest
+}
+Push-Location $Dest
+try {
+  git fetch origin $Branch
+  git checkout $Branch
+  Write-Host "==> flutter pub get" -ForegroundColor Cyan
+  flutter pub get
+  # Build the C CLI (needs the VS toolset; ci/build_cli_windows.sh wraps cl.exe).
+  # Requires a bash — Git for Windows ships one at /usr/bin/bash.
+  $bash = Join-Path $env:ProgramFiles 'Git\bin\bash.exe'
+  if (Test-Path $bash) {
+    Write-Host "==> build C CLI (ci/build_cli_windows.sh)" -ForegroundColor Cyan
+    & $bash -lc "cd '$($Dest -replace '\\','/')' && ci/build_cli_windows.sh"
+  } else {
+    Write-Warning "Git Bash not found; skipping C CLI build. Build later with 'make clide-cli' from Git Bash."
+  }
+}
+finally { Pop-Location }
+
+Write-Host "`nReady. Open a NEW shell, then:" -ForegroundColor Green
+Write-Host "  cd $Dest"
+Write-Host "  pwsh -File tools\windows-verify\soak-conpty.ps1 -Iterations 40"
@@ -0,0 +1,88 @@
+#!/usr/bin/env bash
+# Provision a Windows VM on a Fedora/KVM host (this is "danoontje") for the
+# ConPTY soak verification. REVIEW BEFORE RUNNING — by default this is a
+# DRY RUN that only prints what it would do. Pass --go to actually execute.
+#
+# Scope: this VM verifies the ConPTY *resource leak* (culprits #1-#4). It does
+# NOT verify the GPU/TDR hypothesis (#5): a stock VM renders through a software
+# (WARP) adapter, so `make run` cannot exhibit a real display-driver TDR here.
+# Chasing #5 needs GPU passthrough (vfio) or bare metal — see README appendix.
+#
+# Prereqs you must supply:
+#   WIN_ISO     path to a Windows 10/11 or Server 2022 install ISO
+#   VIRTIO_ISO  path to the virtio-win ISO (storage/net drivers)
+#               https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/
+set -euo pipefail
+
+GO=0
+[[ "${1:-}" == "--go" ]] && GO=1
+
+VM_NAME="${VM_NAME:-clide-win-verify}"
+RAM_MB="${RAM_MB:-8192}"
+VCPUS="${VCPUS:-4}"
+DISK_GB="${DISK_GB:-80}"
+DISK_PATH="${DISK_PATH:-/var/lib/libvirt/images/${VM_NAME}.qcow2}"
+WIN_ISO="${WIN_ISO:-}"
+VIRTIO_ISO="${VIRTIO_ISO:-}"
+OS_VARIANT="${OS_VARIANT:-win11}"   # `osinfo-query os` for the full list; win11 needs TPM+UEFI
+
+run() { echo "+ $*"; [[ "$GO" == 1 ]] && "$@"; }
+
+echo "== clide Windows verify VM provisioner =="
+echo "   mode:    $([[ $GO == 1 ]] && echo EXECUTE || echo 'DRY RUN (pass --go to execute)')"
+echo "   vm:      $VM_NAME  ${VCPUS} vCPU / ${RAM_MB}MB / ${DISK_GB}GB"
+echo "   disk:    $DISK_PATH"
+echo "   variant: $OS_VARIANT"
+echo
+
+# 1. Host tooling (Fedora). Idempotent; safe to re-run.
+if ! command -v virt-install >/dev/null 2>&1; then
+  echo "-- installing virtualization stack (needs sudo) --"
+  run sudo dnf install -y @virtualization
+  run sudo systemctl enable --now libvirtd
+else
+  echo "-- virt-install present --"
+fi
+
+# 2. Validate the ISOs the caller must provide.
+if [[ -z "$WIN_ISO" || ! -f "$WIN_ISO" ]]; then
+  echo "!! set WIN_ISO=/path/to/Windows.iso (got: '${WIN_ISO:-unset}')" >&2
+  [[ "$GO" == 1 ]] && exit 2
+fi
+if [[ -z "$VIRTIO_ISO" || ! -f "$VIRTIO_ISO" ]]; then
+  echo "!! set VIRTIO_ISO=/path/to/virtio-win.iso (got: '${VIRTIO_ISO:-unset}')" >&2
+  [[ "$GO" == 1 ]] && exit 2
+fi
+
+# 3. Backing disk.
+run sudo qemu-img create -f qcow2 "$DISK_PATH" "${DISK_GB}G"
+
+# 4. Define + start the VM. UEFI + TPM 2.0 satisfy Win11; drop --tpm and use
+#    --boot uefi=off for older guests. virtio disk/net need the VIRTIO_ISO
+#    drivers loaded during Windows setup ("Load driver" -> the virtio CD).
+run sudo virt-install \
+  --name "$VM_NAME" \
+  --memory "$RAM_MB" \
+  --vcpus "$VCPUS" \
+  --cpu host-passthrough \
+  --os-variant "$OS_VARIANT" \
+  --boot uefi \
+  --tpm backend.type=emulator,backend.version=2.0,model=tpm-crb \
+  --disk "path=$DISK_PATH,bus=virtio,format=qcow2" \
+  --disk "path=$WIN_ISO,device=cdrom,boot.order=1" \
+  --disk "path=$VIRTIO_ISO,device=cdrom" \
+  --network network=default,model=virtio \
+  --graphics spice \
+  --video qxl \
+  --noautoconsole
+
+cat <<EOF
+
+Next:
+  1. virt-viewer $VM_NAME   # finish the interactive Windows install
+                            # (Load driver -> virtio CD for the disk; install virtio NIC after)
+  2. Inside Windows, fetch this repo's tools and run, from an ELEVATED PowerShell:
+       pwsh -File tools\\windows-verify\\bootstrap-windows.ps1
+  3. New shell, then:
+       pwsh -File tools\\windows-verify\\soak-conpty.ps1 -Iterations 40
+EOF
@@ -0,0 +1,164 @@
+<#
+.SYNOPSIS
+  Soak-test the clide ConPTY backend on Windows and measure the orphaned-
+  process / handle / thread leak — the verification for culprit #1 (and the
+  amplifiers #2-#4) from docs/windows-freeze-analysis.
+
+.DESCRIPTION
+  Hypothesis under test (see the freeze report): each WindowsPty.start()
+  pairs the child with its own conhost.exe/OpenConsole.exe, and because the
+  child is NOT placed in a kill-on-close Job Object and the reader isolate
+  blocks forever in ReadFile, those hosts are NOT reaped — they outlive the
+  dart.exe test process and accumulate at the session level. A power-cycle-
+  grade freeze is the cumulative end state of that leak across many runs.
+
+  This script does NOT try to freeze the box. It runs the pty suite in a
+  FRESH dart.exe per iteration (so anything the OS *should* reclaim at process
+  exit is reclaimed) and then counts the conhost/cmd/OpenConsole processes
+  that SURVIVE that exit. A residual count that climbs across iterations and
+  never returns to baseline is the leak signature — it confirms #1 without
+  needing the box to actually die.
+
+  All samples are written line-buffered + flushed to a CSV OUTSIDE the build
+  tree, so the evidence survives even if a later, harsher run does wedge the
+  machine.
+
+.PARAMETER Iterations   How many times to run the pty suite (default 25).
+.PARAMETER RepoDir      Path to the clide checkout (default: two levels up).
+.PARAMETER OutDir       Where to write the CSV + summary (default LOCALAPPDATA).
+.PARAMETER TestSelector dart test args selecting the ConPTY suite.
+.PARAMETER SettleMs     Pause after each iteration before sampling (let the
+                        OS finish reaping legitimately-exited processes).
+.PARAMETER PerIterTimeoutSec  Kill a dart.exe that runs longer than this (a
+                        wedged test) and record the iteration as a hang, so one
+                        stuck run can't stall the whole soak.
+
+.EXAMPLE
+  pwsh -File soak-conpty.ps1 -Iterations 40
+#>
+[CmdletBinding()]
+param(
+  [int]    $Iterations   = 25,
+  [string] $RepoDir      = (Resolve-Path "$PSScriptRoot\..\..").Path,
+  [string] $OutDir       = "$env:LOCALAPPDATA\clide\windows-verify",
+  [string] $TestSelector = "--concurrency=1 --timeout 60s --tags pty test/pty/windows_pty_test.dart",
+  [int]    $SettleMs     = 1500,
+  [int]    $PerIterTimeoutSec = 180
+)
+
+$ErrorActionPreference = 'Stop'
+$hostNames = @('conhost', 'OpenConsole', 'cmd')
+
+function Get-HostCount {
+  # Count the ConPTY host + shell processes currently alive.
+  (Get-Process -Name $hostNames -ErrorAction SilentlyContinue | Measure-Object).Count
+}
+
+function Get-DartFootprint {
+  # Summed handles + threads across every live dart.exe — a within-run
+  # thrash indicator for culprit #2 (blocked-FFI isolate threads).
+  $ds = Get-Process -Name dart -ErrorAction SilentlyContinue
+  if (-not $ds) { return [pscustomobject]@{ procs = 0; handles = 0; threads = 0 } }
+  [pscustomobject]@{
+    procs   = ($ds | Measure-Object).Count
+    handles = ($ds | Measure-Object -Property HandleCount -Sum).Sum
+    threads = ($ds | ForEach-Object { $_.Threads.Count } | Measure-Object -Sum).Sum
+  }
+}
+
+if (-not (Get-Command dart -ErrorAction SilentlyContinue)) {
+  throw "dart not on PATH. Run bootstrap-windows.ps1 first (installs Flutter/Dart)."
+}
+if (-not (Test-Path (Join-Path $RepoDir 'pubspec.yaml'))) {
+  throw "RepoDir '$RepoDir' does not look like the clide checkout (no pubspec.yaml)."
+}
+
+New-Item -ItemType Directory -Force -Path $OutDir | Out-Null
+$stamp   = Get-Date -Format 'yyyyMMdd-HHmmss'
+$csv     = Join-Path $OutDir "soak-$stamp.csv"
+$summary = Join-Path $OutDir "soak-$stamp.summary.txt"
+
+# Self-describing header so a captured CSV stands alone.
+$os = Get-CimInstance Win32_OperatingSystem
+"# clide ConPTY soak — $(Get-Date -Format o)"                                  | Out-File $summary
+"# OS build: $($os.Version) ($($os.Caption))  cores: $env:NUMBER_OF_PROCESSORS" | Out-File $summary -Append
+"# repo: $RepoDir   iterations: $Iterations   selector: $TestSelector"          | Out-File $summary -Append
+"iter,ts,host_count,host_orphans_vs_baseline,dart_peak_handles,dart_peak_threads,test_exit,test_secs,hung" | Out-File $csv
+
+# Drain pre-existing hosts out of the measurement: baseline is whatever is
+# alive BEFORE we spawn anything (Explorer/Terminal already own some conhosts).
+$baseline = Get-HostCount
+"baseline,$(Get-Date -Format o),$baseline,0,,,,," | Out-File $csv -Append
+Write-Host "baseline ConPTY hosts: $baseline" -ForegroundColor Cyan
+
+$series = @()
+for ($i = 1; $i -le $Iterations; $i++) {
+  $sw = [System.Diagnostics.Stopwatch]::StartNew()
+
+  # Fresh dart.exe per iteration: Push-Location so `dart test` resolves the
+  # package. -PassThru lets us poll its footprint while it runs.
+  Push-Location $RepoDir
+  $p = Start-Process -FilePath 'dart' `
+        -ArgumentList "test $TestSelector" `
+        -NoNewWindow -PassThru
+  $peakHandles = 0; $peakThreads = 0; $hung = $false
+  while (-not $p.HasExited) {
+    if ($sw.Elapsed.TotalSeconds -gt $PerIterTimeoutSec) {
+      # A wedged dart.exe (e.g. a ConPTY reader blocked forever in ReadFile)
+      # would otherwise hang the soak. Kill the whole process tree and record
+      # the iteration as a hang instead of spinning here indefinitely.
+      Start-Process taskkill -ArgumentList "/T /F /PID $($p.Id)" -NoNewWindow -Wait -ErrorAction SilentlyContinue
+      $hung = $true
+      break
+    }
+    $fp = Get-DartFootprint
+    if ($fp.handles -gt $peakHandles) { $peakHandles = $fp.handles }
+    if ($fp.threads -gt $peakThreads) { $peakThreads = $fp.threads }
+    Start-Sleep -Milliseconds 250
+  }
+  $exit = if ($hung) { 'TIMEOUT' } else { $p.ExitCode }
+  Pop-Location
+  $sw.Stop()
+
+  # The dart.exe is gone. Anything the ConPTY teardown reaped properly is
+  # gone with it. Let the OS settle, then count what SURVIVED.
+  Start-Sleep -Milliseconds $SettleMs
+  $now      = Get-HostCount
+  $orphans  = $now - $baseline
+  $series  += $orphans
+
+  $row = "{0},{1},{2},{3},{4},{5},{6},{7},{8}" -f `
+    $i, (Get-Date -Format o), $now, $orphans, $peakHandles, $peakThreads, $exit, [math]::Round($sw.Elapsed.TotalSeconds, 1), $hung
+  $row | Out-File $csv -Append   # Out-File flushes per call — survives a wedge.
+
+  $tag = if ($hung) { 'HANG' } elseif ($orphans -gt 0) { 'LEAK?' } else { 'clean' }
+  $col = if ($hung) { 'Red' } elseif ($orphans -gt 0) { 'Yellow' } else { 'Green' }
+  Write-Host ("iter {0,3}/{1}: hosts={2} orphans={3,+4} dart_peak_handles={4} threads={5} exit={6} {7}" -f `
+    $i, $Iterations, $now, $orphans, $peakHandles, $peakThreads, $exit, $tag) -ForegroundColor $col
+}
+
+# Verdict: did the orphan count trend UP and stay up? A leak shows a positive
+# slope and an end-state well above baseline; a clean run hovers at ~0.
+$final = $series[-1]
+$max   = ($series | Measure-Object -Maximum).Maximum
+$first = $series[0]
+$verdict = if ($final -ge 3 -and $final -ge $first + 2) {
+  "LEAK CONFIRMED (culprit #1): orphaned ConPTY hosts grew to $final over $Iterations runs and did not reclaim."
+} elseif ($max -ge 3) {
+  "INCONCLUSIVE: orphans peaked at $max but did not hold ($final at end) — re-run with more -Iterations."
+} else {
+  "NOT REPRODUCED here: orphan count stayed near baseline (max $max). The leak may need more runs, or the fix is already present."
+}
+
+"" | Out-File $summary -Append
+"final orphans: $final   peak orphans: $max   series: $($series -join ',')" | Out-File $summary -Append
+$verdict | Out-File $summary -Append
+Write-Host "`n$verdict" -ForegroundColor Magenta
+Write-Host "CSV:     $csv"
+Write-Host "summary: $summary"
+
+# Leave the user a cleanup handle for any stranded hosts.
+$stray = Get-Process -Name $hostNames -ErrorAction SilentlyContinue
+if (($stray | Measure-Object).Count -gt $baseline) {
+  Write-Host "`nStray hosts still alive. To reclaim: Get-Process conhost,OpenConsole,cmd | Stop-Process -Force" -ForegroundColor DarkYellow
+}