diff --git a/tools/windows-verify/README.md b/tools/windows-verify/README.md new file mode 100644 index 00000000..ee79cc8f --- /dev/null +++ b/tools/windows-verify/README.md @@ -0,0 +1,86 @@ +# windows-verify — ConPTY leak verification kit + +Tooling to **verify (or refute) the Windows test-freeze assessment** for the +`windows-support` branch on a real Windows VM. See the full analysis in +`docs/windows-freeze-analysis.md` (or the report shared in the session). + +> **Status: nothing here runs automatically.** These scripts are inert until +> you invoke them. `provision-vm.sh` is a **dry run** unless you pass `--go`. + +## What this verifies — and what it can't + +| Hypothesis | In scope here? | Why | +|---|---|---| +| **#1** orphaned `conhost.exe`/`cmd.exe` accumulation (no Job Object) | **✅ yes** | `soak-conpty.ps1` measures the orphan count directly — this is the headline result. | +| **#2** unkillable blocked-FFI isolate threads | ✅ partial | Per-run `dart.exe` peak handle/thread footprint is sampled; the leak is reclaimed at process exit, so it shows as a *within-run* spike, not cross-run growth. | +| **#3** narrow-terminal CRLF conhost spin | ⚠️ manual | Surfaces as a host that pegs a CPU core; watch Task Manager during a soak. The real fix is clamping `cols/rows ≥ 2` + a unit test. | +| **#4** `ClosePseudoConsole` hang (pre-24H2) | ⚠️ partial | Run on a **pre-24H2** image (build < 26100) *and* a 24H2+ image to see the version-gated intermittency. | +| **#5** GPU/display-driver TDR (`0x116`) — the real black-screen | **❌ no** | A stock VM uses a software (WARP) adapter; `make run` cannot trigger a hardware TDR. Needs GPU passthrough or bare metal — see the appendix. | + +The leak (#1) is the **test-path** explanation and the actionable one. A VM is +the right instrument for it; it is the wrong instrument for #5. + +## The three steps + +1. **Provision the VM** (on the Fedora/KVM host — "danoontje"): + ```bash + WIN_ISO=~/iso/Win11.iso VIRTIO_ISO=~/iso/virtio-win.iso \ + tools/windows-verify/provision-vm.sh # dry run — prints the plan + WIN_ISO=... VIRTIO_ISO=... tools/windows-verify/provision-vm.sh --go # execute + ``` + Finish the interactive Windows install in `virt-viewer` (load the virtio + disk driver from the second CD during setup). + +2. **Bootstrap the toolchain** (inside Windows, elevated PowerShell): + ```powershell + pwsh -File tools\windows-verify\bootstrap-windows.ps1 + ``` + Installs Git, Flutter/Dart, and VS 2022 Build Tools (C++ workload), then + clones + checks out `windows-support` and builds the C CLI. + +3. **Run the soak** (new shell, so PATH is fresh): + ```powershell + pwsh -File tools\windows-verify\soak-conpty.ps1 -Iterations 40 + ``` + +## Reading the result + +`soak-conpty.ps1` runs the ConPTY suite in a **fresh `dart.exe` per +iteration** and, after each one exits, counts the `conhost`/`OpenConsole`/`cmd` +processes that **survived** (baseline-subtracted). It writes a per-iteration +CSV + a summary to `%LOCALAPPDATA%\clide\windows-verify\` (flushed each line, +so the data survives even if a later run wedges the box). + +- **Orphan count climbs and stays up** (e.g. +1 per iteration, never reclaimed) + → **leak confirmed (#1)**: ConPTY hosts outlive the test process. This is the + cumulative starvation that, across many runs, thrashes the session to a + power-cycle. +- **Orphan count hovers at ~0** → not reproduced in this config (more + iterations may be needed, or the Job-Object fix is already in place). +- **`dart_peak_handles`/`threads` ratchet up *within* a run** → corroborates #2 + (blocked reader/waiter isolates), reclaimed when `dart.exe` exits. + +The script never tries to crash the machine — it proves the *mechanism* (an +unreclaimed, monotonically growing host population), which is the safe and +sufficient verification. + +## Safety notes + +- Snapshot the VM before soaking (`virsh snapshot-create-as clide-win-verify clean`) + so you can roll back instead of reinstalling. +- If hosts strand after a run: `Get-Process conhost,OpenConsole,cmd | Stop-Process -Force`. +- Do this in a VM, not a machine you care about — the whole point is to provoke + a resource leak. + +## Appendix — chasing the GPU/TDR hypothesis (#5) + +A software-rendered VM can't reproduce a real display-driver TDR. To test #5 +you need **GPU passthrough** (bind the GPU to `vfio-pci`, pass it with +`--hostdev`, install the vendor WDDM driver in the guest) or, more simply, run +`make run` / `make run-testmode` on the **bare-metal Windows box** that +actually froze. Then, as a *diagnostic only*, raise `TdrDelay` (or set +`TdrLevel=0`) under +`HKLM\System\CurrentControlSet\Control\GraphicsDrivers` and see whether a +previously-rebooting `make run` now only stutters/recovers — and read Event +Viewer for **Display 4101** / **BugCheck 0x116** after any freeze. Revert the +registry change afterward. diff --git a/tools/windows-verify/bootstrap-windows.ps1 b/tools/windows-verify/bootstrap-windows.ps1 new file mode 100644 index 00000000..8c70277f --- /dev/null +++ b/tools/windows-verify/bootstrap-windows.ps1 @@ -0,0 +1,81 @@ +<# +.SYNOPSIS + Provision a Windows VM (or bare machine) with everything needed to build and + test clide's windows-support branch, then run the ConPTY soak. + +.DESCRIPTION + Installs (via winget): Git, the Flutter SDK (brings Dart), and Visual Studio + 2022 Build Tools with the C++ desktop workload — required for both the + ConPTY FFI path and the `clide.c` AF_UNIX CLI (ws2_32 / afunix). Then clones + the repo, checks out the branch, runs `flutter pub get`, and builds the C + client. Nothing here freezes the box; it just gets you to a runnable state. + + Run from an ELEVATED PowerShell (winget package installs need admin). After + it finishes, open a NEW shell so PATH updates take effect, then run + soak-conpty.ps1. + +.PARAMETER RepoUrl Git remote (default: the GitHub origin). +.PARAMETER Branch Branch to check out (default: windows-support). +.PARAMETER Dest Checkout directory (default: %USERPROFILE%\src\clide). +#> +[CmdletBinding()] +param( + [string] $RepoUrl = 'https://github.com/postmeridiem/clide.git', + [string] $Branch = 'windows-support', + [string] $Dest = "$env:USERPROFILE\src\clide" +) + +$ErrorActionPreference = 'Stop' + +if (-not (Get-Command winget -ErrorAction SilentlyContinue)) { + throw "winget not found. Install 'App Installer' from the Microsoft Store (or use a Win11/Server 2022+ image)." +} + +function Install-Pkg([string]$id, [string]$override = $null) { + Write-Host "==> winget install $id" -ForegroundColor Cyan + $args = @('install', '--id', $id, '-e', '--accept-source-agreements', '--accept-package-agreements') + if ($override) { $args += @('--override', $override) } + winget @args + if ($LASTEXITCODE -ne 0 -and $LASTEXITCODE -ne -1978335189) { # -1978335189 = already installed + throw "winget install $id failed ($LASTEXITCODE)" + } +} + +Install-Pkg 'Git.Git' +Install-Pkg 'Flutter.Flutter' # provides flutter + bundled dart +# VS 2022 Build Tools with the native C++ desktop workload (cl.exe, ws2_32, afunix.h). +Install-Pkg 'Microsoft.VisualStudio.2022.BuildTools' ` + '--quiet --wait --norestart --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended' + +# Re-resolve PATH for this session so the freshly installed tools are visible. +$env:Path = [System.Environment]::GetEnvironmentVariable('Path','Machine') + ';' + + [System.Environment]::GetEnvironmentVariable('Path','User') + +if (-not (Get-Command git -ErrorAction SilentlyContinue)) { throw "git not on PATH after install — open a new shell and re-run from the clone step." } +if (-not (Get-Command flutter -ErrorAction SilentlyContinue)) { throw "flutter not on PATH after install — open a new shell and re-run from the clone step." } + +if (-not (Test-Path $Dest)) { + Write-Host "==> git clone $RepoUrl -> $Dest" -ForegroundColor Cyan + git clone $RepoUrl $Dest +} +Push-Location $Dest +try { + git fetch origin $Branch + git checkout $Branch + Write-Host "==> flutter pub get" -ForegroundColor Cyan + flutter pub get + # Build the C CLI (needs the VS toolset; ci/build_cli_windows.sh wraps cl.exe). + # Requires a bash — Git for Windows ships one at /usr/bin/bash. + $bash = Join-Path $env:ProgramFiles 'Git\bin\bash.exe' + if (Test-Path $bash) { + Write-Host "==> build C CLI (ci/build_cli_windows.sh)" -ForegroundColor Cyan + & $bash -lc "cd '$($Dest -replace '\\','/')' && ci/build_cli_windows.sh" + } else { + Write-Warning "Git Bash not found; skipping C CLI build. Build later with 'make clide-cli' from Git Bash." + } +} +finally { Pop-Location } + +Write-Host "`nReady. Open a NEW shell, then:" -ForegroundColor Green +Write-Host " cd $Dest" +Write-Host " pwsh -File tools\windows-verify\soak-conpty.ps1 -Iterations 40" diff --git a/tools/windows-verify/provision-vm.sh b/tools/windows-verify/provision-vm.sh new file mode 100755 index 00000000..bd766774 --- /dev/null +++ b/tools/windows-verify/provision-vm.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash +# Provision a Windows VM on a Fedora/KVM host (this is "danoontje") for the +# ConPTY soak verification. REVIEW BEFORE RUNNING — by default this is a +# DRY RUN that only prints what it would do. Pass --go to actually execute. +# +# Scope: this VM verifies the ConPTY *resource leak* (culprits #1-#4). It does +# NOT verify the GPU/TDR hypothesis (#5): a stock VM renders through a software +# (WARP) adapter, so `make run` cannot exhibit a real display-driver TDR here. +# Chasing #5 needs GPU passthrough (vfio) or bare metal — see README appendix. +# +# Prereqs you must supply: +# WIN_ISO path to a Windows 10/11 or Server 2022 install ISO +# VIRTIO_ISO path to the virtio-win ISO (storage/net drivers) +# https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/ +set -euo pipefail + +GO=0 +[[ "${1:-}" == "--go" ]] && GO=1 + +VM_NAME="${VM_NAME:-clide-win-verify}" +RAM_MB="${RAM_MB:-8192}" +VCPUS="${VCPUS:-4}" +DISK_GB="${DISK_GB:-80}" +DISK_PATH="${DISK_PATH:-/var/lib/libvirt/images/${VM_NAME}.qcow2}" +WIN_ISO="${WIN_ISO:-}" +VIRTIO_ISO="${VIRTIO_ISO:-}" +OS_VARIANT="${OS_VARIANT:-win11}" # `osinfo-query os` for the full list; win11 needs TPM+UEFI + +run() { echo "+ $*"; [[ "$GO" == 1 ]] && "$@"; } + +echo "== clide Windows verify VM provisioner ==" +echo " mode: $([[ $GO == 1 ]] && echo EXECUTE || echo 'DRY RUN (pass --go to execute)')" +echo " vm: $VM_NAME ${VCPUS} vCPU / ${RAM_MB}MB / ${DISK_GB}GB" +echo " disk: $DISK_PATH" +echo " variant: $OS_VARIANT" +echo + +# 1. Host tooling (Fedora). Idempotent; safe to re-run. +if ! command -v virt-install >/dev/null 2>&1; then + echo "-- installing virtualization stack (needs sudo) --" + run sudo dnf install -y @virtualization + run sudo systemctl enable --now libvirtd +else + echo "-- virt-install present --" +fi + +# 2. Validate the ISOs the caller must provide. +if [[ -z "$WIN_ISO" || ! -f "$WIN_ISO" ]]; then + echo "!! set WIN_ISO=/path/to/Windows.iso (got: '${WIN_ISO:-unset}')" >&2 + [[ "$GO" == 1 ]] && exit 2 +fi +if [[ -z "$VIRTIO_ISO" || ! -f "$VIRTIO_ISO" ]]; then + echo "!! set VIRTIO_ISO=/path/to/virtio-win.iso (got: '${VIRTIO_ISO:-unset}')" >&2 + [[ "$GO" == 1 ]] && exit 2 +fi + +# 3. Backing disk. +run sudo qemu-img create -f qcow2 "$DISK_PATH" "${DISK_GB}G" + +# 4. Define + start the VM. UEFI + TPM 2.0 satisfy Win11; drop --tpm and use +# --boot uefi=off for older guests. virtio disk/net need the VIRTIO_ISO +# drivers loaded during Windows setup ("Load driver" -> the virtio CD). +run sudo virt-install \ + --name "$VM_NAME" \ + --memory "$RAM_MB" \ + --vcpus "$VCPUS" \ + --cpu host-passthrough \ + --os-variant "$OS_VARIANT" \ + --boot uefi \ + --tpm backend.type=emulator,backend.version=2.0,model=tpm-crb \ + --disk "path=$DISK_PATH,bus=virtio,format=qcow2" \ + --disk "path=$WIN_ISO,device=cdrom,boot.order=1" \ + --disk "path=$VIRTIO_ISO,device=cdrom" \ + --network network=default,model=virtio \ + --graphics spice \ + --video qxl \ + --noautoconsole + +cat < virtio CD for the disk; install virtio NIC after) + 2. Inside Windows, fetch this repo's tools and run, from an ELEVATED PowerShell: + pwsh -File tools\\windows-verify\\bootstrap-windows.ps1 + 3. New shell, then: + pwsh -File tools\\windows-verify\\soak-conpty.ps1 -Iterations 40 +EOF diff --git a/tools/windows-verify/soak-conpty.ps1 b/tools/windows-verify/soak-conpty.ps1 new file mode 100644 index 00000000..e2676a24 --- /dev/null +++ b/tools/windows-verify/soak-conpty.ps1 @@ -0,0 +1,164 @@ +<# +.SYNOPSIS + Soak-test the clide ConPTY backend on Windows and measure the orphaned- + process / handle / thread leak — the verification for culprit #1 (and the + amplifiers #2-#4) from docs/windows-freeze-analysis. + +.DESCRIPTION + Hypothesis under test (see the freeze report): each WindowsPty.start() + pairs the child with its own conhost.exe/OpenConsole.exe, and because the + child is NOT placed in a kill-on-close Job Object and the reader isolate + blocks forever in ReadFile, those hosts are NOT reaped — they outlive the + dart.exe test process and accumulate at the session level. A power-cycle- + grade freeze is the cumulative end state of that leak across many runs. + + This script does NOT try to freeze the box. It runs the pty suite in a + FRESH dart.exe per iteration (so anything the OS *should* reclaim at process + exit is reclaimed) and then counts the conhost/cmd/OpenConsole processes + that SURVIVE that exit. A residual count that climbs across iterations and + never returns to baseline is the leak signature — it confirms #1 without + needing the box to actually die. + + All samples are written line-buffered + flushed to a CSV OUTSIDE the build + tree, so the evidence survives even if a later, harsher run does wedge the + machine. + +.PARAMETER Iterations How many times to run the pty suite (default 25). +.PARAMETER RepoDir Path to the clide checkout (default: two levels up). +.PARAMETER OutDir Where to write the CSV + summary (default LOCALAPPDATA). +.PARAMETER TestSelector dart test args selecting the ConPTY suite. +.PARAMETER SettleMs Pause after each iteration before sampling (let the + OS finish reaping legitimately-exited processes). +.PARAMETER PerIterTimeoutSec Kill a dart.exe that runs longer than this (a + wedged test) and record the iteration as a hang, so one + stuck run can't stall the whole soak. + +.EXAMPLE + pwsh -File soak-conpty.ps1 -Iterations 40 +#> +[CmdletBinding()] +param( + [int] $Iterations = 25, + [string] $RepoDir = (Resolve-Path "$PSScriptRoot\..\..").Path, + [string] $OutDir = "$env:LOCALAPPDATA\clide\windows-verify", + [string] $TestSelector = "--concurrency=1 --timeout 60s --tags pty test/pty/windows_pty_test.dart", + [int] $SettleMs = 1500, + [int] $PerIterTimeoutSec = 180 +) + +$ErrorActionPreference = 'Stop' +$hostNames = @('conhost', 'OpenConsole', 'cmd') + +function Get-HostCount { + # Count the ConPTY host + shell processes currently alive. + (Get-Process -Name $hostNames -ErrorAction SilentlyContinue | Measure-Object).Count +} + +function Get-DartFootprint { + # Summed handles + threads across every live dart.exe — a within-run + # thrash indicator for culprit #2 (blocked-FFI isolate threads). + $ds = Get-Process -Name dart -ErrorAction SilentlyContinue + if (-not $ds) { return [pscustomobject]@{ procs = 0; handles = 0; threads = 0 } } + [pscustomobject]@{ + procs = ($ds | Measure-Object).Count + handles = ($ds | Measure-Object -Property HandleCount -Sum).Sum + threads = ($ds | ForEach-Object { $_.Threads.Count } | Measure-Object -Sum).Sum + } +} + +if (-not (Get-Command dart -ErrorAction SilentlyContinue)) { + throw "dart not on PATH. Run bootstrap-windows.ps1 first (installs Flutter/Dart)." +} +if (-not (Test-Path (Join-Path $RepoDir 'pubspec.yaml'))) { + throw "RepoDir '$RepoDir' does not look like the clide checkout (no pubspec.yaml)." +} + +New-Item -ItemType Directory -Force -Path $OutDir | Out-Null +$stamp = Get-Date -Format 'yyyyMMdd-HHmmss' +$csv = Join-Path $OutDir "soak-$stamp.csv" +$summary = Join-Path $OutDir "soak-$stamp.summary.txt" + +# Self-describing header so a captured CSV stands alone. +$os = Get-CimInstance Win32_OperatingSystem +"# clide ConPTY soak — $(Get-Date -Format o)" | Out-File $summary +"# OS build: $($os.Version) ($($os.Caption)) cores: $env:NUMBER_OF_PROCESSORS" | Out-File $summary -Append +"# repo: $RepoDir iterations: $Iterations selector: $TestSelector" | Out-File $summary -Append +"iter,ts,host_count,host_orphans_vs_baseline,dart_peak_handles,dart_peak_threads,test_exit,test_secs,hung" | Out-File $csv + +# Drain pre-existing hosts out of the measurement: baseline is whatever is +# alive BEFORE we spawn anything (Explorer/Terminal already own some conhosts). +$baseline = Get-HostCount +"baseline,$(Get-Date -Format o),$baseline,0,,,,," | Out-File $csv -Append +Write-Host "baseline ConPTY hosts: $baseline" -ForegroundColor Cyan + +$series = @() +for ($i = 1; $i -le $Iterations; $i++) { + $sw = [System.Diagnostics.Stopwatch]::StartNew() + + # Fresh dart.exe per iteration: Push-Location so `dart test` resolves the + # package. -PassThru lets us poll its footprint while it runs. + Push-Location $RepoDir + $p = Start-Process -FilePath 'dart' ` + -ArgumentList "test $TestSelector" ` + -NoNewWindow -PassThru + $peakHandles = 0; $peakThreads = 0; $hung = $false + while (-not $p.HasExited) { + if ($sw.Elapsed.TotalSeconds -gt $PerIterTimeoutSec) { + # A wedged dart.exe (e.g. a ConPTY reader blocked forever in ReadFile) + # would otherwise hang the soak. Kill the whole process tree and record + # the iteration as a hang instead of spinning here indefinitely. + Start-Process taskkill -ArgumentList "/T /F /PID $($p.Id)" -NoNewWindow -Wait -ErrorAction SilentlyContinue + $hung = $true + break + } + $fp = Get-DartFootprint + if ($fp.handles -gt $peakHandles) { $peakHandles = $fp.handles } + if ($fp.threads -gt $peakThreads) { $peakThreads = $fp.threads } + Start-Sleep -Milliseconds 250 + } + $exit = if ($hung) { 'TIMEOUT' } else { $p.ExitCode } + Pop-Location + $sw.Stop() + + # The dart.exe is gone. Anything the ConPTY teardown reaped properly is + # gone with it. Let the OS settle, then count what SURVIVED. + Start-Sleep -Milliseconds $SettleMs + $now = Get-HostCount + $orphans = $now - $baseline + $series += $orphans + + $row = "{0},{1},{2},{3},{4},{5},{6},{7},{8}" -f ` + $i, (Get-Date -Format o), $now, $orphans, $peakHandles, $peakThreads, $exit, [math]::Round($sw.Elapsed.TotalSeconds, 1), $hung + $row | Out-File $csv -Append # Out-File flushes per call — survives a wedge. + + $tag = if ($hung) { 'HANG' } elseif ($orphans -gt 0) { 'LEAK?' } else { 'clean' } + $col = if ($hung) { 'Red' } elseif ($orphans -gt 0) { 'Yellow' } else { 'Green' } + Write-Host ("iter {0,3}/{1}: hosts={2} orphans={3,+4} dart_peak_handles={4} threads={5} exit={6} {7}" -f ` + $i, $Iterations, $now, $orphans, $peakHandles, $peakThreads, $exit, $tag) -ForegroundColor $col +} + +# Verdict: did the orphan count trend UP and stay up? A leak shows a positive +# slope and an end-state well above baseline; a clean run hovers at ~0. +$final = $series[-1] +$max = ($series | Measure-Object -Maximum).Maximum +$first = $series[0] +$verdict = if ($final -ge 3 -and $final -ge $first + 2) { + "LEAK CONFIRMED (culprit #1): orphaned ConPTY hosts grew to $final over $Iterations runs and did not reclaim." +} elseif ($max -ge 3) { + "INCONCLUSIVE: orphans peaked at $max but did not hold ($final at end) — re-run with more -Iterations." +} else { + "NOT REPRODUCED here: orphan count stayed near baseline (max $max). The leak may need more runs, or the fix is already present." +} + +"" | Out-File $summary -Append +"final orphans: $final peak orphans: $max series: $($series -join ',')" | Out-File $summary -Append +$verdict | Out-File $summary -Append +Write-Host "`n$verdict" -ForegroundColor Magenta +Write-Host "CSV: $csv" +Write-Host "summary: $summary" + +# Leave the user a cleanup handle for any stranded hosts. +$stray = Get-Process -Name $hostNames -ErrorAction SilentlyContinue +if (($stray | Measure-Object).Count -gt $baseline) { + Write-Host "`nStray hosts still alive. To reclaim: Get-Process conhost,OpenConsole,cmd | Stop-Process -Force" -ForegroundColor DarkYellow +}