diff --git a/CHANGELOG.md b/CHANGELOG.md index f72a57ff..72dc12ab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,7 @@ heading, and (b) bumping `project.yaml` `version:` in the same commit. ### Added +- Testing docs under `docs/testing/`: `README.md` (what each layer covers, how to run, local vs CI flow), `a11y-manual.md` (15-minute Orca + VoiceOver checklist run at every tier cut), `claude-ui-workflow.md` (how Claude Code drives the app through the Playwright harness, including the `flt-semantics-placeholder` quirk). - Makefile targets for every test layer and the UI harness: `test`, `test-a11y`, `test-integration`, `test-e2e`, `test-all`, `coverage`, `smoke-bundle`, `ui-dev`, `ui-stop`, `ui-smoke`. `push-check` now runs `test + test-a11y` (fast pre-push gate, <90s). - Per-layer CI shell scripts under `ci/`: `test.sh` (analyze + format + unit + widget + golden, ~5s), `test_a11y.sh` (a11y contract), `test_integration.sh` (integration_test one file at a time — desktop can't batch them reliably), `test_e2e.sh` (daemon subprocess + browser WASM Playwright smoke), `smoke_bundle.sh` (xvfb-run the Linux release bundle for 5s; catches dynamic-linker / asset-bundle / plugin-init regressions that widget tests can't see), `coverage.sh` (flutter test --coverage + lcov summary). - `.gitea/workflows/test.yml` — four-job pipeline (`unit`, `integration`, `startup-bundle`, `e2e`) that shells out to the `ci/*.sh` scripts. **Not activated yet** — Gitea Actions has to be enabled in the instance settings first. GitHub-Actions-syntax-compatible, so copying to `.github/workflows/` is a one-file move when the repo migrates. diff --git a/docs/testing/README.md b/docs/testing/README.md new file mode 100644 index 00000000..f711a77e --- /dev/null +++ b/docs/testing/README.md @@ -0,0 +1,58 @@ +# Testing clide + +Every layer of the test pyramid is wired. The suite is intentionally +designed to run client-side only — `git clone && make push-check` works +on any Linux or macOS dev box without network access or shared state. + +## Layers + +| layer | location | runner | time | when | +|---|---|---|---|---| +| unit (root) | `test/` | `dart test` | ~5s | `make test` | +| unit + widget + golden (app) | `app/test/` | `flutter test` | ~30s | `make test` | +| a11y contract | `app/test/a11y/` | `flutter test` | ~5s | `make test-a11y` | +| integration (startup gate) | `app/integration_test/` | `flutter test integration_test/` | ~60s | `make test-integration` | +| daemon E2E + web WASM smoke | `test/daemon/` + `tools/ui/tests/` | `dart test` + Playwright | ~60s | `make test-e2e` | +| startup bundle smoke | `ci/smoke_bundle.sh` | xvfb-run, 5s timeout | ~30s | `make smoke-bundle` | + +## Dev loop + +```bash +make push-check # gate before `git push` (~90s) +make test-all # full pyramid (~3-5min) +``` + +## A11y + i18n + +A11y is a day-one contract, not a Tier-6 polish. Every interactive widget +emits a `Semantics` node with label + hint. The a11y suite enforces: + +- **semantic_coverage_test.dart** — every built-in declares title + version + and every interactive contribution carries the fields needed to build a + Semantics node. +- **contrast_test.dart** — every bundled theme meets WCAG-AA on the token + pairs in `lib/kernel/src/theme/contrast.dart`. Catches token-pair + regressions (foreground-on-background, tab.active-on-bg, etc.). +- **i18n_coverage_test.dart** — every i18n key referenced by Tier-0 + built-ins exists in its en_US catalog. Asserts key presence directly + (not "returned value == key" — that's ambiguous for keys that happen + to equal their English translation). +- **keyboard_traversal_test.dart** — interactive widgets are focusable + and expose tap actions to a11y. + +See [`a11y-manual.md`](a11y-manual.md) for the 15-minute manual +screen-reader checklist run at every tier cut. + +## Interacting with the app (Claude + humans) + +Claude Code drives the Flutter WASM build through a Playwright harness +— not via screenshots. Semantics are the contract. See +[`claude-ui-workflow.md`](claude-ui-workflow.md) for the loop. + +## CI + +`.gitea/workflows/test.yml` is ready to run but Gitea Actions is not +enabled on the instance yet. When the user flips it, four jobs kick off +per push: `unit`, `integration`, `startup-bundle`, `e2e`. The workflow +file is GitHub-Actions compatible — copying it to `.github/workflows/` +is the entire migration if the repo moves to GitHub. diff --git a/docs/testing/a11y-manual.md b/docs/testing/a11y-manual.md new file mode 100644 index 00000000..c1c911b3 --- /dev/null +++ b/docs/testing/a11y-manual.md @@ -0,0 +1,47 @@ +# Manual a11y checklist — run at every tier cut + +Automated a11y tests catch structural regressions (missing Semantics, +broken contrast, missing i18n keys). They don't catch "this label is +technically correct but reads like robot noise." A 15-minute manual +pass at every tier cut does. + +## Linux — Orca + +1. Start Orca: `orca -s` (settings), toggle on "Focus follows mouse". +2. Launch clide: `flutter run -d linux` or the built bundle. +3. Tab through every slot in order. Every stop should read + - panel name, then + - a one-line verb describing what the widget does. +4. Open the command palette (`ctrl+k` once it's wired). Read each + command out loud. Flag commands whose name alone doesn't make + it obvious what they do. +5. Open the theme picker. Orca should announce "Select theme", then + read each row as "theme name, activate this theme". Selecting + announces the new theme. +6. Disconnect the daemon mid-session. Orca should announce the + statusbar change (it's a live region). + +## macOS — VoiceOver + +VoiceOver: `cmd+F5`. Same checklist as Orca. Extra pass: +- Rotor navigation (`ctrl+opt+u`) should list every Semantic + landmark; no "unknown group" entries. +- Per-element hints read cleanly when pressing `ctrl+opt+shift+h`. + +## Report a drift + +Any label that feels off — noise, tech jargon, abbreviations screen +readers mispronounce — becomes a follow-up ticket. Do not "fix" in +the branch being cut; queue it so each tier stays focused. File +under `docs/todo-a11y.md` (create if absent) with: + +``` +- [ ] ext.: — , +``` + +## Why this isn't automated + +Automation catches structural violations. Automation can't judge prose. +A 15-minute human pass per tier stays cheap and catches the gap. +Revisit automation (axe-core via Playwright, Flutter a11y harnesses) +if we ship a public build. diff --git a/docs/testing/claude-ui-workflow.md b/docs/testing/claude-ui-workflow.md new file mode 100644 index 00000000..a0582386 --- /dev/null +++ b/docs/testing/claude-ui-workflow.md @@ -0,0 +1,86 @@ +# How Claude Code uses clide + +Claude Code needs to *actually use* the app while building features. +The pipeline: + +1. **Flutter builds the app to WASM.** `flutter build web --wasm` ships + a CanvasKit/Skwasm bundle under `app/build/web/`. +2. **A local server serves it.** `tools/ui/serve.sh` starts + `http://localhost:4280` in the background with a pidfile. +3. **Playwright drives a headless Chromium.** Instead of click-by-pixel + — which is brittle on a CanvasKit `` — the driver queries + Flutter's semantic tree (`flt-semantics[aria-label=…]`), the same + tree screen readers use. This is only reliable because every + interactive widget in clide already ships `Semantics(label:, hint:)` + wrappers for a11y — the automation surface is a free side-benefit. + +## Claude's one-liners + +```bash +# Bring the app up (builds wasm, starts server) +make ui-dev + +# Run an ad-hoc probe +cd tools/ui && npx playwright test ... + +# Everything in one shot (build + serve + smoke + stop) +make ui-smoke + +# Bring it down +make ui-stop +``` + +## Writing a driver script + +```ts +import { test, expect } from '@playwright/test'; +import { ClideDriver } from '../driver'; + +test('opens the theme picker and selects summer-night', async ({ page }) => { + const clide = new ClideDriver(page); + await clide.goto('/'); + + // Keyboard shortcut that invokes `theme.pick`: + await page.keyboard.press('Control+K'); + + // Pick via semantic label. + await clide.click('summer-night'); + + // Dump the whole tree to inspect state after an interaction. + const tree = await clide.dumpSemanticsTree(); + console.log(JSON.stringify(tree, null, 2)); + + // Screenshot into out/ (Claude reads the PNG via the Read tool). + await clide.screenshot('out/theme-picker.png'); +}); +``` + +## Prerequisites (one-time per machine) + +```bash +cd tools/ui +npm install +npx playwright install chromium +``` + +## Known quirks + +- **Labels are merged.** Flutter web concatenates sibling Semantics + labels into a single `aria-label` separated by newlines. `byLabel` + uses substring match for that reason. When two Semantics nodes share + a substring, narrow with `.filter()` on the returned locator. +- **The placeholder button.** Flutter web ships semantics disabled by + default, behind an invisible `` button. + `ClideDriver.waitUntilReady()` clicks it automatically. +- **No daemon on web.** The WASM build has no unix socket; the status + indicator always says `disconnected`. That's honest — a web-hosted + clide has no local daemon to talk to. The Playwright flow is for + UI-only verification. + +## When Playwright is overkill + +If all I need is "does the extension register its contributions," a +widget test (`test/builtin/*/widget_test.dart`) is faster and cheaper. +Reach for Playwright when the question is about the real rendering +pipeline, keyboard behavior, or multi-widget interactions that mirror +a user workflow.