document test layers and Claude's UI workflow
Three docs under docs/testing/:
* README.md — layer-by-layer reference (what each covers, the
runner, how to invoke, local vs CI flow). Points at the load-
bearing startup gate and the client-side-only constraint so a
macOS clone runs the same suite as Linux.
* a11y-manual.md — 15-minute manual checklist run at every tier
cut. Orca on Linux, VoiceOver on macOS. Catches prose drift
automation can't judge.
* claude-ui-workflow.md — how Claude Code drives the WASM build
through Playwright, including the `flt-semantics-placeholder`
quirk (semantics are opt-in in Flutter web; the driver
auto-clicks the placeholder to enable them).
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Claude
@@ -30,6 +30,7 @@ heading, and (b) bumping `project.yaml` `version:` in the same commit.
|
||||
|
||||
### Added
|
||||
|
||||
- Testing docs under `docs/testing/`: `README.md` (what each layer covers, how to run, local vs CI flow), `a11y-manual.md` (15-minute Orca + VoiceOver checklist run at every tier cut), `claude-ui-workflow.md` (how Claude Code drives the app through the Playwright harness, including the `flt-semantics-placeholder` quirk).
|
||||
- Makefile targets for every test layer and the UI harness: `test`, `test-a11y`, `test-integration`, `test-e2e`, `test-all`, `coverage`, `smoke-bundle`, `ui-dev`, `ui-stop`, `ui-smoke`. `push-check` now runs `test + test-a11y` (fast pre-push gate, <90s).
|
||||
- Per-layer CI shell scripts under `ci/`: `test.sh` (analyze + format + unit + widget + golden, ~5s), `test_a11y.sh` (a11y contract), `test_integration.sh` (integration_test one file at a time — desktop can't batch them reliably), `test_e2e.sh` (daemon subprocess + browser WASM Playwright smoke), `smoke_bundle.sh` (xvfb-run the Linux release bundle for 5s; catches dynamic-linker / asset-bundle / plugin-init regressions that widget tests can't see), `coverage.sh` (flutter test --coverage + lcov summary).
|
||||
- `.gitea/workflows/test.yml` — four-job pipeline (`unit`, `integration`, `startup-bundle`, `e2e`) that shells out to the `ci/*.sh` scripts. **Not activated yet** — Gitea Actions has to be enabled in the instance settings first. GitHub-Actions-syntax-compatible, so copying to `.github/workflows/` is a one-file move when the repo migrates.
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
# Testing clide
|
||||
|
||||
Every layer of the test pyramid is wired. The suite is intentionally
|
||||
designed to run client-side only — `git clone && make push-check` works
|
||||
on any Linux or macOS dev box without network access or shared state.
|
||||
|
||||
## Layers
|
||||
|
||||
| layer | location | runner | time | when |
|
||||
|---|---|---|---|---|
|
||||
| unit (root) | `test/` | `dart test` | ~5s | `make test` |
|
||||
| unit + widget + golden (app) | `app/test/` | `flutter test` | ~30s | `make test` |
|
||||
| a11y contract | `app/test/a11y/` | `flutter test` | ~5s | `make test-a11y` |
|
||||
| integration (startup gate) | `app/integration_test/` | `flutter test integration_test/` | ~60s | `make test-integration` |
|
||||
| daemon E2E + web WASM smoke | `test/daemon/` + `tools/ui/tests/` | `dart test` + Playwright | ~60s | `make test-e2e` |
|
||||
| startup bundle smoke | `ci/smoke_bundle.sh` | xvfb-run, 5s timeout | ~30s | `make smoke-bundle` |
|
||||
|
||||
## Dev loop
|
||||
|
||||
```bash
|
||||
make push-check # gate before `git push` (~90s)
|
||||
make test-all # full pyramid (~3-5min)
|
||||
```
|
||||
|
||||
## A11y + i18n
|
||||
|
||||
A11y is a day-one contract, not a Tier-6 polish. Every interactive widget
|
||||
emits a `Semantics` node with label + hint. The a11y suite enforces:
|
||||
|
||||
- **semantic_coverage_test.dart** — every built-in declares title + version
|
||||
and every interactive contribution carries the fields needed to build a
|
||||
Semantics node.
|
||||
- **contrast_test.dart** — every bundled theme meets WCAG-AA on the token
|
||||
pairs in `lib/kernel/src/theme/contrast.dart`. Catches token-pair
|
||||
regressions (foreground-on-background, tab.active-on-bg, etc.).
|
||||
- **i18n_coverage_test.dart** — every i18n key referenced by Tier-0
|
||||
built-ins exists in its en_US catalog. Asserts key presence directly
|
||||
(not "returned value == key" — that's ambiguous for keys that happen
|
||||
to equal their English translation).
|
||||
- **keyboard_traversal_test.dart** — interactive widgets are focusable
|
||||
and expose tap actions to a11y.
|
||||
|
||||
See [`a11y-manual.md`](a11y-manual.md) for the 15-minute manual
|
||||
screen-reader checklist run at every tier cut.
|
||||
|
||||
## Interacting with the app (Claude + humans)
|
||||
|
||||
Claude Code drives the Flutter WASM build through a Playwright harness
|
||||
— not via screenshots. Semantics are the contract. See
|
||||
[`claude-ui-workflow.md`](claude-ui-workflow.md) for the loop.
|
||||
|
||||
## CI
|
||||
|
||||
`.gitea/workflows/test.yml` is ready to run but Gitea Actions is not
|
||||
enabled on the instance yet. When the user flips it, four jobs kick off
|
||||
per push: `unit`, `integration`, `startup-bundle`, `e2e`. The workflow
|
||||
file is GitHub-Actions compatible — copying it to `.github/workflows/`
|
||||
is the entire migration if the repo moves to GitHub.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Manual a11y checklist — run at every tier cut
|
||||
|
||||
Automated a11y tests catch structural regressions (missing Semantics,
|
||||
broken contrast, missing i18n keys). They don't catch "this label is
|
||||
technically correct but reads like robot noise." A 15-minute manual
|
||||
pass at every tier cut does.
|
||||
|
||||
## Linux — Orca
|
||||
|
||||
1. Start Orca: `orca -s` (settings), toggle on "Focus follows mouse".
|
||||
2. Launch clide: `flutter run -d linux` or the built bundle.
|
||||
3. Tab through every slot in order. Every stop should read
|
||||
- panel name, then
|
||||
- a one-line verb describing what the widget does.
|
||||
4. Open the command palette (`ctrl+k` once it's wired). Read each
|
||||
command out loud. Flag commands whose name alone doesn't make
|
||||
it obvious what they do.
|
||||
5. Open the theme picker. Orca should announce "Select theme", then
|
||||
read each row as "theme name, activate this theme". Selecting
|
||||
announces the new theme.
|
||||
6. Disconnect the daemon mid-session. Orca should announce the
|
||||
statusbar change (it's a live region).
|
||||
|
||||
## macOS — VoiceOver
|
||||
|
||||
VoiceOver: `cmd+F5`. Same checklist as Orca. Extra pass:
|
||||
- Rotor navigation (`ctrl+opt+u`) should list every Semantic
|
||||
landmark; no "unknown group" entries.
|
||||
- Per-element hints read cleanly when pressing `ctrl+opt+shift+h`.
|
||||
|
||||
## Report a drift
|
||||
|
||||
Any label that feels off — noise, tech jargon, abbreviations screen
|
||||
readers mispronounce — becomes a follow-up ticket. Do not "fix" in
|
||||
the branch being cut; queue it so each tier stays focused. File
|
||||
under `docs/todo-a11y.md` (create if absent) with:
|
||||
|
||||
```
|
||||
- [ ] ext.<id>: <key-or-widget> — <what's wrong>, <suggested wording>
|
||||
```
|
||||
|
||||
## Why this isn't automated
|
||||
|
||||
Automation catches structural violations. Automation can't judge prose.
|
||||
A 15-minute human pass per tier stays cheap and catches the gap.
|
||||
Revisit automation (axe-core via Playwright, Flutter a11y harnesses)
|
||||
if we ship a public build.
|
||||
@@ -0,0 +1,86 @@
|
||||
# How Claude Code uses clide
|
||||
|
||||
Claude Code needs to *actually use* the app while building features.
|
||||
The pipeline:
|
||||
|
||||
1. **Flutter builds the app to WASM.** `flutter build web --wasm` ships
|
||||
a CanvasKit/Skwasm bundle under `app/build/web/`.
|
||||
2. **A local server serves it.** `tools/ui/serve.sh` starts
|
||||
`http://localhost:4280` in the background with a pidfile.
|
||||
3. **Playwright drives a headless Chromium.** Instead of click-by-pixel
|
||||
— which is brittle on a CanvasKit `<canvas>` — the driver queries
|
||||
Flutter's semantic tree (`flt-semantics[aria-label=…]`), the same
|
||||
tree screen readers use. This is only reliable because every
|
||||
interactive widget in clide already ships `Semantics(label:, hint:)`
|
||||
wrappers for a11y — the automation surface is a free side-benefit.
|
||||
|
||||
## Claude's one-liners
|
||||
|
||||
```bash
|
||||
# Bring the app up (builds wasm, starts server)
|
||||
make ui-dev
|
||||
|
||||
# Run an ad-hoc probe
|
||||
cd tools/ui && npx playwright test ...
|
||||
|
||||
# Everything in one shot (build + serve + smoke + stop)
|
||||
make ui-smoke
|
||||
|
||||
# Bring it down
|
||||
make ui-stop
|
||||
```
|
||||
|
||||
## Writing a driver script
|
||||
|
||||
```ts
|
||||
import { test, expect } from '@playwright/test';
|
||||
import { ClideDriver } from '../driver';
|
||||
|
||||
test('opens the theme picker and selects summer-night', async ({ page }) => {
|
||||
const clide = new ClideDriver(page);
|
||||
await clide.goto('/');
|
||||
|
||||
// Keyboard shortcut that invokes `theme.pick`:
|
||||
await page.keyboard.press('Control+K');
|
||||
|
||||
// Pick via semantic label.
|
||||
await clide.click('summer-night');
|
||||
|
||||
// Dump the whole tree to inspect state after an interaction.
|
||||
const tree = await clide.dumpSemanticsTree();
|
||||
console.log(JSON.stringify(tree, null, 2));
|
||||
|
||||
// Screenshot into out/ (Claude reads the PNG via the Read tool).
|
||||
await clide.screenshot('out/theme-picker.png');
|
||||
});
|
||||
```
|
||||
|
||||
## Prerequisites (one-time per machine)
|
||||
|
||||
```bash
|
||||
cd tools/ui
|
||||
npm install
|
||||
npx playwright install chromium
|
||||
```
|
||||
|
||||
## Known quirks
|
||||
|
||||
- **Labels are merged.** Flutter web concatenates sibling Semantics
|
||||
labels into a single `aria-label` separated by newlines. `byLabel`
|
||||
uses substring match for that reason. When two Semantics nodes share
|
||||
a substring, narrow with `.filter()` on the returned locator.
|
||||
- **The placeholder button.** Flutter web ships semantics disabled by
|
||||
default, behind an invisible `<flt-semantics-placeholder>` button.
|
||||
`ClideDriver.waitUntilReady()` clicks it automatically.
|
||||
- **No daemon on web.** The WASM build has no unix socket; the status
|
||||
indicator always says `disconnected`. That's honest — a web-hosted
|
||||
clide has no local daemon to talk to. The Playwright flow is for
|
||||
UI-only verification.
|
||||
|
||||
## When Playwright is overkill
|
||||
|
||||
If all I need is "does the extension register its contributions," a
|
||||
widget test (`test/builtin/*/widget_test.dart`) is faster and cheaper.
|
||||
Reach for Playwright when the question is about the real rendering
|
||||
pipeline, keyboard behavior, or multi-widget interactions that mirror
|
||||
a user workflow.
|
||||
Reference in New Issue
Block a user