jpmschweitzer/odysseus
1
0
Fork 0
mirror of https://github.com/pewdiepie-archdaemon/odysseus.git synced 2026-10-11 09:22:21 +02:00
Code Issues Packages Projects Releases Wiki Activity

test(smoke): add a release smoke suite over every advertised feature area

Browse Source 
The decomposition lanes have two safety nets and neither covers the
product. The checkpoint benchmark measures the agent runtime; the
computed-style snapshot pins the CSS. Nothing checked that Notes,
Calendar, Documents, Email, Memory, Cookbook or Settings still worked
after a route package moved or a 17,000-line module was split - and the
unit suite does not, since a byte-identical file move can break tests
that pass on the base branch with CI green throughout. The 28 Playwright
specs we do have are all under tests/e2e/photo-editor/ and no workflow
runs them.

scripts/odysseus-smoke boots this worktree through `odysseus dev` and
runs tests/smoke/: one scenario per area, each asserting a user-visible
outcome rather than a status code. Models come from a deterministic
OpenAI-compatible stub on an ephemeral loopback port; email reuses the
existing ODYSSEUS_EMAIL_FIXTURE path rather than inventing a second
mechanism. No scenario touches a live endpoint or the network.

The report is a per-area table that prints the areas the suite does not
cover next to the ones it does, and builds its rows from the registry
rather than from what happened to run, so an area cannot go missing by
having its module deleted or renamed. Under a plain pytest with nothing
booted every scenario skips with the reason, so the full suite stays
green.
This commit is contained in:
Léo
2026-09-30 12:12:58 +02:00
parent 6105702901
commit 12f74ec9ea
19 changed files with 1689 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/README.md
+25
View File
@@ -156,6 +156,31 @@ The inventory, the baseline and the capture live in `tests/css_snapshot/`;
not, and how to find the property that moved when it fails. The run takes about
21 seconds and skips when `npm ci` has not been run.
## Release smoke suite
`tests/smoke/` drives every advertised feature area once, end to end,
against a real instance - the safety net the unit suite does not provide
for a route move or a module split. One command boots the worktree and
runs it:
```bash
scripts/odysseus-smoke # boot, run every area, stop again
scripts/odysseus-smoke --keep-up # leave the instance running
scripts/odysseus-smoke --areas # the coverage table, without booting
```
It reads its target instance out of the environment (`APP_PORT` through
`internal_api_base()`, plus the dev admin account), so under a plain
`pytest` with nothing booted every scenario skips with the reason and
the full suite stays green. Models are served by a deterministic
loopback stub, never a live endpoint; email uses the repo's existing
`ODYSSEUS_EMAIL_FIXTURE` path.
The report is a per-area table that also prints the areas the suite
deliberately does not cover, so it cannot be read as coverage of
everything it omits. `tests/smoke/README.md` documents what is in each
list and why.
## Core principles
- Keep PRs small and homogeneous: one kind of change per PR.