odysseus/tests/WAVE6_TEST_PERFORMANCE_REPORT.md

Wave 6 test performance and worker isolation

Starting branch: wave6/test-performance-foundation. Starting HEAD: bfa5ebb379713790bd1b47c75237fb1fbe989375. Starting tree: 3aa232c1c7d479d7ed48e22aa59ddfe8f53058ad.

The interrupted working tree was inspected before edits. It contained changes in pyproject.toml, tests/conftest.py, tests/test_research_report_read.py, and tests/test_stt_leak.py, plus untracked requirements-dev.txt, tests/helpers/worker_runtime.py, and tests/test_worker_runtime.py. All seven files belonged to this lane. No working-tree state was discarded.

The interrupted implementation established private filesystem defaults before application imports, isolated saved research reports and STT temporary-file observations, declared optional xdist tooling, and guarded externally launched smoke tests. The temporary-directory context manager, restoration tests, subprocess inheritance test, report fixture, and STT fixture were coherent. The root integration and refusal diagnostics needed additional validation and correction. None of the recovered files was rejected or replaced wholesale.

The final root wiring preserves the existing generated configuration-reference locations and does not add test fixture environment reads to the public configuration inventory. The first serial profile exposed two stale-reference failures caused by the interrupted wiring. Both are fixed within test code; website/configuration-reference.md and its generator remain unchanged.

The live-smoke guard now runs after marker/shard deselection, before xdist publishes runnable items. It emits a normal collection failure. Raising a worker UsageError after notification had produced an xdist internal error and lost the useful refusal message; the diagnostic probe exposed this and the final probe proves the intended failure is visible. No smoke request executes in that refused invocation.

Resource ownership

tests/helpers/worker_runtime.py owns an atomically created temporary root for each pytest process. The worker label is diagnostic, not an allocation key. The random suffix prevents two independent invocations of gw0 from adopting each other's files. Paths below that root are stable: data, mail, fastembed, runtime, tmp, and the controller's pytest basetemp. This is deterministic ownership, not a fixed reusable path. Ordinary serial runs use the same architecture with label main; xdist knowledge stays in the helper and root hooks.

Resource	Isolation boundary
DATABASE_URL and collection engine	Existing foundation forces sqlite:///:memory: before import in each process; inherited developer URLs are never opened.
File-backed SQLite	Existing fixtures own temporary engines/files, restore bindings without replacing ORM classes, dispose engines before deletion.
Postgres	Audited tests use mock dialects/DDL engines; this lane does not connect workers to a shared Postgres database.
Runtime data, jobs, publications, process/browser registries	Existing constants derive from private ODYSSEUS_DATA_DIR before collection; narrower fixtures still patch their owned roots.
Effects/provenance/evidence	In-memory structures are process-local; file-backed resource publications live under private data/process/browser roots or explicit fixture workspaces.
Attachments and embedding cache	Dedicated environment overrides point inside the owned root.
Search/cache/generated artifacts	Data-derived caches use the owned data root; explicit artifact fixtures use tmp_path. Installed source/browser assets are read together.
Temporary files and shell logs	TMPDIR/TMP/TEMP and cached tempfile.tempdir point into the owned root and are inherited by subprocesses. pytest's basetemp is the controller root's pytest directory, with distinct xdist popen-gw<n> worker basetemps beneath it.
Browser profiles and Unix sockets	Private runtime/data roots; existing short AF_UNIX socket helper owns atomic /tmp directories rather than long pytest paths.
Static HTTP servers	Each process holds its kernel-assigned loopback socket; fixed external pins are rejected for parallel execution. Connections get their own threads, so a silent client cannot stall others.
Chroma refusal probes	Bound non-listening sockets reserve closed ports until teardown, replacing bind-and-release guesses.
Launcher refusal probe	An owned ephemeral listener exercises foreign-server refusal without a host-dependent derived-port skip. Port derivation retains its separate assertions.
Process tests	Sleeper fixture owns process sessions/groups, signals the group before reaping its leader, and propagates cleanup errors; it previously killed only the shell and swallowed errors.
Module globals/environment	Process-local under xdist; foundation import-state/database restoration guards and test monkeypatches remain intact.
Pytest advisory cache	Measurements disable cacheprovider. Built-in worker last-failed/node-ID writers are suppressed by pytest; concurrent controllers should not share advisory metadata. No suite test uses the cache fixture.
Live smoke application	Explicit APP_PORT opts into external application ownership; selected live smoke tests require -n 0. Existing no-application skips remain visible.

Normal cleanup restores the caller's environment/tempfile cache and removes only the root owned by that context. Cleanup exceptions are not suppressed. A hard-killed standalone process can leave its root behind; subsequent runs allocate a different root, so abandoned state is not reused. This is isolation against crash residue, not a claim that arbitrary SIGKILL removes every file or process. No global sweeper, kill-by-name, PID-derived namespace, or guessed per-worker port was added.

The audit found actual filesystem/port/process fixture boundaries to repair, not a need to serialize ordinary unit tests. Process-local globals alone are not cross-worker shared state. A delegated broad audit was partial and offered speculative concerns; those were checked locally rather than accepted as proven defects.

Dependencies and release modes

Foundation requirements declared pytest and pytest-asyncio, but not xdist. The recovered development declaration is retained: requirements-dev.txt includes requirements.txt and pytest-xdist[psutil]>=3.8,<4. Runtime dependency requirements remain unchanged. Installed tooling: pytest 9.1.1, xdist 3.8.0, psutil 7.2.2, Python 3.11. Physical cores: 8; logical CPUs: 16. The psutil extra makes auto choose physical cores on this host.

Parallelism stays opt-in. The full serial release oracle and four deterministic CI shards remain unchanged. The two strict negative-web production xfails remain xfails: memory-only turns are still incorrectly short-circuited with "Web access is disabled." No production/runtime file, Wave 4 behavior, CI workflow, or production contract was repaired in this lane.

Takeover after the interrupted session

Implementation commits a94fc54c and 915e6ed1 were inherited at takeover HEAD 915e6ed12571b0c9036a3546c334c3561443a310, tree 6a646603ace8e40e1497212c85ef47b57443f043. The uncommitted README section and this report were retained and completed. The inherited commits were audited against their diffs and kept unchanged.

Two full-suite -n 2 runs on that HEAD each failed one test. Neither failure was a product defect, and both are fixed in test infrastructure:

1. test_legacy_cleanup_against_a_private_real_tmux_server failed deterministically under any worker count with File name too long. This lane had moved pytest's default basetemp beneath the private TMPDIR, which gave /tmp/ody-main-XXXXXXXX/tmp/pytest-of-<user>/pytest-0/popen-gw0/<test>/tmux.sock, 110 bytes against Linux's 107-byte AF_UNIX limit. The same path was 87 bytes before this lane and 100 bytes in a serial run, which is why the serial profile passed. The controller now sets basetemp to the private root's pytest directory, and xdist hands each worker popen-gw<n> beneath it. That path is now 81 bytes regardless of the username length. An explicit --basetemp is still honoured. test_worker_runtime.py binds an AF_UNIX socket at the same path budget without needing tmux, and fails under -n 2 without the fix.
2. With the basetemp fix in place, the next -n 2 run had a single different failure. test_reordering_two_conflicting_declarations_moves_the_digest ran 31.7 s instead of about 5.3 s, and node crashed with route.fetch: Request context disposed. That message hides the real failure, a 30 s page.goto stall: the capture's finally closed the browser while a stylesheet route.fetch was still pending. The shared static server was a single-threaded socketserver.TCPServer. Chromium sometimes opens a speculative connection that never sends a request, and every queued request then waited behind it. A server-instrumented capture loop under full CPU saturation reproduced it: 4 of 12 captures had a request-less connection holding the server for about 28.9 s, and those captures failed. With a threaded server, 12 of 12 captures passed. The idle connection still appeared in 5 of them but lived 1.0 to 1.5 s without blocking anything. The fixture now uses ThreadingTCPServer with daemon handler threads. test_static_server_serves_this_worktree holds a silent connection open while it fetches; against the serial server, that request times out after 5 s. This defect predates the lane and depends on load, which parallel execution increases. Assertions, timeouts, and the capture harness are unchanged.

Both fixes keep the existing first-location lines that website/configuration-reference.md records, and that page is unchanged. The abandoned root of the first -n 2 run, killed by its monitor, was /tmp/ody-main-x7hmcf4_ (261 MB, last written before takeover). No process held it, and it was removed. No later run left a root behind.

Measurements

Host: Python 3.11.15, pytest 9.1.1, pytest-xdist 3.8.0, psutil 7.2.2, 8 physical cores, 16 logical CPUs, 29 GiB RAM. Every run collects the same full suite. The command is python -m pytest -q -p no:cacheprovider --durations=100 -rsx --max-worker-restart=0 -o faulthandler_timeout=60 -n <N>. The serial performance-lane profile predates the last option pair. The measurement wrapper records wall time and descendant RSS. After each run it checks for surviving descendants (pid plus create time), new TCP listeners, mutated files under data/ and the inherited data directory, and leftover /tmp/ody-main-* runtime roots.

Run	Code	Workers	Wall s	Passed	Failed	Skipped	XFail	Subtests	Peak RSS
Isolation foundation oracle	bfa5ebb3	0	—	12,556	0	59	2	6	—
Lane serial profile	pre-a94fc54c wiring	0	506.4	12,558	2*	59	2	6	—
full-n2	915e6ed1	2	n/a†	—	1 (F marker)	—	—	—	—
full-n2-measured	915e6ed1	2	262.1	12,559	1 (tmux)	59	2	6	5.5 GB
full-n2-r2	+ basetemp fix	2	271.1	12,560	1 (CSS stall)	59	2	6	6.3 GB
full-n2-r3	final	2	267.6	12,561	0	59	2	6	5.7 GB
full-n4-r1	final	4	155.7	12,561	0	59	2	6	9.1 GB
full-n4-r2	final	4	155.5	12,561	0	59	2	6	8.7 GB
serial-final	final	0	496.3	12,561	0	59	2	6	3.9 GB

* Stale configuration-reference failures from the interrupted wiring, fixed before a94fc54c. † The monitor crashed on a hardened Chromium process (AccessDenied) after pytest reached 100%, so no summary was retained. Its single progress-line F matches the deterministic tmux failure.

Every run with the final code had no worker crashes, no hangs, no surviving test descendants, no new listeners, no persistent-state mutations, and no remaining runtime roots. Passed counts increase only with the lane's added tests. The skips are the same 59: 16 smoke tests without APP_PORT plus opt-in live and platform gates. The two xfails are the known strict negative-web production defects.

Monitoring gaps: the wrapper could not read /proc/<pid>/environ for two to four short-lived python descendants per run (AccessDenied). That only limits worker-ID observation. Leak detection uses process identity, not the environment, and reported nothing. Hardened Chromium processes are tolerated as unreadable instead of aborting the measurement. No unrelated process was signalled.

Scaling and recommendation

Compared with the 496.3 s final serial oracle, two workers are 1.85x faster (267.6 s) and four workers are 3.19x faster (155.6 s mean). Going from two to four workers is 1.72x faster. Each worker collects 12,622 items in about 11 s (0.9 GB RSS) before running anything. The longest single test is the computed-style capture at about 23 s. The auth-concurrency, rich-document browser, media, and generated-reference groups keep their real work.

Recommended local configuration: -n 4. Both full repeats were green and within 0.3 s of each other, at about 2.2 GB RSS per worker.

-n auto (8 workers here) was not run. Projected from the measured per-worker peak, it needs about 18 GB, against about 21 GB available on this desktop host while its browser and Chroma services run. Fixed collection cost and the 23 s longest test bound the best case to roughly 85 to 100 s. The CSS stall above also showed that browser timing is sensitive to load. That gain does not justify the risk of swapping or OOM, or of timing distortion. Evaluate six workers on a host with more headroom before considering -n auto.

The full serial run remains the release oracle and the CI shards are unchanged. Live smoke tests under APP_PORT are the only serial-only tests. They share one external application, its accounts, and its endpoints, and are refused under xdist.

No new product defect was found. A follow-up for test tooling: when a route.fetch handler is still pending, tests/css_snapshot/capture.mjs can replace a navigation error with Request context disposed. Fixing that would make future capture failures easier to diagnose; it is not needed for correctness here.