odysseus/tests/WAVE6_TEST_ISOLATION_REPORT.md

Wave 6 test-isolation foundation

Provenance and boundary

• Worktree: /home/alteixeira20/Projects/Odysseus/odysseus-wave6-test-isolation.
• Starting branch: wave6/test-isolation-foundation.
• Starting HEAD: b1666951faf8285054e1ca90f11533b0fb53fb57.
• Starting TREE: a3dee24e3c56c22e1d3f9805f409bf16b086c3d4.
• git status --short --branch: ## wave6/test-isolation-foundation only.
• git status --porcelain: empty. Starting worktree clean.
• No rebase, merge, push, PR edit, or access to the Wave 4 worktree.
• All changes are test code, test helpers, or this test validation report. No production code changed; no STOP-level exception was needed.

Contamination map

Source	Mutation / binding	Previous cleanup	Escaping state / effect
Both tests in tests/test_truncate_message_count_regression.py, through _make_manager	Creates an unowned mkstemp DB; directly sets os.environ["DATABASE_URL"]; reloads core.database then core.session_manager	Only the assertion's ORM session is closed	Environment, engine, factory, declarative Base, ORM classes, manager globals, and registered engine listeners survive; database files are never deleted
Collection of tests/test_manage_tasks_owner_scope.py	Creates a module-lifetime temporary DB and assigns _TS permanently to core.database.SessionLocal	Some individual rows deleted; ORM sessions closed	The collection-time factory assignment leaks to unrelated tests; truncation reload replaces it, so production handlers and _seed / _get use different databases
test_checkin_digest_owner_scope.py, test_gallery_owner_filter_single_user.py, test_document_session_owner_scope.py	Create module-lifetime DB files / engines and retain local factories	Some rows deleted / route factory restored / ORM sessions closed	Owned files and engines lack teardown; retained imported ORM identities can disagree with reloaded module globals
tests/conftest.py bootstrap	DATABASE_URL was set only if absent before importing core.database	No environment restoration	An inherited developer URL could be opened and initialized during collection

importlib.reload updates the existing module dictionary in place; it does not change the sys.modules module object's identity. Previously imported symbols nevertheless keep their old class/factory identities. In particular, src.database re-exports old objects, while reloaded core.session_manager rebinds its DbSession, DbChatMessage, DbDocument, SessionLocal, and SessionManager. Old functions still resolve globals through those changed module dictionaries. Each database reload also registers a new SQLite connect listener on SQLAlchemy's Engine class.

The current failures are principally in test_manage_tasks_owner_scope.py: _seed writes through its retained _TS, but do_manage_tasks imports the now-replaced factory from core.database at call time. Ownerless / cross-owner mutations incorrectly observe "task not found"; successful mutation/list/create witnesses likewise read or write the wrong test database. Four invalid-date cases still pass, so the observed contamination count is 16 rather than all 20 task-owner cases.

src.settings has TTL caches, but those caches do not resolve DATABASE_URL. The contaminating helper neither patches nor reloads settings. The database URL is read directly from the environment at database import/reload time; settings cache invalidation is not the correction for this leak.

Baseline reproduction before editing

All baseline commands explicitly selected in-memory bootstrap storage. Python with installed pytest/SQLAlchemy was /tmp/pr40-final-validation-venv/bin/python (pytest 9.1.1, SQLAlchemy 2.1.1).

DATABASE_URL=sqlite:///:memory: /tmp/pr40-final-validation-venv/bin/python -m pytest -q -p no:cacheprovider tests/test_manage_tasks_owner_scope.py tests/test_checkin_digest_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_document_session_owner_scope.py tests/test_truncate_message_count_regression.py
# 30 passed, 9 warnings in 2.48s

DATABASE_URL=sqlite:///:memory: /tmp/pr40-final-validation-venv/bin/python -m pytest -q -p no:cacheprovider tests/test_truncate_message_count_regression.py tests/test_manage_tasks_owner_scope.py tests/test_checkin_digest_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_document_session_owner_scope.py
# 16 failed, 14 passed, 9 warnings in 2.40s

DATABASE_URL=sqlite:///:memory: /tmp/pr40-final-validation-venv/bin/python -m pytest -q -p no:cacheprovider -rx --runxfail tests/test_runtime_behavior_regressions.py -k negative_web
# 2 failed (IndexError), 1 passed, 13 deselected, 1 warning in 1.61s

The same 30 database tests shuffled with random.Random(42) before editing reported 9 failed, 21 passed, 9 warnings in 2.37s. Exact command:

/tmp/pr40-final-validation-venv/bin/python /tmp/odysseus-wave6-isolation-evidence/run_order.py seed:42 before-seed42 tests/test_manage_tasks_owner_scope.py tests/test_checkin_digest_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_document_session_owner_scope.py tests/test_truncate_message_count_regression.py

Isolation correction and regression evidence

• tests/helpers/database.py owns a fresh TemporaryDirectory, file-backed SQLite engine with NullPool, real schema, and factory for each invocation. Teardown disposes the engine and removes the directory including sidecars.
• The truncation fixture temporarily patches only resource bindings in core.database, src.database, and core.session_manager, with a matching temporary environment URL. Patches are undone before resource disposal. No module reload, module eviction, new ORM class, or listener registration.
• Four owner modules now own a disposable database per test rather than at collection. Only the task-owner module patches the canonical factory; that patch is scoped to the test and restored during teardown. The other modules retain their existing handler/route wiring and ownership assertions.
• The existing pytest bootstrap always chooses in-memory storage before production database imports and registers environment restoration with the pytest configuration lifecycle. No new process-wide autouse cleanup fixture.
• tests/test_database_test_isolation.py checks real persistent truncation, exact class/factory restoration, file removal, clean subsequent database state, and exception teardown. Child pytest runs execute the actual original witness entry points in both directions and check process-visible bindings after collection and after every test. An inherited developer-DB sentinel remains byte-identical, with no database sidecars created, and the inherited environment URL is restored after pytest exits.

Mutation checks temporarily restored one original file at a time, ran test_actual_tests_restore_process_state_and_ignore_inherited_database[True], and restored the corrected file in finally. Both mutations made the new regression fail (pytest exit 1): original truncation was caught as "test leaked database or module state", original task-owner collection as "collection changed database bindings". No mutation was retained.

Historical negative-web witnesses and production blocker

Both parameter cases in test_negative_web_wording_withholds_the_web_tools_unhandled intended to prove that inferred negative-web wording can receive a memory-only answer without web_search or web_fetch being offered to the model:

1. Summarise what you already know. Do not search the web.
2. No web search please, just tell me what you know about Python decorators.

Both previously indexed an empty fake-model call list and xfailed on unrelated IndexError. The fake also emitted provider-shaped choices JSON at a seam that now consumes normalized {"delta": ...} SSE.

The repaired witnesses drive the unmodified public stream_agent_loop, mock only model output and existing infrastructure edges, capture the actual SSE, and require one model call, withheld web schemas, and delivered fake answer. The existing passing "don't search online" control and positive search control remain. The fake now emits normalized SSE.

The two historical xfails remain strict, with raises=_MemoryOnlyTurnBlocked. That exception is emitted only when the real loop makes zero model calls and returns the exact known disabled-web response. IndexError, arbitrary assertions, or other exceptions no longer qualify.

This is a genuine production defect outside the lane: negative-web detection is true for both phrasings, but _classify_agent_request returns {'web'} and _web_search_unavailable_for_turn returns true after the negative has disabled web tools. The loop short-circuits the requested memory-only response with:

Web access is disabled for this turn. Enable web search and resend the request.

The passing "Answer from memory only, don't search online." control has {'memory', 'web'} intent domains and does not short-circuit. A minimized reproduction using those three production functions requires neither a model nor a mock. Relevant production locations (intentionally untouched): src/agent_loop.py negative detector near line 864, classifier near 10371, negative-tool disabling near 20735, unavailable calculation near 22233, and short-circuit response near 25101.

/tmp/pr40-final-validation-venv/bin/python -m pytest -q -p no:cacheprovider --runxfail tests/test_runtime_behavior_regressions.py -k negative_web_wording_withholds_the_web_tools_unhandled
# 2 failed with _MemoryOnlyTurnBlocked, 14 deselected, 1 warning in 0.75s

No production repair was attempted. Runtime intent/availability behavior, authority, completion/evidence, and Wave 4 effects files remain untouched. This blocker prevents a READY release-foundation verdict despite passing test isolation validation.

Final order validation

The focused manifest has 50 cases: the five baseline database modules plus test_database_test_isolation.py and test_runtime_behavior_regressions.py. Normal collection order is alphabetical by the seven explicitly selected files. Reversed and seeded runs reorder individual collected test cases, including parameterized cases. Both contamination directions also vary file collection order. Every execution manifest is retained as JSON in the evidence directory.

Run	Result
Normal	48 passed, 2 xfailed, 0 skipped, 11 warnings in 9.01s
Reverse of all collected cases	48 passed, 2 xfailed, 0 skipped, 11 warnings in 8.47s
Truncation module first, owner witnesses afterward	48 passed, 2 xfailed, 0 skipped, 11 warnings in 8.00s
All four owner modules first, truncation afterward	48 passed, 2 xfailed, 0 skipped, 11 warnings in 8.98s
Seed 1	48 passed, 2 xfailed, 0 skipped, 11 warnings in 9.01s
Seed 42	48 passed, 2 xfailed, 0 skipped, 11 warnings in 8.98s
Seed 2026	48 passed, 2 xfailed, 0 skipped, 11 warnings in 8.97s

Exact executed commands:

/tmp/pr40-final-validation-venv/bin/python /tmp/odysseus-wave6-isolation-evidence/run_order.py normal after-normal tests/test_checkin_digest_owner_scope.py tests/test_database_test_isolation.py tests/test_document_session_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_manage_tasks_owner_scope.py tests/test_runtime_behavior_regressions.py tests/test_truncate_message_count_regression.py
/tmp/pr40-final-validation-venv/bin/python /tmp/odysseus-wave6-isolation-evidence/run_order.py reverse after-reverse tests/test_checkin_digest_owner_scope.py tests/test_database_test_isolation.py tests/test_document_session_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_manage_tasks_owner_scope.py tests/test_runtime_behavior_regressions.py tests/test_truncate_message_count_regression.py
/tmp/pr40-final-validation-venv/bin/python /tmp/odysseus-wave6-isolation-evidence/run_order.py normal after-truncation-first tests/test_truncate_message_count_regression.py tests/test_checkin_digest_owner_scope.py tests/test_database_test_isolation.py tests/test_document_session_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_manage_tasks_owner_scope.py tests/test_runtime_behavior_regressions.py
/tmp/pr40-final-validation-venv/bin/python /tmp/odysseus-wave6-isolation-evidence/run_order.py normal after-owners-first tests/test_checkin_digest_owner_scope.py tests/test_document_session_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_manage_tasks_owner_scope.py tests/test_truncate_message_count_regression.py tests/test_database_test_isolation.py tests/test_runtime_behavior_regressions.py
/tmp/pr40-final-validation-venv/bin/python /tmp/odysseus-wave6-isolation-evidence/run_order.py seed:1 after-seed1 tests/test_checkin_digest_owner_scope.py tests/test_database_test_isolation.py tests/test_document_session_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_manage_tasks_owner_scope.py tests/test_runtime_behavior_regressions.py tests/test_truncate_message_count_regression.py
/tmp/pr40-final-validation-venv/bin/python /tmp/odysseus-wave6-isolation-evidence/run_order.py seed:42 after-seed42 tests/test_checkin_digest_owner_scope.py tests/test_database_test_isolation.py tests/test_document_session_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_manage_tasks_owner_scope.py tests/test_runtime_behavior_regressions.py tests/test_truncate_message_count_regression.py
/tmp/pr40-final-validation-venv/bin/python /tmp/odysseus-wave6-isolation-evidence/run_order.py seed:2026 after-seed2026 tests/test_checkin_digest_owner_scope.py tests/test_database_test_isolation.py tests/test_document_session_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_manage_tasks_owner_scope.py tests/test_runtime_behavior_regressions.py tests/test_truncate_message_count_regression.py

The order driver is test-session-local; it is not installed into conftest or production and never changes the normal suite's ordering. Its exact source (for recreating /tmp/odysseus-wave6-isolation-evidence/run_order.py) is:

import json
import os
from pathlib import Path
import random
import sys
import pytest

out = Path('/tmp/odysseus-wave6-isolation-evidence')
mode, label, *modules = sys.argv[1:]
class Order:
    @pytest.hookimpl(trylast=True)
    def pytest_collection_modifyitems(self, items):
        if mode == 'reverse':
            items.reverse()
        elif mode.startswith('seed:'):
            random.Random(int(mode.split(':')[1])).shuffle(items)
        (out / f'{label}-manifest.json').write_text(json.dumps([i.nodeid for i in items], indent=2) + '\n')
os.environ['DATABASE_URL'] = 'sqlite:///:memory:'
args = ['-q', '-p', 'no:cacheprovider', '-rsx', *modules]
(out / f'{label}-command.json').write_text(json.dumps({'argv':sys.argv,'pytest_args':args,'DATABASE_URL':os.environ['DATABASE_URL']},indent=2) + '\n')
raise SystemExit(pytest.main(args, plugins=[Order()]))

Adjacent and complete validation

The first complete run finished with 44 failed, 12509 passed, 62 skipped, 2 xfailed, 184 warnings and 6 passing subtests in 391.85s. Its failure inventory was 42 missing-Playwright-package failures and two generated configuration-reference freshness failures. The local worktree had not had npm ci run; the generator includes test-source line references, and the bootstrap edit moved the existing references by one line.

npm ci --ignore-scripts installed the four already-locked packages; package and lock files were unchanged. Cached Chromium launched successfully (version 151.0.7922.34). A bootstrap comment now explains why isolation must precede collection and preserves the existing referenced line locations, without editing the generated website or the generator. This follow-up is commit f67d0388fd805867a0920dbffd4da056b1361e38 (fix(tests): preserve bootstrap reference locations).

npm ci --ignore-scripts
# exit 0; 4 packages installed, lockfile and package policy unchanged

python scripts/generate_env_reference.py --check
# exit 0; website/configuration-reference.md is up to date (112 variables)

/tmp/pr40-final-validation-venv/bin/python -m pytest -q -p no:cacheprovider tests/test_document_module_api.py tests/test_env_reference.py
# 16 passed, 1 warning in 17.00s

The first full run and the 44-case focused rerun are retained separately from the final complete run. The 44-case rerun reported 1 failed, 43 passed in 73.61s: the existing font/color browser witness observed its font-size icon still showing 13 after its text had changed to size 5 (expected 24). An isolated reproduction of that unchanged test passed in 2.13s. This is an observed intermittent browser UI witness; no database failure was involved, and a production root cause has not been established. The test and editor implementation remain unchanged, with both observations retained rather than suppressing the failed run. No failure was hidden by skipping or deleting a test.

/tmp/pr40-final-validation-venv/bin/python -m pytest -q -p no:cacheprovider -rsx tests/test_session_manager.py tests/test_session_manager_cleanup.py tests/test_session_manager_persist_guard.py tests/test_session_mutation_lookup.py tests/test_session_discovery_message_count.py tests/test_history_order_by_timestamp_regression.py tests/test_history_display_model_hydration.py tests/test_session_search.py tests/test_session_search_batch_fetch.py tests/test_session_list_owner_scope.py tests/test_session_owner_attribution.py tests/test_session_endpoint_owner_scope.py tests/test_session_concurrent.py tests/test_app_db_permissions.py tests/test_database_utcnow.py tests/test_tool_policy.py tests/test_turn_contract.py tests/test_chat_preprocess_tool_policy.py tests/test_chat_route_tool_policy.py tests/test_resend_message_nondestructive.py
# 1262 passed, 0 skipped, 0 xfailed, 7 warnings in 40.56s

/tmp/pr40-final-validation-venv/bin/python -m pytest -q -p no:cacheprovider -rsx
# 12556 passed, 59 skipped, 2 xfailed, 184 warnings, 6 subtests passed in 468.49s (0:07:48); exit 0

python -m compileall -q app.py core routes src services scripts tests
# exit 0

git diff --check
# exit 0, no output

git diff --check b1666951faf8285054e1ca90f11533b0fb53fb57
# exit 0, no output

git diff --name-only --diff-filter=U
# exit 0, no output

rg -n '^(<<<<<<< |=======$|>>>>>>> )' <each modified file>
# exit 1, no matches

No test was deleted, no skip was added, and no unrelated xfail was added. The same two historical parameter cases remain xfailed for the demonstrated production blocker, now restricted to that actual contract failure.

The isolation commit is e2b847c84d27ce785565757e81ca5293896eaab3 (fix(tests): isolate database and module state). The final bounded commit contains the repaired web witnesses and this validation report. Final commit, HEAD, TREE and clean status are supplied in the accompanying final report to avoid placing a commit's own hash inside its committed contents.

Full logs, commands, and per-run manifests from this session are in /tmp/odysseus-wave6-isolation-evidence/.

The follow-up normal focused run after preserving bootstrap reference locations is recorded in after-bootstrap-followup.log. It reported 48 passed, 2 xfailed, 11 warnings in 9.55s. Its exact command is:

/tmp/pr40-final-validation-venv/bin/python /tmp/odysseus-wave6-isolation-evidence/run_order.py normal after-bootstrap-followup tests/test_checkin_digest_owner_scope.py tests/test_database_test_isolation.py tests/test_document_session_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_manage_tasks_owner_scope.py tests/test_runtime_behavior_regressions.py tests/test_truncate_message_count_regression.py

The additional browser observation came from the 44-case manifest retained in full-failures-manifest.json; every exact node ID in that diagnostic command is retained in full-failures-rerun-command.json. The isolated diagnostic command:

/tmp/pr40-final-validation-venv/bin/python -m pytest -q -p no:cacheprovider tests/test_document_rich_font_color_controls.py::test_numeric_font_size_and_custom_colors_work_on_desktop_and_mobile
# 1 passed, 1 warning in 2.13s

Independent review disposition

A read-only review of isolation commit e2b847c8 raised two concerns, which were checked against the frozen source rather than accepted as proven leaks:

• core.models._SESSION_MANAGER_INSTANCE: SessionManager.__init__ only initializes its owned cache and loads metadata; it does not call the global setter. SessionManager.add_message explicitly persists through its own manager instead of calling Session.add_message / the singleton. Neither truncation witness nor the helper invokes a singleton setter or API route setup. Thus the asserted singleton mutation is absent from this lane's real seam. Resetting that unrelated singleton would add cleanup without a proven owned mutation and could hide a genuine leak elsewhere.
• Eager real-database import versus clear_fake_database_modules: the exact pre-import block already exists on frozen b1666951. The clearing helper intentionally leaves real modules with a string __file__ intact and evicts replacement fake modules lacking it. The correction neither adds that pre-import nor changes fake-module identification. Its "real module" early return is intended behavior, not a newly introduced cleanup conflict.

Neither concern supplied a failing reproducer. Both were resolved by the existing source paths and the actual collection/teardown identity regression; no speculative global cleanup was introduced.

The minimized production reproduction command was:

DATABASE_URL=sqlite:///:memory: /tmp/pr40-final-validation-venv/bin/python - <<'PYTHON'
from src.agent_loop import _explicitly_avoids_web_lookup, _classify_agent_request, _web_search_unavailable_for_turn
from src.tool_policy import WEB_TOOL_NAMES
for text in (
    "Answer from memory only, don't search online.",
    'Summarise what you already know. Do not search the web.',
    'No web search please, just tell me what you know about Python decorators.',
):
    intent = _classify_agent_request([{'role': 'user', 'content': text}], text)
    domains = set(intent['domains'])
    print(repr(text))
    print('negative_web:', _explicitly_avoids_web_lookup(text))
    print('intent_domains:', sorted(domains))
    print('web_disabled_short_circuit:', _web_search_unavailable_for_turn(domains, set(WEB_TOOL_NAMES), text, None, None))
PYTHON

Results: negative-web detection true for all three; intent domains respectively ['memory', 'web'], ['web'], ['web']; short-circuit false, true, true.

Final validation result and verdict

The final complete run in full-pytest-final.log exited 0: 12556 passed, 59 skipped, 2 xfailed, 184 warnings and 6 passing subtests in 468.49s. All 12617 collected cases are accounted for. The existing optional, platform-specific and opt-in skips remain; no skip marker was added. Installing the already-locked Node dependencies enabled three previously skipped cases, so the skip count decreased from 62 in the first run to 59. No tests were deleted and no unrelated xfail was introduced.

The font/color browser witness passed in the final complete run. Its failed 44-case diagnostic and passing isolated result remain documented as an intermittent observation; no production root cause was established or fixed.

Database/environment/module isolation validation is green. Both historical negative-web cases remain narrowly expected failures for a reproduced genuine runtime defect outside the test-isolation boundary. These are integration blockers; production intent/short-circuit behavior must be corrected in an appropriate lane before removing their strict markers.

WAVE 6 TEST-ISOLATION FOUNDATION NOT READY