mirror of
https://github.com/pewdiepie-archdaemon/odysseus.git
synced 2026-10-16 11:52:25 +02:00
feat(harness): authority, turn contract, tools, clean_agent_preview
Chat/agent harness work: request-authority model, turn contract, tool
policy/parsing/execution, clean_agent_preview streaming/thinking, ask_user
persistence, session probes, and scheduler.
Source: vendor/photocraft (ad1cff44) harness files, 3-way merged onto
upstream/dev (a8c147b2). Harness core files were independently developed
in both lineages; resolution policy documented in pr_work/C_conflicts.md
(vendor kept where ahead, upstream kept where vendor is behind; full delta
preserved in pr_work/C_upstream_delta.patch).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
pewdiepie-archdaemon
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
a8c147b238
commit
364229f2dc
@@ -0,0 +1,369 @@
|
||||
# Wave 6 test-isolation foundation
|
||||
|
||||
## Provenance and boundary
|
||||
|
||||
- Worktree: `/home/alteixeira20/Projects/Odysseus/odysseus-wave6-test-isolation`.
|
||||
- Starting branch: `wave6/test-isolation-foundation`.
|
||||
- Starting HEAD: `b1666951faf8285054e1ca90f11533b0fb53fb57`.
|
||||
- Starting TREE: `a3dee24e3c56c22e1d3f9805f409bf16b086c3d4`.
|
||||
- `git status --short --branch`: `## wave6/test-isolation-foundation` only.
|
||||
- `git status --porcelain`: empty. Starting worktree clean.
|
||||
- No rebase, merge, push, PR edit, or access to the Wave 4 worktree.
|
||||
- All changes are test code, test helpers, or this test validation report.
|
||||
No production code changed; no STOP-level exception was needed.
|
||||
|
||||
## Contamination map
|
||||
|
||||
| Source | Mutation / binding | Previous cleanup | Escaping state / effect |
|
||||
| --- | --- | --- | --- |
|
||||
| Both tests in `tests/test_truncate_message_count_regression.py`, through `_make_manager` | Creates an unowned `mkstemp` DB; directly sets `os.environ["DATABASE_URL"]`; reloads `core.database` then `core.session_manager` | Only the assertion's ORM session is closed | Environment, engine, factory, declarative Base, ORM classes, manager globals, and registered engine listeners survive; database files are never deleted |
|
||||
| Collection of `tests/test_manage_tasks_owner_scope.py` | Creates a module-lifetime temporary DB and assigns `_TS` permanently to `core.database.SessionLocal` | Some individual rows deleted; ORM sessions closed | The collection-time factory assignment leaks to unrelated tests; truncation reload replaces it, so production handlers and `_seed` / `_get` use different databases |
|
||||
| `test_checkin_digest_owner_scope.py`, `test_gallery_owner_filter_single_user.py`, `test_document_session_owner_scope.py` | Create module-lifetime DB files / engines and retain local factories | Some rows deleted / route factory restored / ORM sessions closed | Owned files and engines lack teardown; retained imported ORM identities can disagree with reloaded module globals |
|
||||
| `tests/conftest.py` bootstrap | `DATABASE_URL` was set only if absent before importing `core.database` | No environment restoration | An inherited developer URL could be opened and initialized during collection |
|
||||
|
||||
`importlib.reload` updates the existing module dictionary in place; it does
|
||||
not change the `sys.modules` module object's identity. Previously imported
|
||||
symbols nevertheless keep their old class/factory identities. In particular,
|
||||
`src.database` re-exports old objects, while reloaded `core.session_manager`
|
||||
rebinds its `DbSession`, `DbChatMessage`, `DbDocument`, `SessionLocal`, and
|
||||
`SessionManager`. Old functions still resolve globals through those changed
|
||||
module dictionaries. Each database reload also registers a new SQLite connect
|
||||
listener on SQLAlchemy's Engine class.
|
||||
|
||||
The current failures are principally in `test_manage_tasks_owner_scope.py`:
|
||||
`_seed` writes through its retained `_TS`, but `do_manage_tasks` imports the
|
||||
now-replaced factory from `core.database` at call time. Ownerless / cross-owner
|
||||
mutations incorrectly observe "task not found"; successful mutation/list/create
|
||||
witnesses likewise read or write the wrong test database. Four invalid-date
|
||||
cases still pass, so the observed contamination count is 16 rather than all
|
||||
20 task-owner cases.
|
||||
|
||||
`src.settings` has TTL caches, but those caches do not resolve DATABASE_URL.
|
||||
The contaminating helper neither patches nor reloads settings. The database
|
||||
URL is read directly from the environment at database import/reload time;
|
||||
settings cache invalidation is not the correction for this leak.
|
||||
|
||||
## Baseline reproduction before editing
|
||||
|
||||
All baseline commands explicitly selected in-memory bootstrap storage. Python
|
||||
with installed pytest/SQLAlchemy was
|
||||
`/tmp/pr40-final-validation-venv/bin/python` (pytest 9.1.1, SQLAlchemy 2.1.1).
|
||||
|
||||
```sh
|
||||
DATABASE_URL=sqlite:///:memory: /tmp/pr40-final-validation-venv/bin/python -m pytest -q -p no:cacheprovider tests/test_manage_tasks_owner_scope.py tests/test_checkin_digest_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_document_session_owner_scope.py tests/test_truncate_message_count_regression.py
|
||||
# 30 passed, 9 warnings in 2.48s
|
||||
|
||||
DATABASE_URL=sqlite:///:memory: /tmp/pr40-final-validation-venv/bin/python -m pytest -q -p no:cacheprovider tests/test_truncate_message_count_regression.py tests/test_manage_tasks_owner_scope.py tests/test_checkin_digest_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_document_session_owner_scope.py
|
||||
# 16 failed, 14 passed, 9 warnings in 2.40s
|
||||
|
||||
DATABASE_URL=sqlite:///:memory: /tmp/pr40-final-validation-venv/bin/python -m pytest -q -p no:cacheprovider -rx --runxfail tests/test_runtime_behavior_regressions.py -k negative_web
|
||||
# 2 failed (IndexError), 1 passed, 13 deselected, 1 warning in 1.61s
|
||||
```
|
||||
|
||||
The same 30 database tests shuffled with `random.Random(42)` before editing
|
||||
reported **9 failed, 21 passed**, 9 warnings in 2.37s. Exact command:
|
||||
|
||||
```sh
|
||||
/tmp/pr40-final-validation-venv/bin/python /tmp/odysseus-wave6-isolation-evidence/run_order.py seed:42 before-seed42 tests/test_manage_tasks_owner_scope.py tests/test_checkin_digest_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_document_session_owner_scope.py tests/test_truncate_message_count_regression.py
|
||||
```
|
||||
|
||||
## Isolation correction and regression evidence
|
||||
|
||||
- `tests/helpers/database.py` owns a fresh TemporaryDirectory, file-backed
|
||||
SQLite engine with NullPool, real schema, and factory for each invocation.
|
||||
Teardown disposes the engine and removes the directory including sidecars.
|
||||
- The truncation fixture temporarily patches only resource bindings in
|
||||
`core.database`, `src.database`, and `core.session_manager`, with a matching
|
||||
temporary environment URL. Patches are undone before resource disposal.
|
||||
No module reload, module eviction, new ORM class, or listener registration.
|
||||
- Four owner modules now own a disposable database per test rather than at
|
||||
collection. Only the task-owner module patches the canonical factory; that
|
||||
patch is scoped to the test and restored during teardown. The other modules
|
||||
retain their existing handler/route wiring and ownership assertions.
|
||||
- The existing pytest bootstrap always chooses in-memory storage before
|
||||
production database imports and registers environment restoration with the
|
||||
pytest configuration lifecycle. No new process-wide autouse cleanup fixture.
|
||||
- `tests/test_database_test_isolation.py` checks real persistent truncation,
|
||||
exact class/factory restoration, file removal, clean subsequent database
|
||||
state, and exception teardown. Child pytest runs execute the actual original
|
||||
witness entry points in both directions and check process-visible bindings
|
||||
after collection and after every test. An inherited developer-DB sentinel
|
||||
remains byte-identical, with no database sidecars created, and the inherited
|
||||
environment URL is restored after pytest exits.
|
||||
|
||||
Mutation checks temporarily restored one original file at a time, ran
|
||||
`test_actual_tests_restore_process_state_and_ignore_inherited_database[True]`,
|
||||
and restored the corrected file in `finally`. Both mutations made the new
|
||||
regression fail (pytest exit 1): original truncation was caught as
|
||||
"test leaked database or module state", original task-owner collection as
|
||||
"collection changed database bindings". No mutation was retained.
|
||||
|
||||
## Historical negative-web witnesses and production blocker
|
||||
|
||||
Both parameter cases in
|
||||
`test_negative_web_wording_withholds_the_web_tools_unhandled` intended to prove
|
||||
that inferred negative-web wording can receive a memory-only answer without
|
||||
`web_search` or `web_fetch` being offered to the model:
|
||||
|
||||
1. `Summarise what you already know. Do not search the web.`
|
||||
2. `No web search please, just tell me what you know about Python decorators.`
|
||||
|
||||
Both previously indexed an empty fake-model call list and xfailed on unrelated
|
||||
`IndexError`. The fake also emitted provider-shaped `choices` JSON at a seam
|
||||
that now consumes normalized `{"delta": ...}` SSE.
|
||||
|
||||
The repaired witnesses drive the unmodified public `stream_agent_loop`, mock
|
||||
only model output and existing infrastructure edges, capture the actual SSE,
|
||||
and require one model call, withheld web schemas, and delivered fake answer.
|
||||
The existing passing "don't search online" control and positive search
|
||||
control remain. The fake now emits normalized SSE.
|
||||
|
||||
The two historical xfails remain **strict**, with
|
||||
`raises=_MemoryOnlyTurnBlocked`. That exception is emitted only when the real
|
||||
loop makes zero model calls and returns the exact known disabled-web response.
|
||||
IndexError, arbitrary assertions, or other exceptions no longer qualify.
|
||||
|
||||
This is a genuine production defect outside the lane: negative-web detection
|
||||
is true for both phrasings, but `_classify_agent_request` returns `{'web'}` and
|
||||
`_web_search_unavailable_for_turn` returns true after the negative has disabled
|
||||
web tools. The loop short-circuits the requested memory-only response with:
|
||||
|
||||
> Web access is disabled for this turn. Enable web search and resend the request.
|
||||
|
||||
The passing "Answer from memory only, don't search online." control has
|
||||
`{'memory', 'web'}` intent domains and does not short-circuit. A minimized
|
||||
reproduction using those three production functions requires neither a model
|
||||
nor a mock. Relevant production locations (intentionally untouched):
|
||||
`src/agent_loop.py` negative detector near line 864, classifier near 10371,
|
||||
negative-tool disabling near 20735, unavailable calculation near 22233, and
|
||||
short-circuit response near 25101.
|
||||
|
||||
```sh
|
||||
/tmp/pr40-final-validation-venv/bin/python -m pytest -q -p no:cacheprovider --runxfail tests/test_runtime_behavior_regressions.py -k negative_web_wording_withholds_the_web_tools_unhandled
|
||||
# 2 failed with _MemoryOnlyTurnBlocked, 14 deselected, 1 warning in 0.75s
|
||||
```
|
||||
|
||||
No production repair was attempted. Runtime intent/availability behavior,
|
||||
authority, completion/evidence, and Wave 4 effects files remain untouched.
|
||||
This blocker prevents a READY release-foundation verdict despite passing test
|
||||
isolation validation.
|
||||
|
||||
## Final order validation
|
||||
|
||||
The focused manifest has 50 cases: the five baseline database modules plus
|
||||
`test_database_test_isolation.py` and `test_runtime_behavior_regressions.py`.
|
||||
Normal collection order is alphabetical by the seven explicitly selected files.
|
||||
Reversed and seeded runs reorder **individual collected test cases**, including
|
||||
parameterized cases. Both contamination directions also vary file collection
|
||||
order. Every execution manifest is retained as JSON in the evidence directory.
|
||||
|
||||
| Run | Result |
|
||||
| --- | --- |
|
||||
| Normal | 48 passed, 2 xfailed, 0 skipped, 11 warnings in 9.01s |
|
||||
| Reverse of all collected cases | 48 passed, 2 xfailed, 0 skipped, 11 warnings in 8.47s |
|
||||
| Truncation module first, owner witnesses afterward | 48 passed, 2 xfailed, 0 skipped, 11 warnings in 8.00s |
|
||||
| All four owner modules first, truncation afterward | 48 passed, 2 xfailed, 0 skipped, 11 warnings in 8.98s |
|
||||
| Seed 1 | 48 passed, 2 xfailed, 0 skipped, 11 warnings in 9.01s |
|
||||
| Seed 42 | 48 passed, 2 xfailed, 0 skipped, 11 warnings in 8.98s |
|
||||
| Seed 2026 | 48 passed, 2 xfailed, 0 skipped, 11 warnings in 8.97s |
|
||||
|
||||
Exact executed commands:
|
||||
|
||||
```sh
|
||||
/tmp/pr40-final-validation-venv/bin/python /tmp/odysseus-wave6-isolation-evidence/run_order.py normal after-normal tests/test_checkin_digest_owner_scope.py tests/test_database_test_isolation.py tests/test_document_session_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_manage_tasks_owner_scope.py tests/test_runtime_behavior_regressions.py tests/test_truncate_message_count_regression.py
|
||||
/tmp/pr40-final-validation-venv/bin/python /tmp/odysseus-wave6-isolation-evidence/run_order.py reverse after-reverse tests/test_checkin_digest_owner_scope.py tests/test_database_test_isolation.py tests/test_document_session_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_manage_tasks_owner_scope.py tests/test_runtime_behavior_regressions.py tests/test_truncate_message_count_regression.py
|
||||
/tmp/pr40-final-validation-venv/bin/python /tmp/odysseus-wave6-isolation-evidence/run_order.py normal after-truncation-first tests/test_truncate_message_count_regression.py tests/test_checkin_digest_owner_scope.py tests/test_database_test_isolation.py tests/test_document_session_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_manage_tasks_owner_scope.py tests/test_runtime_behavior_regressions.py
|
||||
/tmp/pr40-final-validation-venv/bin/python /tmp/odysseus-wave6-isolation-evidence/run_order.py normal after-owners-first tests/test_checkin_digest_owner_scope.py tests/test_document_session_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_manage_tasks_owner_scope.py tests/test_truncate_message_count_regression.py tests/test_database_test_isolation.py tests/test_runtime_behavior_regressions.py
|
||||
/tmp/pr40-final-validation-venv/bin/python /tmp/odysseus-wave6-isolation-evidence/run_order.py seed:1 after-seed1 tests/test_checkin_digest_owner_scope.py tests/test_database_test_isolation.py tests/test_document_session_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_manage_tasks_owner_scope.py tests/test_runtime_behavior_regressions.py tests/test_truncate_message_count_regression.py
|
||||
/tmp/pr40-final-validation-venv/bin/python /tmp/odysseus-wave6-isolation-evidence/run_order.py seed:42 after-seed42 tests/test_checkin_digest_owner_scope.py tests/test_database_test_isolation.py tests/test_document_session_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_manage_tasks_owner_scope.py tests/test_runtime_behavior_regressions.py tests/test_truncate_message_count_regression.py
|
||||
/tmp/pr40-final-validation-venv/bin/python /tmp/odysseus-wave6-isolation-evidence/run_order.py seed:2026 after-seed2026 tests/test_checkin_digest_owner_scope.py tests/test_database_test_isolation.py tests/test_document_session_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_manage_tasks_owner_scope.py tests/test_runtime_behavior_regressions.py tests/test_truncate_message_count_regression.py
|
||||
```
|
||||
|
||||
The order driver is test-session-local; it is not installed into conftest or
|
||||
production and never changes the normal suite's ordering. Its exact source
|
||||
(for recreating `/tmp/odysseus-wave6-isolation-evidence/run_order.py`) is:
|
||||
|
||||
```python
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import random
|
||||
import sys
|
||||
import pytest
|
||||
|
||||
out = Path('/tmp/odysseus-wave6-isolation-evidence')
|
||||
mode, label, *modules = sys.argv[1:]
|
||||
class Order:
|
||||
@pytest.hookimpl(trylast=True)
|
||||
def pytest_collection_modifyitems(self, items):
|
||||
if mode == 'reverse':
|
||||
items.reverse()
|
||||
elif mode.startswith('seed:'):
|
||||
random.Random(int(mode.split(':')[1])).shuffle(items)
|
||||
(out / f'{label}-manifest.json').write_text(json.dumps([i.nodeid for i in items], indent=2) + '\n')
|
||||
os.environ['DATABASE_URL'] = 'sqlite:///:memory:'
|
||||
args = ['-q', '-p', 'no:cacheprovider', '-rsx', *modules]
|
||||
(out / f'{label}-command.json').write_text(json.dumps({'argv':sys.argv,'pytest_args':args,'DATABASE_URL':os.environ['DATABASE_URL']},indent=2) + '\n')
|
||||
raise SystemExit(pytest.main(args, plugins=[Order()]))
|
||||
```
|
||||
|
||||
## Adjacent and complete validation
|
||||
|
||||
The first complete run finished with **44 failed, 12509 passed, 62 skipped,
|
||||
2 xfailed**, 184 warnings and 6 passing subtests in 391.85s. Its failure
|
||||
inventory was 42 missing-Playwright-package failures and two generated
|
||||
configuration-reference freshness failures. The local worktree had not had
|
||||
`npm ci` run; the generator includes test-source line references, and the
|
||||
bootstrap edit moved the existing references by one line.
|
||||
|
||||
`npm ci --ignore-scripts` installed the four already-locked packages; package
|
||||
and lock files were unchanged. Cached Chromium launched successfully
|
||||
(version 151.0.7922.34). A bootstrap comment now explains why isolation must
|
||||
precede collection and preserves the existing referenced line locations,
|
||||
without editing the generated website or the generator. This follow-up is
|
||||
commit `f67d0388fd805867a0920dbffd4da056b1361e38`
|
||||
(`fix(tests): preserve bootstrap reference locations`).
|
||||
|
||||
```sh
|
||||
npm ci --ignore-scripts
|
||||
# exit 0; 4 packages installed, lockfile and package policy unchanged
|
||||
|
||||
python scripts/generate_env_reference.py --check
|
||||
# exit 0; website/configuration-reference.md is up to date (112 variables)
|
||||
|
||||
/tmp/pr40-final-validation-venv/bin/python -m pytest -q -p no:cacheprovider tests/test_document_module_api.py tests/test_env_reference.py
|
||||
# 16 passed, 1 warning in 17.00s
|
||||
```
|
||||
|
||||
The first full run and the 44-case focused rerun are retained separately from
|
||||
the final complete run. The 44-case rerun reported **1 failed, 43 passed**
|
||||
in 73.61s: the existing font/color browser witness observed its font-size
|
||||
icon still showing `13` after its text had changed to size 5 (expected `24`).
|
||||
An isolated reproduction of that unchanged test passed in 2.13s. This is an
|
||||
observed intermittent browser UI witness; no database failure was involved,
|
||||
and a production root cause has not been established. The test and editor
|
||||
implementation remain unchanged, with both observations retained rather
|
||||
than suppressing the failed run. No failure was hidden by skipping or deleting a test.
|
||||
|
||||
|
||||
```sh
|
||||
/tmp/pr40-final-validation-venv/bin/python -m pytest -q -p no:cacheprovider -rsx tests/test_session_manager.py tests/test_session_manager_cleanup.py tests/test_session_manager_persist_guard.py tests/test_session_mutation_lookup.py tests/test_session_discovery_message_count.py tests/test_history_order_by_timestamp_regression.py tests/test_history_display_model_hydration.py tests/test_session_search.py tests/test_session_search_batch_fetch.py tests/test_session_list_owner_scope.py tests/test_session_owner_attribution.py tests/test_session_endpoint_owner_scope.py tests/test_session_concurrent.py tests/test_app_db_permissions.py tests/test_database_utcnow.py tests/test_tool_policy.py tests/test_turn_contract.py tests/test_chat_preprocess_tool_policy.py tests/test_chat_route_tool_policy.py tests/test_resend_message_nondestructive.py
|
||||
# 1262 passed, 0 skipped, 0 xfailed, 7 warnings in 40.56s
|
||||
|
||||
/tmp/pr40-final-validation-venv/bin/python -m pytest -q -p no:cacheprovider -rsx
|
||||
# 12556 passed, 59 skipped, 2 xfailed, 184 warnings, 6 subtests passed in 468.49s (0:07:48); exit 0
|
||||
|
||||
python -m compileall -q app.py core routes src services scripts tests
|
||||
# exit 0
|
||||
|
||||
git diff --check
|
||||
# exit 0, no output
|
||||
|
||||
git diff --check b1666951faf8285054e1ca90f11533b0fb53fb57
|
||||
# exit 0, no output
|
||||
|
||||
git diff --name-only --diff-filter=U
|
||||
# exit 0, no output
|
||||
|
||||
rg -n '^(<<<<<<< |=======$|>>>>>>> )' <each modified file>
|
||||
# exit 1, no matches
|
||||
```
|
||||
|
||||
No test was deleted, no skip was added, and no unrelated xfail was added. The
|
||||
same two historical parameter cases remain xfailed for the demonstrated
|
||||
production blocker, now restricted to that actual contract failure.
|
||||
|
||||
The isolation commit is
|
||||
`e2b847c84d27ce785565757e81ca5293896eaab3`
|
||||
(`fix(tests): isolate database and module state`). The final bounded commit
|
||||
contains the repaired web witnesses and this validation report. Final commit,
|
||||
HEAD, TREE and clean status are supplied in the accompanying final report to
|
||||
avoid placing a commit's own hash inside its committed contents.
|
||||
|
||||
Full logs, commands, and per-run manifests from this session are in
|
||||
`/tmp/odysseus-wave6-isolation-evidence/`.
|
||||
|
||||
The follow-up normal focused run after preserving bootstrap reference
|
||||
locations is recorded in `after-bootstrap-followup.log`. It reported **48 passed, 2 xfailed**, 11 warnings in 9.55s. Its exact command is:
|
||||
|
||||
```sh
|
||||
/tmp/pr40-final-validation-venv/bin/python /tmp/odysseus-wave6-isolation-evidence/run_order.py normal after-bootstrap-followup tests/test_checkin_digest_owner_scope.py tests/test_database_test_isolation.py tests/test_document_session_owner_scope.py tests/test_gallery_owner_filter_single_user.py tests/test_manage_tasks_owner_scope.py tests/test_runtime_behavior_regressions.py tests/test_truncate_message_count_regression.py
|
||||
```
|
||||
|
||||
The additional browser observation came from the 44-case manifest retained in
|
||||
`full-failures-manifest.json`; every exact node ID in that diagnostic command is
|
||||
retained in `full-failures-rerun-command.json`. The isolated diagnostic command:
|
||||
|
||||
```sh
|
||||
/tmp/pr40-final-validation-venv/bin/python -m pytest -q -p no:cacheprovider tests/test_document_rich_font_color_controls.py::test_numeric_font_size_and_custom_colors_work_on_desktop_and_mobile
|
||||
# 1 passed, 1 warning in 2.13s
|
||||
```
|
||||
|
||||
## Independent review disposition
|
||||
|
||||
A read-only review of isolation commit e2b847c8 raised two concerns, which
|
||||
were checked against the frozen source rather than accepted as proven leaks:
|
||||
|
||||
- `core.models._SESSION_MANAGER_INSTANCE`: `SessionManager.__init__` only
|
||||
initializes its owned cache and loads metadata; it does not call the global
|
||||
setter. `SessionManager.add_message` explicitly persists through its own
|
||||
manager instead of calling `Session.add_message` / the singleton. Neither
|
||||
truncation witness nor the helper invokes a singleton setter or API route
|
||||
setup. Thus the asserted singleton mutation is absent from this lane's real
|
||||
seam. Resetting that unrelated singleton would add cleanup without a proven
|
||||
owned mutation and could hide a genuine leak elsewhere.
|
||||
- Eager real-database import versus `clear_fake_database_modules`: the exact
|
||||
pre-import block already exists on frozen b1666951. The clearing helper
|
||||
intentionally leaves real modules with a string `__file__` intact and evicts
|
||||
replacement fake modules lacking it. The correction neither adds that
|
||||
pre-import nor changes fake-module identification. Its "real module" early
|
||||
return is intended behavior, not a newly introduced cleanup conflict.
|
||||
|
||||
Neither concern supplied a failing reproducer. Both were resolved by the
|
||||
existing source paths and the actual collection/teardown identity regression;
|
||||
no speculative global cleanup was introduced.
|
||||
|
||||
The minimized production reproduction command was:
|
||||
|
||||
```sh
|
||||
DATABASE_URL=sqlite:///:memory: /tmp/pr40-final-validation-venv/bin/python - <<'PYTHON'
|
||||
from src.agent_loop import _explicitly_avoids_web_lookup, _classify_agent_request, _web_search_unavailable_for_turn
|
||||
from src.tool_policy import WEB_TOOL_NAMES
|
||||
for text in (
|
||||
"Answer from memory only, don't search online.",
|
||||
'Summarise what you already know. Do not search the web.',
|
||||
'No web search please, just tell me what you know about Python decorators.',
|
||||
):
|
||||
intent = _classify_agent_request([{'role': 'user', 'content': text}], text)
|
||||
domains = set(intent['domains'])
|
||||
print(repr(text))
|
||||
print('negative_web:', _explicitly_avoids_web_lookup(text))
|
||||
print('intent_domains:', sorted(domains))
|
||||
print('web_disabled_short_circuit:', _web_search_unavailable_for_turn(domains, set(WEB_TOOL_NAMES), text, None, None))
|
||||
PYTHON
|
||||
```
|
||||
|
||||
Results: negative-web detection true for all three; intent domains respectively
|
||||
`['memory', 'web']`, `['web']`, `['web']`; short-circuit false, true, true.
|
||||
|
||||
## Final validation result and verdict
|
||||
|
||||
The final complete run in `full-pytest-final.log` exited 0:
|
||||
**12556 passed, 59 skipped, 2 xfailed**, 184 warnings and 6 passing subtests in
|
||||
468.49s. All 12617 collected cases are accounted for. The existing optional,
|
||||
platform-specific and opt-in skips remain; no skip marker was added. Installing
|
||||
the already-locked Node dependencies enabled three previously skipped cases,
|
||||
so the skip count decreased from 62 in the first run to 59. No tests were
|
||||
deleted and no unrelated xfail was introduced.
|
||||
|
||||
The font/color browser witness passed in the final complete run. Its failed
|
||||
44-case diagnostic and passing isolated result remain documented as an
|
||||
intermittent observation; no production root cause was established or fixed.
|
||||
|
||||
Database/environment/module isolation validation is green. Both historical
|
||||
negative-web cases remain narrowly expected failures for a reproduced genuine
|
||||
runtime defect outside the test-isolation boundary. These are integration
|
||||
blockers; production intent/short-circuit behavior must be corrected in an
|
||||
appropriate lane before removing their strict markers.
|
||||
|
||||
WAVE 6 TEST-ISOLATION FOUNDATION NOT READY
|
||||
@@ -0,0 +1,45 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { chromium } from 'playwright';
|
||||
|
||||
const browser = await chromium.launch({ headless: true });
|
||||
try {
|
||||
for (const width of [390, 900]) {
|
||||
const page = await browser.newPage({ viewport: { width, height: 844 }, acceptDownloads: true });
|
||||
await page.goto('http://127.0.0.1:7011/static/js/documentStats.js');
|
||||
await page.setContent('<link rel="stylesheet" href="/static/style.css"><div id="toast"></div><div id="chat-container"></div><div id="sidebar"></div>');
|
||||
await page.evaluate(async () => {
|
||||
const mod = await import('/static/js/document.js?mobile-export-regression=1');
|
||||
mod.init('/api');
|
||||
mod.injectFreshDoc({ id: 'export-test', title: 'Export test', language: 'markdown', version_count: 1, current_content: '# Mobile export\n\nActual document content.' });
|
||||
});
|
||||
await page.locator('#doc-footer-export-btn').waitFor({ state: 'visible' });
|
||||
await page.evaluate(() => {
|
||||
document.querySelector('.doc-editor-pane').style.setProperty('z-index', '12000', 'important');
|
||||
});
|
||||
await page.locator('#doc-footer-export-btn').click();
|
||||
const menu = page.locator('#doc-export-menu');
|
||||
await menu.waitFor({ state: 'visible' });
|
||||
const bounds = await menu.boundingBox();
|
||||
assert(bounds.x >= 0 && bounds.y >= 0);
|
||||
assert(bounds.x + bounds.width <= width && bounds.y + bounds.height <= 844);
|
||||
assert(await menu.evaluate(el => Number(getComputedStyle(el).zIndex) > 12000));
|
||||
await page.screenshot({ path: `/tmp/document-export-${width}.png` });
|
||||
// A second click closes the menu, then reopening must still allow export.
|
||||
await page.locator('#doc-footer-export-btn').click();
|
||||
assert.equal(await menu.count(), 0);
|
||||
await page.locator('#doc-footer-export-btn').click();
|
||||
const pending = page.waitForEvent('download');
|
||||
await page.locator('#doc-export-menu button').filter({ hasText: /^Export Markdown$/ }).click();
|
||||
const download = await pending;
|
||||
assert(download.suggestedFilename().endsWith('.md'));
|
||||
assert.equal(await download.failure(), null);
|
||||
const stream = await download.createReadStream();
|
||||
let content = '';
|
||||
for await (const chunk of stream) content += chunk;
|
||||
assert.equal(content, '# Mobile export\n\nActual document content.');
|
||||
await page.close();
|
||||
console.log(`PASS ${width}px: visible menu, toggle, download contents`);
|
||||
}
|
||||
} finally {
|
||||
await browser.close();
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { chromium } from 'playwright';
|
||||
|
||||
const source = readFileSync(new URL('../static/js/document.js', import.meta.url), 'utf8');
|
||||
const build = source.slice(source.indexOf(' function _buildAnnotation('), source.indexOf(' // Prompt user for an instruction'));
|
||||
const state = source.slice(source.indexOf(' let _pdfAnnotationDragging'), source.indexOf(' // Per-doc last-used line spacing'));
|
||||
const placement = source.slice(source.indexOf(' const isBlankPageTarget'), source.indexOf(' pane.appendChild(pageWrap);', source.indexOf(' const isBlankPageTarget')));
|
||||
const browser = await chromium.launch({ headless: true });
|
||||
try {
|
||||
for (const touch of [true, false]) {
|
||||
const page = await browser.newPage({ viewport: { width: 600, height: 800 }, hasTouch: touch });
|
||||
await page.setContent('<div id="page" style="position:relative;width:500px;height:700px;background:white"></div>');
|
||||
await page.evaluate(({ build, state, placement }) => {
|
||||
new Function(`
|
||||
const activeDocId = 'test', docId = activeDocId;
|
||||
const _pdfPaneAnnotationsByDoc = new Map([['test', []]]);
|
||||
const annotationRefs = _pdfPaneAnnotationsByDoc.get('test');
|
||||
const _pdfLastLineHeight = new Map(), _pdfLastFontSize = new Map();
|
||||
let _pdfDropMode = 'check', serial = 0;
|
||||
const _newAnnotationId = () => 'new-' + (++serial);
|
||||
const _pushPdfUndoSnapshot = () => {};
|
||||
const _schedulePdfPaneSave = () => {};
|
||||
const pageWrap = document.getElementById('page');
|
||||
const page = {page: 1};
|
||||
${state}
|
||||
${build}
|
||||
${placement}
|
||||
for (let i = 0; i < 2; i++) {
|
||||
const {ref} = _buildAnnotation(pageWrap, {id:'a'+i, page:1, x:20+i*35, y:30, w:5, h:5, kind:'check'});
|
||||
annotationRefs.push(ref);
|
||||
ref._setHandlesVisible(true);
|
||||
}
|
||||
window.refs = annotationRefs;
|
||||
`)();
|
||||
}, { build, state, placement });
|
||||
const grip = page.locator('[data-ann-id="a0"] [title="Drag to move"]');
|
||||
const box = await grip.boundingBox();
|
||||
await page.mouse.move(box.x + box.width / 2, box.y + box.height / 2);
|
||||
await page.mouse.down();
|
||||
await page.mouse.move(300, 220);
|
||||
assert(await page.evaluate(() => refs.every(ref => [...ref.wrap.querySelectorAll('[title]')].every(el => el.style.display === 'none'))));
|
||||
// Proximity updates must not bring controls back while dragging.
|
||||
await page.evaluate(() => refs.forEach(ref => ref._setHandlesVisible(true)));
|
||||
assert(await page.locator('[data-ann-id="a1"] [title="Delete annotation"]').isHidden());
|
||||
await page.mouse.up();
|
||||
assert.equal(await page.locator('.pdf-annotation-wrap').count(), 2);
|
||||
assert(await page.locator('[data-ann-id="a1"] [title="Delete annotation"]').isVisible());
|
||||
await page.locator('[data-ann-id="a1"] [title="Delete annotation"]').click();
|
||||
assert.equal(await page.locator('.pdf-annotation-wrap').count(), 1);
|
||||
await page.locator('#page').click({ position: { x: 100, y: 450 } });
|
||||
assert.equal(await page.locator('.pdf-annotation-wrap').count(), 2);
|
||||
console.log(`PASS ${touch ? 'touch' : 'mouse'}: drag controls hidden, delete only deletes, blank page still places`);
|
||||
await page.close();
|
||||
}
|
||||
} finally {
|
||||
await browser.close();
|
||||
}
|
||||
@@ -89,7 +89,7 @@ a status code. `scripts/odysseus-smoke --areas` prints the current list.
|
||||
| Email | the fixture inbox lists, opens with its body, and the unread count drops on mark-read |
|
||||
| Memory | a fact is listed, found by search, and gone after delete |
|
||||
| Uploads | an attachment reads back byte for byte |
|
||||
| Cookbook | hardware is detected; a fresh install's recommendation request returns the explicit empty-catalog Rescan guidance; state persists |
|
||||
| Cookbook | hardware is detected and recommendations come back sized against it; state persists |
|
||||
| Settings | a preference written on one session is still there after a new login |
|
||||
|
||||
## What is not covered, and why
|
||||
|
||||
@@ -20,3 +20,4 @@ def test_terse_timed_calendar_create_promotes_to_calendar():
|
||||
|
||||
assert intent.needs_tools
|
||||
assert intent.category == "calendar"
|
||||
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from src.agent_evidence import (
|
||||
CompletionRequirements,
|
||||
CompletionStatus,
|
||||
@@ -934,3 +936,66 @@ def test_exhaustion_and_awaiting_user_are_non_completion_states():
|
||||
ledger = EvidenceLedger()
|
||||
assert ledger.evaluate(exhausted=True).status == CompletionStatus.EXHAUSTED
|
||||
assert ledger.evaluate(awaiting_user=True).status == CompletionStatus.AWAITING_USER
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tool", ["create_document", "update_document", "edit_document"])
|
||||
def test_authoritative_document_write_records_document_mutation_evidence(tool):
|
||||
ledger = EvidenceLedger.from_tool_events([
|
||||
{"tool": tool, "command": "fix grammar", "exit_code": 0},
|
||||
])
|
||||
|
||||
mutations = [event for event in ledger.events
|
||||
if event.kind == EvidenceKind.DOCUMENT_MUTATION]
|
||||
assert len(mutations) == 1
|
||||
assert mutations[0].authoritative and mutations[0].success
|
||||
assert ledger._supports_document_mutation() is True
|
||||
|
||||
|
||||
def test_non_authoritative_document_write_is_not_mutation_evidence():
|
||||
ledger = EvidenceLedger.from_tool_events([
|
||||
{"tool": "update_document", "command": "fix grammar"},
|
||||
])
|
||||
|
||||
assert ledger._supports_document_mutation() is False
|
||||
|
||||
|
||||
def test_successful_private_write_supports_mutation_claim_without_artifact():
|
||||
# manage_calendar:create_event has no filesystem artifact, so the completion
|
||||
# verifier previously erased the model's "I created the meeting" claim. A
|
||||
# successful authoritative owned-state write must now support it.
|
||||
from src.agent_runtime.completion import completion_answer
|
||||
|
||||
ledger = EvidenceLedger.from_tool_events([
|
||||
{
|
||||
"round": 1,
|
||||
"tool": "manage_calendar",
|
||||
"command": json.dumps({
|
||||
"action": "create_event",
|
||||
"title": "Meeting with Sam",
|
||||
"start": "2026-10-16T15:00:00",
|
||||
}),
|
||||
"output": "Created event [Meeting with Sam]",
|
||||
"exit_code": 0,
|
||||
"execution_attempted": True,
|
||||
},
|
||||
])
|
||||
assert ledger._private_action_mutations == ["manage_calendar:create_event"]
|
||||
|
||||
answer, _ = completion_answer(
|
||||
"I created the meeting with Sam.", ledger, ledger.evaluate())
|
||||
assert "created the meeting with Sam" in answer
|
||||
assert not answer.startswith("Unsupported execution claims")
|
||||
|
||||
|
||||
def test_failed_private_write_does_not_support_mutation_claim():
|
||||
ledger = EvidenceLedger.from_tool_events([
|
||||
{
|
||||
"round": 1,
|
||||
"tool": "manage_calendar",
|
||||
"command": json.dumps({"action": "create_event", "title": "x"}),
|
||||
"output": "denied",
|
||||
"exit_code": 1,
|
||||
"execution_attempted": True,
|
||||
},
|
||||
])
|
||||
assert ledger._private_action_mutations == []
|
||||
|
||||
@@ -87,6 +87,91 @@ def test_ask_user_is_emitted_last_and_persisted(monkeypatch):
|
||||
assert metrics["tool_events"][0]["ask_user"] == payload
|
||||
|
||||
|
||||
def test_clean_preview_ask_user_ends_turn_without_second_round(monkeypatch):
|
||||
"""The compact ``stream_preview`` runtime must treat ask_user as a turn boundary.
|
||||
|
||||
Regression for chat d8dca715: the preview fed the ask_user tool result
|
||||
back to the model, so it burned a second provider round and persisted the
|
||||
model's follow-up prose instead of the question.
|
||||
"""
|
||||
import src.clean_agent_preview as module
|
||||
from src.tool_schemas import FUNCTION_TOOL_SCHEMAS
|
||||
from src.tool_policy import ToolPolicy
|
||||
from src.turn_contract import resolve_full_inventory_contract
|
||||
|
||||
payload = {
|
||||
"question": "What would be most useful to start with?",
|
||||
"options": [
|
||||
{"label": "A 30-day meditation plan"},
|
||||
{"label": "How to measure brain activity"},
|
||||
],
|
||||
"multi": False,
|
||||
}
|
||||
requests = []
|
||||
responses = iter([
|
||||
{'choices': [{'delta': {'tool_calls': [{'index': 0, 'id': 'ask-1',
|
||||
'function': {'name': 'ask_user',
|
||||
'arguments': json.dumps(payload, ensure_ascii=False)}}]}}]},
|
||||
{'choices': [{'delta': {'content': 'SHOULD NOT BE REQUESTED'}}]},
|
||||
])
|
||||
|
||||
class Response:
|
||||
def __init__(self, body): self.body = body
|
||||
async def __aenter__(self): return self
|
||||
async def __aexit__(self, *args): pass
|
||||
def raise_for_status(self): pass
|
||||
async def aiter_lines(self):
|
||||
yield 'data: ' + json.dumps(self.body)
|
||||
yield 'data: [DONE]'
|
||||
|
||||
class Client:
|
||||
def __init__(self, **kwargs): pass
|
||||
async def __aenter__(self): return self
|
||||
async def __aexit__(self, *args): pass
|
||||
def stream(self, *args, **kwargs):
|
||||
requests.append(kwargs['json'])
|
||||
return Response(next(responses))
|
||||
|
||||
async def execute(block, **kwargs):
|
||||
parsed = json.loads(block.content)
|
||||
return 'ask_user', {
|
||||
'ask_user': parsed,
|
||||
'output': 'Awaiting their selection.',
|
||||
'exit_code': 0,
|
||||
}
|
||||
|
||||
monkeypatch.setattr(module.httpx, 'AsyncClient', Client)
|
||||
monkeypatch.setattr(module, 'execute_tool_block', execute)
|
||||
|
||||
schema = next(s for s in FUNCTION_TOOL_SCHEMAS if s['function']['name'] == 'ask_user')
|
||||
contract = resolve_full_inventory_contract(schemas=[schema], policy=ToolPolicy())
|
||||
raw = _collect(module.stream_preview(
|
||||
endpoint_url='http://test', model='test', headers={},
|
||||
messages=[{'role': 'user', 'content': 'Help me pick a project.'}],
|
||||
turn_contract=contract, session_id='test', owner='test',
|
||||
disabled_tools=set(), tool_policy=ToolPolicy(),
|
||||
))
|
||||
events = _events(raw)
|
||||
|
||||
# Ending the turn means no second provider round is requested.
|
||||
assert len(requests) == 1
|
||||
ask_user_events = [event for event in events if event.get('type') == 'ask_user']
|
||||
assert len(ask_user_events) == 1
|
||||
assert ask_user_events[0]['data'] == payload
|
||||
# The saved assistant text is the question, not the model's follow-up prose.
|
||||
final = next(event for event in events if event.get('type') == 'final_response')
|
||||
assert final['content'] == payload['question']
|
||||
types = [event.get('type') for event in events]
|
||||
assert types.index('ask_user') < types.index('metrics')
|
||||
# Reconstructable from persisted metadata, and no phantom extra rounds.
|
||||
metrics = next(event['data'] for event in events if event.get('type') == 'metrics')
|
||||
assert metrics['tool_events'][0]['ask_user'] == payload
|
||||
assert metrics['agent_rounds'] == 1
|
||||
assert metrics['clean_v3_turn'][-1] == {
|
||||
'role': 'assistant', 'content': payload['question'],
|
||||
}
|
||||
|
||||
|
||||
def test_frontend_uses_one_renderer_for_live_and_restored_cards():
|
||||
chat = (ROOT / "static" / "js" / "chat.js").read_text(encoding="utf-8")
|
||||
renderer = (ROOT / "static" / "js" / "chatRenderer.js").read_text(encoding="utf-8")
|
||||
|
||||
@@ -60,6 +60,7 @@ def test_login_page_static_assets_resolve_under_mount_path():
|
||||
mounted_login_url = "https://example.test/odysseus/login"
|
||||
for relative_asset, mounted_path in (
|
||||
("static/manifest.json", "/odysseus/static/manifest.json"),
|
||||
("static/icons/icon-192.png", "/odysseus/static/icons/icon-192.png"),
|
||||
(
|
||||
"static/fonts/FiraCode-Regular.woff2",
|
||||
"/odysseus/static/fonts/FiraCode-Regular.woff2",
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
"""A valid login session cookie must survive a deploy restart of the server.
|
||||
|
||||
AuthManager keeps tokens in memory but persists them to sessions.json on create
|
||||
and reloads them at startup, so a restart must NOT log the user out (the 302 the
|
||||
bench saw was the restart gap / a client-side artifact, not session loss).
|
||||
"""
|
||||
from core import auth
|
||||
|
||||
|
||||
def test_session_token_survives_a_restart(tmp_path):
|
||||
auth_path = str(tmp_path / "auth.json")
|
||||
first = auth.AuthManager(auth_path)
|
||||
assert first.create_user("pewds", "hunter2", is_admin=True)
|
||||
token = first.create_session_trusted("pewds")
|
||||
assert token and first.validate_token(token)
|
||||
|
||||
# Simulate a process restart: a brand-new AuthManager on the same data dir.
|
||||
second = auth.AuthManager(auth_path)
|
||||
assert second.validate_token(token) is True
|
||||
assert second.get_username_for_token(token) == "pewds"
|
||||
|
||||
|
||||
def test_revoked_token_does_not_survive_restart(tmp_path):
|
||||
auth_path = str(tmp_path / "auth.json")
|
||||
first = auth.AuthManager(auth_path)
|
||||
first.create_user("pewds", "hunter2", is_admin=True)
|
||||
token = first.create_session_trusted("pewds")
|
||||
first.revoke_token(token)
|
||||
|
||||
second = auth.AuthManager(auth_path)
|
||||
assert second.validate_token(token) is False
|
||||
@@ -0,0 +1,42 @@
|
||||
"""Spec-surface admission: the strict router picks the tools after authority is
|
||||
built, so spec turns admit the router's families; compact/full stay unchanged."""
|
||||
import json
|
||||
|
||||
from src.agent_runtime.authority import create_request_authority, ExactOperation
|
||||
|
||||
|
||||
def _permits(auth, tool, args=None):
|
||||
return auth.permits(ExactOperation.normalize(tool, json.dumps(args or {})))
|
||||
|
||||
|
||||
def _auth(text, **kw):
|
||||
return create_request_authority(text, owner="pewds", session_id="s1", workspace="", **kw)
|
||||
|
||||
|
||||
def test_spec_surface_admits_a_router_chosen_family_tool():
|
||||
auth = _auth("Search for the appointment email", spec_surface=True)
|
||||
assert _permits(auth, "mcp__email__search_emails", {"action": "list", "query": "appointment"})
|
||||
|
||||
|
||||
def test_non_spec_turn_does_not_admit_the_same_call():
|
||||
auth = _auth("Search for the appointment email")
|
||||
assert not _permits(auth, "mcp__email__search_emails", {"action": "list", "query": "appointment"})
|
||||
|
||||
|
||||
def test_spec_surface_admits_browser_interaction_and_batch():
|
||||
auth = _auth("do the thing", spec_surface=True)
|
||||
assert _permits(auth, "private_browser", {"action": "batch"})
|
||||
assert _permits(auth, "private_browser", {"action": "click", "selector": "#go"})
|
||||
assert _permits(auth, "private_browser", {"action": "fill", "selector": "#q", "text": "x"})
|
||||
|
||||
|
||||
def test_batch_is_read_but_click_stays_verb_gated_off_spec():
|
||||
auth = _auth("search the web for the appointment")
|
||||
assert _permits(auth, "private_browser", {"action": "batch"})
|
||||
assert not _permits(auth, "private_browser", {"action": "click", "selector": "#go"})
|
||||
|
||||
|
||||
def test_spec_surface_keeps_the_exact_read_input_seal():
|
||||
auth = _auth("Read note id abc123", spec_surface=True)
|
||||
assert _permits(auth, "manage_notes", {"action": "view", "id": "abc123"})
|
||||
assert not _permits(auth, "manage_notes", {"action": "view", "id": "zzz999"})
|
||||
@@ -1,8 +1,28 @@
|
||||
import json
|
||||
import pytest
|
||||
|
||||
from src.clean_agent_preview import preview_tool_result_text
|
||||
|
||||
|
||||
@pytest.mark.parametrize('snapshot', [None, '', ' '])
|
||||
def test_text_and_title_survive_without_snapshot(snapshot):
|
||||
result = {'url': 'https://example.org', 'title': 'Access denied',
|
||||
'text': 'Unusual traffic. Complete the CAPTCHA.', 'snapshot': snapshot}
|
||||
output = preview_tool_result_text({'output': json.dumps(result)}, 'private_browser', {})
|
||||
assert 'Access denied' in output
|
||||
assert 'Unusual traffic. Complete the CAPTCHA.' in output
|
||||
from src.clean_agent_preview import browser_observation_access_blocked
|
||||
assert browser_observation_access_blocked(output)
|
||||
|
||||
|
||||
def test_snapshot_takes_precedence_over_duplicate_page_text():
|
||||
output = preview_tool_result_text({'output': json.dumps({
|
||||
'snapshot': '- link "Report" [ref=e1]', 'text': 'Duplicated page body',
|
||||
})}, 'private_browser', {})
|
||||
assert '[ref=e1]' in output
|
||||
assert 'Duplicated page body' not in output
|
||||
|
||||
|
||||
def test_legacy_page_text_retains_access_block_evidence():
|
||||
result = {'url': 'https://example.org', 'title': 'Security verification',
|
||||
'text': 'Unusual traffic. Complete the CAPTCHA.'}
|
||||
|
||||
@@ -231,6 +231,25 @@ def test_disabled_tools_respects_missing_vs_explicit_toggles():
|
||||
)
|
||||
|
||||
|
||||
def test_request_authority_grants_web_when_turn_offers_web_tools():
|
||||
"""Web tools offered for a turn must carry request authority too.
|
||||
|
||||
Regression: the authority capabilities gate only recognized explicit
|
||||
browser wording (``_explicit_browser_intent``/``_external_discovery_intent``),
|
||||
so an implicit research question with the Web toggle on offered
|
||||
``web_search`` in the model's schema but rejected every call with
|
||||
``request_authority_denied``. The gate must also grant ``search_browser``
|
||||
whenever web tools are not disabled for the turn.
|
||||
"""
|
||||
source = _CHAT_ROUTES.read_text(encoding="utf-8")
|
||||
assert "({'search_browser'} if (" in source, (
|
||||
"request authority must be built with a search_browser capabilities gate"
|
||||
)
|
||||
assert "or \"web_search\" not in disabled_tools" in source, (
|
||||
"authority must grant search_browser whenever web_search is offered"
|
||||
)
|
||||
|
||||
|
||||
def test_explicit_private_browser_workflow_survives_web_search_disabled():
|
||||
source = _CHAT_ROUTES.read_text(encoding="utf-8")
|
||||
assert "_explicit_private_browser_intent" in source
|
||||
@@ -748,3 +767,21 @@ def test_frontend_sends_explicit_allow_web_search_false_in_agent_mode():
|
||||
assert "fd.append('allow_web_search', el('web-toggle').checked ? 'true' : 'false')" in source, (
|
||||
"Frontend must send explicit allow_web_search=false in agent mode when toggle is off"
|
||||
)
|
||||
|
||||
|
||||
def test_spec_surface_skips_the_contract_scope_tool_denial():
|
||||
"""On the strict-router (spec) surface the regex turn contract must NOT
|
||||
scope disabled_tools, or the spec router menu (email/notes/...) is emptied
|
||||
before src/spec_router.py runs. Compact/full/none keep the denial."""
|
||||
source = _CHAT_ROUTES.read_text(encoding="utf-8")
|
||||
assert "if not _spec_surface:" in source, (
|
||||
"the contract-scope disabled_tools.update must be guarded for spec turns"
|
||||
)
|
||||
idx = source.index("if not _spec_surface:")
|
||||
segment = source[idx:idx + 500]
|
||||
assert "disabled_tools.update(" in segment, (
|
||||
"the contract-scope denial must live inside the spec guard"
|
||||
)
|
||||
assert "if not _turn_contract.permits(" in segment, (
|
||||
"the guarded block must be the contract-scope denial"
|
||||
)
|
||||
|
||||
@@ -109,12 +109,37 @@ def test_deepseek_flash_keeps_tools_but_drops_unsupported_forced_choice():
|
||||
}
|
||||
|
||||
compatible = provider_compatible_tool_choice_request(request, 'deepseek-flash')
|
||||
assert [tool['function']['name'] for tool in compatible['tools']] == ['inspect_media']
|
||||
# A forced named choice must not hide the other offered tools.
|
||||
assert [tool['function']['name'] for tool in compatible['tools']] == [
|
||||
'inspect_media', 'python'
|
||||
]
|
||||
assert 'tool_choice' not in compatible
|
||||
assert request['tool_choice']['function']['name'] == 'inspect_media'
|
||||
assert provider_compatible_tool_choice_request(request, 'qwen3.5-9b') is request
|
||||
|
||||
|
||||
def test_deepseek_forced_read_does_not_hide_other_offered_tools():
|
||||
# Regression: a required calendar read forced tool_choice=manage_calendar on
|
||||
# a turn that also offered the email tools; the DeepSeek compatibility path
|
||||
# used to shrink `tools` to manage_calendar only, so the model reported the
|
||||
# email tools the user asked for were unavailable.
|
||||
request = {
|
||||
'model': 'deepseek-v4-flash',
|
||||
'tools': [
|
||||
{'type': 'function', 'function': {'name': 'manage_calendar'}},
|
||||
{'type': 'function', 'function': {'name': 'mcp__email__search_emails'}},
|
||||
{'type': 'function', 'function': {'name': 'mcp__email__read_email'}},
|
||||
],
|
||||
'tool_choice': {'type': 'function', 'function': {'name': 'manage_calendar'}},
|
||||
}
|
||||
compatible = provider_compatible_tool_choice_request(request, 'deepseek-v4-flash')
|
||||
names = [tool['function']['name'] for tool in compatible['tools']]
|
||||
assert names == [
|
||||
'manage_calendar', 'mcp__email__search_emails', 'mcp__email__read_email'
|
||||
]
|
||||
assert 'tool_choice' not in compatible
|
||||
|
||||
|
||||
def test_deepseek_v4_flash_drops_named_choice_for_thinking_mode():
|
||||
request = {
|
||||
'tools': [{'type': 'function', 'function': {'name': 'write_file'}}],
|
||||
@@ -136,8 +161,10 @@ def test_kimi_k3_drops_named_choice_incompatible_with_thinking_mode():
|
||||
|
||||
compatible = provider_compatible_tool_choice_request(request, 'kimi-k3')
|
||||
|
||||
# The full tool surface is preserved; only the unsupported forced choice
|
||||
# is dropped.
|
||||
assert [tool['function']['name'] for tool in compatible['tools']] == [
|
||||
'inspect_media'
|
||||
'inspect_media', 'write_file'
|
||||
]
|
||||
assert 'tool_choice' not in compatible
|
||||
|
||||
@@ -210,8 +237,8 @@ def test_private_browser_covered_click_does_not_advance_dom_revision():
|
||||
def test_compact_notes_preserves_create_vs_edit_and_replacement_semantics():
|
||||
schema = next(s for s in FUNCTION_TOOL_SCHEMAS if s['function']['name'] == 'manage_notes')
|
||||
compact = compact_schemas([schema])[0]['function']
|
||||
assert 'add creates a new note' in compact['description']
|
||||
assert 'update with id' in compact['description']
|
||||
assert 'ONE add call' in compact['description']
|
||||
assert 'update with id edits an existing note, never add' in compact['description']
|
||||
assert 'replaces the whole checklist' in compact['parameters']['properties']['checklist_items']['description']
|
||||
|
||||
|
||||
@@ -3511,6 +3538,10 @@ def test_every_compactly_offered_preview_tool_has_valid_policy_permitted_call():
|
||||
'web_fetch': ({'url': 'https://example.com'}, 'read this page'),
|
||||
'web_search': ({'query': 'current AI news'}, 'search the web'),
|
||||
'youtube_tool': ({'action': 'metadata', 'video_url': 'https://youtube.com/watch?v=x'}, 'read YouTube metadata'),
|
||||
'plan_edit': ({'prompt': 'make it black and white'}, 'make it black and white'),
|
||||
'run_editor_command': ({'op': 'document.info'}, 'inspect the document'),
|
||||
'get_editor_screenshot': ({}, 'take a screenshot of the editor'),
|
||||
'ai_edit_image': ({'op': 'remove-bg'}, 'remove the background'),
|
||||
}
|
||||
schemas = {
|
||||
schema['function']['name']: schema
|
||||
@@ -3537,6 +3568,9 @@ def test_every_compactly_offered_preview_tool_has_valid_policy_permitted_call():
|
||||
}
|
||||
else {'image_editing'} if name == 'edit_image'
|
||||
else {'image_generation'} if name == 'generate_image'
|
||||
else {'photo_editor'} if name in {
|
||||
'plan_edit', 'run_editor_command', 'get_editor_screenshot', 'ai_edit_image',
|
||||
}
|
||||
else frozenset()
|
||||
),
|
||||
contract_required_tools={name},
|
||||
@@ -4366,21 +4400,20 @@ def test_v3_schema_uses_configured_versioned_contract_root(tmp_path, monkeypatch
|
||||
(contract_root / 'eval_alltools_unseen_compare.py').write_text(
|
||||
"def tools_for_mode(tools, mode):\n"
|
||||
" assert mode == 'compact_contract_v5'\n"
|
||||
" tools[0]['function']['description'] = 'versioned-contract-loaded'\n"
|
||||
" tools[0]['function']['parameters']['properties']['action']['description'] = 'versioned-contract-loaded'\n"
|
||||
" return tools\n",
|
||||
encoding='utf-8',
|
||||
)
|
||||
monkeypatch.setenv('ODYSSEUS_TOOL_CONTRACT_ROOT', str(contract_root))
|
||||
module.contract_builder.cache_clear()
|
||||
try:
|
||||
# Probe a tool whose compact description the harness does not
|
||||
# replace; manage_notes now carries a full harness-owned override.
|
||||
search = next(
|
||||
notes = next(
|
||||
s for s in FUNCTION_TOOL_SCHEMAS
|
||||
if s['function']['name'] == 'web_search'
|
||||
if s['function']['name'] == 'manage_notes'
|
||||
)
|
||||
compact = module.compact_schemas([search])[0]
|
||||
assert compact['function']['description'].startswith('versioned-contract-loaded')
|
||||
compact = module.compact_schemas([notes])[0]
|
||||
assert compact['function']['parameters']['properties']['action']['description'] == 'versioned-contract-loaded'
|
||||
assert 'update with id edits an existing note, never add' in compact['function']['description']
|
||||
finally:
|
||||
module.contract_builder.cache_clear()
|
||||
|
||||
@@ -4392,6 +4425,7 @@ def test_v3_document_edit_schema_has_one_unambiguous_structured_form():
|
||||
assert parameters['required'] == ['edits']
|
||||
assert set(parameters['properties']) == {'edits', 'more'}
|
||||
assert parameters['properties']['more']['type'] == 'boolean'
|
||||
assert 'more' not in parameters['required']
|
||||
|
||||
|
||||
def test_v3_ui_schema_advertises_only_policy_executable_client_local_actions():
|
||||
@@ -4526,7 +4560,7 @@ async def test_stream_emits_incremental_text_and_persistable_history(monkeypatch
|
||||
raw = [chunk async for chunk in stream_preview(endpoint_url='http://test', model='test', messages=[{'role': 'user', 'content': 'hi'}], headers={}, turn_contract=contract, session_id='test', owner='test', disabled_tools=set(), tool_policy=ToolPolicy())]
|
||||
events = [json.loads(s[6:]) for s in raw if '[DONE]' not in s]
|
||||
assert [e['delta'] for e in events if 'delta' in e] == ['Hello', ' there']
|
||||
assert requests[0]['messages'][0]['content'].startswith('You are Odysseus.')
|
||||
assert requests[0]['messages'][0]['content'].startswith('You are Ajax, the assistant in Odysseus.')
|
||||
metrics = next(e['data'] for e in events if e.get('type') == 'metrics')
|
||||
assert metrics['clean_v3_turn'] == [{'role': 'assistant', 'content': 'Hello there'}]
|
||||
assert raw[-1] == 'data: [DONE]\n\n'
|
||||
@@ -4562,9 +4596,7 @@ async def test_preview_provider_stream_error_is_terminal_not_empty_answer(monkey
|
||||
disabled_tools=set(), tool_policy=ToolPolicy(),
|
||||
)]
|
||||
assert raw[-1].startswith('event: error\ndata: ')
|
||||
assert 'selected model provider failed' in raw[-1]
|
||||
assert 'provider_stream_error' in raw[-1]
|
||||
assert 'Qwen3_5MTPDraftModel' not in raw[-1]
|
||||
assert 'Qwen3_5MTPDraftModel' in raw[-1]
|
||||
assert all('returned no answer' not in chunk for chunk in raw)
|
||||
assert all('"type": "metrics"' not in chunk for chunk in raw)
|
||||
|
||||
@@ -4664,6 +4696,74 @@ def test_ajax_c375_tool_surface_disables_progressive_thinking():
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize('name,args,reason', [
|
||||
('python', {}, 'code'),
|
||||
('apply_patch', {}, 'code'),
|
||||
('edit_document', {}, 'document_edit'),
|
||||
('draft_email_reply', {}, 'email_draft'),
|
||||
('manage_tasks', {'action': 'create'}, 'task_design'),
|
||||
('manage_skills', {'action': 'publish'}, 'skill_evaluation'),
|
||||
('manage_tasks', {'action': 'list'}, ''),
|
||||
('manage_tasks', {'action': 'pause'}, ''),
|
||||
('manage_skills', {'action': 'view'}, ''),
|
||||
('read_file', {}, ''),
|
||||
('web_search', {}, ''),
|
||||
('manage_notes', {'action': 'add'}, ''),
|
||||
])
|
||||
def test_tool_deliberation_is_operation_specific(name, args, reason):
|
||||
from src.clean_agent_preview import tool_deliberation_reason, progressive_thinking_for_turn
|
||||
assert tool_deliberation_reason(name, args) == reason
|
||||
schemas = [{'function': {'name': name}}]
|
||||
assert progressive_thinking_for_turn('ajax_c375', schemas, deliberate=bool(reason)) == bool(reason)
|
||||
assert not progressive_thinking_for_turn('other-model', schemas, deliberate=True)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_task_creation_deliberates_before_executing_once(monkeypatch):
|
||||
import src.clean_agent_preview as module
|
||||
requests, executions = [], []
|
||||
arguments = {'action': 'create', 'name': 'Morning news', 'prompt': 'Summarize news', 'schedule': 'daily', 'time': '09:00'}
|
||||
proposal = {'choices': [{'delta': {'tool_calls': [{'index': 0, 'id': 'task-call',
|
||||
'function': {'name': 'manage_tasks', 'arguments': json.dumps(arguments)}}]}}]}
|
||||
responses = iter([proposal, proposal, {'choices': [{'delta': {'content': 'Created the task.'}}]}])
|
||||
|
||||
class Response:
|
||||
def __init__(self, payload): self.payload = payload
|
||||
async def __aenter__(self): return self
|
||||
async def __aexit__(self, *args): pass
|
||||
def raise_for_status(self): pass
|
||||
async def aiter_lines(self):
|
||||
yield 'data: ' + json.dumps(self.payload)
|
||||
yield 'data: [DONE]'
|
||||
|
||||
class Client:
|
||||
def __init__(self, **kwargs): pass
|
||||
async def __aenter__(self): return self
|
||||
async def __aexit__(self, *args): pass
|
||||
def stream(self, *args, **kwargs):
|
||||
requests.append(kwargs['json'])
|
||||
return Response(next(responses))
|
||||
|
||||
async def execute(block, **kwargs):
|
||||
executions.append(block)
|
||||
return 'manage_tasks', {'output': 'Created task task-1', 'exit_code': 0}
|
||||
|
||||
monkeypatch.setattr(module.httpx, 'AsyncClient', Client)
|
||||
monkeypatch.setattr(module, 'execute_tool_block', execute)
|
||||
schemas = [s for s in FUNCTION_TOOL_SCHEMAS if s['function']['name'] == 'manage_tasks']
|
||||
contract = resolve_full_inventory_contract(schemas=schemas, policy=ToolPolicy())
|
||||
raw = [chunk async for chunk in stream_preview(
|
||||
endpoint_url='http://test', model='ajax_c375',
|
||||
messages=[{'role': 'user', 'content': 'Create a daily task at 9am to summarize news.'}],
|
||||
headers={}, turn_contract=contract, session_id='test', owner='test',
|
||||
disabled_tools=set(), tool_policy=ToolPolicy(), max_rounds=4,
|
||||
)]
|
||||
assert len(executions) == 1, '\n'.join(raw)
|
||||
assert [r['chat_template_kwargs']['enable_thinking'] for r in requests] == [False, True, False]
|
||||
assert not any(m.get('tool_calls') for m in requests[1]['messages'])
|
||||
assert not any('event: error' in chunk for chunk in raw)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_calendar_list_structured_result_owns_linked_terminal_render(monkeypatch):
|
||||
import src.clean_agent_preview as module
|
||||
@@ -5427,7 +5527,9 @@ async def test_model_choice_budget_preserves_evidence_for_final_answer_without_e
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_failed_static_fetch_recovers_once_through_rendered_browser(monkeypatch):
|
||||
@pytest.mark.parametrize('thinking_mode', ['auto', 'off', 'on'])
|
||||
@pytest.mark.parametrize('token_limit', [None, 1024])
|
||||
async def test_failed_static_fetch_recovers_once_through_rendered_browser(monkeypatch, thinking_mode, token_limit):
|
||||
from dataclasses import replace
|
||||
import src.clean_agent_preview as module
|
||||
|
||||
@@ -5489,13 +5591,17 @@ async def test_failed_static_fetch_recovers_once_through_rendered_browser(monkey
|
||||
routing_experiment='recent_model_choice',
|
||||
)
|
||||
raw = [chunk async for chunk in stream_preview(
|
||||
endpoint_url='http://test', model='test',
|
||||
endpoint_url='http://test', model='ajax_c375', thinking_mode=thinking_mode,
|
||||
messages=[{'role': 'user', 'content': 'Tell me more about that Reuters report.'}],
|
||||
headers={}, turn_contract=contract, session_id='test', owner='test',
|
||||
disabled_tools=set(), tool_policy=ToolPolicy(), max_rounds=4,
|
||||
disabled_tools=set(), tool_policy=ToolPolicy(), max_rounds=4, max_tokens=token_limit,
|
||||
)]
|
||||
|
||||
assert executions == ['web_fetch', 'private_browser', 'web_fetch']
|
||||
thinking = [r['chat_template_kwargs']['enable_thinking'] for r in requests]
|
||||
# Compact owns the protocol, including for old sessions with On/Off saved.
|
||||
assert thinking == [False, True, True, False]
|
||||
assert requests[1]['max_tokens'] == (token_limit or 4096)
|
||||
assert all(
|
||||
schema['function']['name'] != 'web_fetch'
|
||||
for schema in requests[1].get('tools', [])
|
||||
@@ -5572,7 +5678,8 @@ async def test_duplicate_fetch_does_not_hide_distinct_pagination_request(monkeyp
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_blocked_search_engine_browser_forces_native_web_search(monkeypatch):
|
||||
@pytest.mark.parametrize('empty_challenge', [False, True])
|
||||
async def test_blocked_search_engine_browser_forces_native_web_search(monkeypatch, empty_challenge):
|
||||
import src.clean_agent_preview as module
|
||||
|
||||
responses = iter([
|
||||
@@ -5594,10 +5701,8 @@ async def test_blocked_search_engine_browser_forces_native_web_search(monkeypatc
|
||||
},
|
||||
}]}}]},
|
||||
{'choices': [{'delta': {'content': (
|
||||
'Recent AI developments include new model releases and policy updates. '
|
||||
'The search results identify the relevant primary sources and dates for each item, '
|
||||
'including publication details, concrete findings, and links readers can inspect '
|
||||
'for the full context behind each development.'
|
||||
'I found the [AI update](https://example.com/ai) page. '
|
||||
'The browser search was blocked, so I used native search to locate it.'
|
||||
)}}]},
|
||||
])
|
||||
requests = []
|
||||
@@ -5626,6 +5731,9 @@ async def test_blocked_search_engine_browser_forces_native_web_search(monkeypatc
|
||||
if block.tool_type == 'private_browser':
|
||||
return 'private_browser', {
|
||||
'output': json.dumps({
|
||||
'url': 'https://www.google.com/sorry/index?continue=search',
|
||||
'title': 'Client Challenge', 'snapshot': '(empty page)',
|
||||
} if empty_challenge else {
|
||||
'url': 'https://www.google.com/sorry/index?continue=search',
|
||||
'text': 'Our systems have detected unusual traffic. Complete the reCAPTCHA.',
|
||||
}),
|
||||
@@ -5651,7 +5759,7 @@ async def test_blocked_search_engine_browser_forces_native_web_search(monkeypatc
|
||||
)
|
||||
raw = [chunk async for chunk in stream_preview(
|
||||
endpoint_url='http://test', model='test',
|
||||
messages=[{'role': 'user', 'content': 'Open browser and find an AI model release.'}],
|
||||
messages=[{'role': 'user', 'content': 'Open browser and find the AI update page on example.com. Give me its link.'}],
|
||||
headers={}, turn_contract=contract, session_id='test', owner='test',
|
||||
disabled_tools=set(), tool_policy=ToolPolicy(), max_rounds=4,
|
||||
)]
|
||||
@@ -5659,6 +5767,7 @@ async def test_blocked_search_engine_browser_forces_native_web_search(monkeypatc
|
||||
assert executions == ['private_browser', 'web_search']
|
||||
assert requests[1]['tool_choice'] == 'required'
|
||||
assert [s['function']['name'] for s in requests[1]['tools']] == ['web_search']
|
||||
assert not any(chunk.startswith('event: error') for chunk in raw), raw
|
||||
events = [json.loads(chunk[6:]) for chunk in raw if '[DONE]' not in chunk]
|
||||
assert any(
|
||||
event.get('type') == 'tool_loop_recovery'
|
||||
@@ -5721,9 +5830,10 @@ async def test_native_stream_reserves_remaining_budget_for_required_artifact(mon
|
||||
|
||||
async def execute(block, **kwargs):
|
||||
executed.append(block.tool_type)
|
||||
return block.tool_type, {"output": "ok", "exit_code": 0,
|
||||
"materialized_artifacts": ["/tmp_workspace/results"]
|
||||
if block.tool_type == "python" and "out.md" in block.content else []}
|
||||
if block.tool_type == 'python' and '/tmp_workspace/results/out.md' in block.content:
|
||||
return block.tool_type, {'output': 'ok', 'exit_code': 0,
|
||||
'materialized_artifacts': ['/tmp_workspace/results']}
|
||||
return block.tool_type, {"output": "ok", "exit_code": 0}
|
||||
|
||||
monkeypatch.setattr(module.httpx, "AsyncClient", Client)
|
||||
monkeypatch.setattr(module, "execute_tool_block", execute)
|
||||
@@ -5835,9 +5945,10 @@ async def test_native_stream_reserves_wall_time_for_required_artifact(monkeypatc
|
||||
executed.append(block.tool_type)
|
||||
if block.tool_type == "web_search":
|
||||
now[0] = 450.0
|
||||
return block.tool_type, {"output": "ok", "exit_code": 0,
|
||||
"materialized_artifacts": ["/tmp_workspace/results"]
|
||||
if block.tool_type == "python" and "out.md" in block.content else []}
|
||||
if block.tool_type == 'python' and '/tmp_workspace/results/out.md' in block.content:
|
||||
return block.tool_type, {'output': 'ok', 'exit_code': 0,
|
||||
'materialized_artifacts': ['/tmp_workspace/results']}
|
||||
return block.tool_type, {"output": "ok", "exit_code": 0}
|
||||
|
||||
monkeypatch.setattr(module.time, "monotonic", lambda: now[0])
|
||||
monkeypatch.setattr(module.httpx, "AsyncClient", Client)
|
||||
@@ -6429,14 +6540,15 @@ async def test_native_stream_permanently_withholds_tool_when_model_ignores_dupli
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_native_stream_terminates_after_calling_a_permanently_suppressed_tool(monkeypatch):
|
||||
@pytest.mark.parametrize('ignores_final_instruction', [False, True])
|
||||
async def test_native_stream_terminates_after_calling_a_permanently_suppressed_tool(monkeypatch, ignores_final_instruction):
|
||||
import src.clean_agent_preview as module
|
||||
arguments = json.dumps({"path": "/workspace/fixtures/video.mp4"})
|
||||
responses = iter([
|
||||
{"choices": [{"delta": {"tool_calls": [{"index": 0, "id": f"inspect-{index}", "function": {
|
||||
"name": "inspect_media", "arguments": arguments,
|
||||
}}]}}]}
|
||||
for index in range(1, 4)
|
||||
for index in range(1, 5 if ignores_final_instruction else 4)
|
||||
] + [{"choices": [{"delta": {"content": "Final answer from existing evidence."}}]}])
|
||||
|
||||
class Response:
|
||||
@@ -6488,9 +6600,15 @@ async def test_native_stream_terminates_after_calling_a_permanently_suppressed_t
|
||||
assert 'tools' not in requests[3]
|
||||
assert 'best concise final answer' in requests[3]['messages'][-1]['content'].lower()
|
||||
final = [event for event in events if event.get("type") == "final_response"]
|
||||
assert final == []
|
||||
metrics = next(event['data'] for event in events if event.get('type') == 'metrics')
|
||||
assert metrics['clean_v3_turn'][-1]['content'] == 'Final answer from existing evidence.'
|
||||
if ignores_final_instruction:
|
||||
assert len(final) == 1
|
||||
assert 'model repeated a tool call' in final[0]['content']
|
||||
assert metrics['clean_v3_turn'][-1]['content'] == final[0]['content']
|
||||
else:
|
||||
assert final == []
|
||||
assert metrics['clean_v3_turn'][-1]['content'] == 'Final answer from existing evidence.'
|
||||
assert raw[-1] == 'data: [DONE]\n\n'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -7284,3 +7402,275 @@ async def test_tool_round_leadin_is_replaced_by_terminal_synthesis(monkeypatch):
|
||||
"I'll browse IKEA and look at their chairs.",
|
||||
'The most epic option is the DYVLINGE swivel chair.',
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_denied_call_after_successful_evidence_recovers_instead_of_refusing(monkeypatch):
|
||||
"""A blocked call must not discard evidence an earlier call already produced."""
|
||||
import src.clean_agent_preview as module
|
||||
|
||||
def call(index, name, arguments):
|
||||
return {'index': index, 'id': f'call-{index}', 'function': {
|
||||
'name': name, 'arguments': json.dumps(arguments),
|
||||
}}
|
||||
|
||||
packets = iter([
|
||||
{'choices': [{'delta': {'tool_calls': [
|
||||
call(0, 'web_search', {'query': 'EEG headset prices'})]}}]},
|
||||
{'choices': [{'delta': {'tool_calls': [
|
||||
call(0, 'web_fetch', {'url': 'https://vendor.example/eeg'})]}}]},
|
||||
{'choices': [{'delta': {'content': (
|
||||
'Based on the search result, the EEG headset starts at $299 from the '
|
||||
'vendor. I could not open the vendor page because that operation was '
|
||||
'not permitted.'
|
||||
)}}]},
|
||||
])
|
||||
requests = []
|
||||
|
||||
class Response:
|
||||
def __init__(self, payload): self.payload = payload
|
||||
async def __aenter__(self): return self
|
||||
async def __aexit__(self, *args): pass
|
||||
def raise_for_status(self): pass
|
||||
async def aiter_lines(self):
|
||||
yield 'data: ' + json.dumps(self.payload)
|
||||
yield 'data: [DONE]'
|
||||
|
||||
class Client:
|
||||
def __init__(self, **kwargs): pass
|
||||
async def __aenter__(self): return self
|
||||
async def __aexit__(self, *args): pass
|
||||
def stream(self, *args, **kwargs):
|
||||
requests.append(kwargs['json'])
|
||||
return Response(next(packets))
|
||||
|
||||
async def execute(block, **kwargs):
|
||||
if block.tool_type == 'web_search':
|
||||
return 'web_search', {
|
||||
'output': '[1] EEG Headset Prices\n https://vendor.example/eeg',
|
||||
'exit_code': 0,
|
||||
'evidence_status': 'available',
|
||||
}
|
||||
return 'web_fetch', {
|
||||
'output': 'The exact operation is outside server request authority.',
|
||||
'exit_code': 1,
|
||||
'blocked': True,
|
||||
'failure_kind': 'request_authority_denied',
|
||||
}
|
||||
|
||||
monkeypatch.setattr(module.httpx, 'AsyncClient', Client)
|
||||
monkeypatch.setattr(module, 'execute_tool_block', execute)
|
||||
schemas = [
|
||||
schema for schema in FUNCTION_TOOL_SCHEMAS
|
||||
if schema['function']['name'] in {'web_search', 'web_fetch'}
|
||||
]
|
||||
contract = resolve_full_inventory_contract(schemas=schemas, policy=ToolPolicy())
|
||||
raw = [chunk async for chunk in stream_preview(
|
||||
endpoint_url='http://test', model='test',
|
||||
messages=[{'role': 'user', 'content': (
|
||||
'Search the web for the latest EEG headset prices and open the vendor page.'
|
||||
)}],
|
||||
headers={}, turn_contract=contract, session_id='test', owner='test',
|
||||
disabled_tools=set(), tool_policy=ToolPolicy(), max_rounds=5,
|
||||
)]
|
||||
events = [json.loads(chunk[6:]) for chunk in raw if '[DONE]' not in chunk]
|
||||
assert any(
|
||||
event.get('type') == 'completion_recovery'
|
||||
and event.get('reason') == 'denial_after_successful_evidence'
|
||||
for event in events
|
||||
)
|
||||
visible = ''.join(
|
||||
str(e.get('delta') or e.get('content') or '') for e in events
|
||||
)
|
||||
assert '$299' in visible
|
||||
assert denied_response() not in visible
|
||||
# The recovery round must synthesize without tools.
|
||||
assert requests[-1].get('tools') in (None, [])
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize('thinking_mode, expected_thinking', [
|
||||
('on', {'type': 'enabled'}),
|
||||
(None, {'type': 'enabled'}),
|
||||
('off', {'type': 'disabled'}),
|
||||
])
|
||||
async def test_deepseek_v4_preview_reserves_reasoning_budget_and_honors_toggle(
|
||||
monkeypatch, thinking_mode, expected_thinking,
|
||||
):
|
||||
import src.clean_agent_preview as module
|
||||
|
||||
requests = []
|
||||
|
||||
class Response:
|
||||
def __init__(self, payload): self.payload = payload
|
||||
async def __aenter__(self): return self
|
||||
async def __aexit__(self, *args): pass
|
||||
def raise_for_status(self): pass
|
||||
async def aiter_lines(self):
|
||||
yield 'data: ' + json.dumps(self.payload)
|
||||
yield 'data: [DONE]'
|
||||
|
||||
class Client:
|
||||
def __init__(self, **kwargs): pass
|
||||
async def __aenter__(self): return self
|
||||
async def __aexit__(self, *args): pass
|
||||
def stream(self, *args, **kwargs):
|
||||
requests.append(kwargs['json'])
|
||||
return Response({'choices': [{'delta': {
|
||||
'reasoning_content': 'Let me reason about this.',
|
||||
'content': 'The answer is 42.',
|
||||
}}]})
|
||||
|
||||
monkeypatch.setattr(module.httpx, 'AsyncClient', Client)
|
||||
contract = resolve_full_inventory_contract(schemas=[], policy=ToolPolicy())
|
||||
kwargs = {} if thinking_mode is None else {'thinking_mode': thinking_mode}
|
||||
raw = [chunk async for chunk in stream_preview(
|
||||
endpoint_url='https://api.deepseek.com/v1/chat/completions',
|
||||
model='deepseek-v4-flash',
|
||||
messages=[{'role': 'user', 'content': 'What is the answer?'}],
|
||||
headers={}, turn_contract=contract, session_id='test', owner='test',
|
||||
disabled_tools=set(), tool_policy=ToolPolicy(), **kwargs,
|
||||
)]
|
||||
|
||||
assert requests, 'a provider request should have been sent'
|
||||
sent = requests[0]
|
||||
if expected_thinking == {'type': 'disabled'}:
|
||||
assert sent.get('reasoning_effort') == 'none'
|
||||
else:
|
||||
# Reasoning-capable models must reserve budget above the 768 default.
|
||||
assert sent['max_tokens'] >= 4096
|
||||
assert sent.get('reasoning_effort')
|
||||
assert sent.get('thinking') == expected_thinking
|
||||
assert any('The answer is 42.' in chunk for chunk in raw)
|
||||
|
||||
|
||||
def test_workspace_process_denial_uses_an_honest_refusal(tmp_path):
|
||||
from src.clean_agent_preview import (
|
||||
workspace_process_denial,
|
||||
WORKSPACE_REQUIRED_REFUSAL,
|
||||
denied_response,
|
||||
)
|
||||
# A resource-identity / sealed-workspace denial is recognised from any of
|
||||
# the tool-event fields (error / output JSON / failure_kind).
|
||||
assert workspace_process_denial([
|
||||
{'tool': 'bash', 'exit_code': 1, 'blocked': True,
|
||||
'output': '{"error":"Process creation requires a sealed workspace and launch scope",'
|
||||
'"failure_kind":"resource_identity_denied"}'},
|
||||
])
|
||||
assert workspace_process_denial([{'failure_kind': 'resource_identity_denied'}])
|
||||
# A plain request-authority denial is not a workspace denial.
|
||||
assert not workspace_process_denial([
|
||||
{'tool': 'bash', 'blocked': True, 'failure_kind': 'request_authority_denied',
|
||||
'error': 'The exact operation is outside server request authority.'},
|
||||
])
|
||||
assert not workspace_process_denial([])
|
||||
assert 'workspace' in WORKSPACE_REQUIRED_REFUSAL.lower()
|
||||
assert WORKSPACE_REQUIRED_REFUSAL != denied_response()
|
||||
# The terminal-denial branch picks the workspace message when appropriate.
|
||||
source = (Path(__file__).resolve().parents[1] / 'src/clean_agent_preview.py').read_text()
|
||||
assert 'if workspace_process_denial(executions)' in source
|
||||
assert 'WORKSPACE_REQUIRED_REFUSAL' in source
|
||||
|
||||
|
||||
def test_spec_router_think_decision_and_reason(monkeypatch):
|
||||
import sys
|
||||
import types
|
||||
from src.clean_agent_preview import (
|
||||
spec_router_thinking_decision,
|
||||
spec_router_think_reason,
|
||||
)
|
||||
|
||||
# --- router disabled -> no decision, no reason (default OFF) -------------
|
||||
fake = types.ModuleType('src.spec_router')
|
||||
fake.think_router_enabled = lambda: False
|
||||
fake.route_thinking = lambda text: {'think': True, 'p_think': 0.9, 'ms': 1.0}
|
||||
monkeypatch.setitem(sys.modules, 'src.spec_router', fake)
|
||||
assert spec_router_thinking_decision('u', 'm', 'o', {}, 'hi') is None
|
||||
assert spec_router_think_reason(1, None, '') == ''
|
||||
|
||||
# --- enabled but not a spec surface -> no decision ----------------------
|
||||
fake.think_router_enabled = lambda: True
|
||||
monkeypatch.setattr(
|
||||
'src.agent_loop._configured_model_tool_surface_raw',
|
||||
lambda *a, **k: 'native', raising=False,
|
||||
)
|
||||
assert spec_router_thinking_decision('u', 'm', 'o', {}, 'hi') is None
|
||||
|
||||
# --- enabled on spec + think=True -> reason only on round 1 when no other
|
||||
monkeypatch.setattr(
|
||||
'src.agent_loop._configured_model_tool_surface_raw',
|
||||
lambda *a, **k: 'spec', raising=False,
|
||||
)
|
||||
decision = spec_router_thinking_decision('u', 'm', 'o', {}, 'plan the refactor')
|
||||
assert decision == {'think': True, 'p_think': 0.9, 'ms': 1.0}
|
||||
assert spec_router_think_reason(1, decision, '') == 'router_think'
|
||||
# existing rule wins; only round 1; think=False never fires
|
||||
assert spec_router_think_reason(1, decision, 'code') == ''
|
||||
assert spec_router_think_reason(2, decision, '') == ''
|
||||
assert spec_router_think_reason(1, {'think': False}, '') == ''
|
||||
|
||||
# --- route_thinking raises -> fail open (None), rules only --------------
|
||||
def _boom(text):
|
||||
raise RuntimeError('router down')
|
||||
fake.route_thinking = _boom
|
||||
assert spec_router_thinking_decision('u', 'm', 'o', {}, 'hi') is None
|
||||
assert spec_router_think_reason(1, None, '') == ''
|
||||
|
||||
# --- wiring present in stream_preview -----------------------------------
|
||||
source = (Path(__file__).resolve().parents[1] / 'src/clean_agent_preview.py').read_text()
|
||||
assert 'spec_router_thinking_decision(' in source
|
||||
assert 'spec_router_think_reason(' in source
|
||||
|
||||
|
||||
def test_qwen_think_split_streams_thinking_then_answer_without_buffering():
|
||||
from src.clean_agent_preview import qwen_think_split_events
|
||||
|
||||
state = {'think_emitted': 0, 'answer_emitted': 0, 'closed': False}
|
||||
events = []
|
||||
content = ''
|
||||
for piece in ['<think>', 'abc', '</think>', 'Hello']:
|
||||
content += piece
|
||||
events.extend(qwen_think_split_events(content, state))
|
||||
|
||||
thinking = ''.join(e['delta'] for e in events if e.get('thinking'))
|
||||
answer = ''.join(e['delta'] for e in events if not e.get('thinking'))
|
||||
assert thinking == 'abc'
|
||||
assert answer == 'Hello'
|
||||
# every emitted delta is non-empty and typed
|
||||
assert all(e.get('delta') for e in events)
|
||||
assert [e.get('thinking') for e in events] == [True, None]
|
||||
|
||||
# the stored answer stays identical to visible_content_after_qwen_thinking
|
||||
from src.clean_agent_preview import visible_content_after_qwen_thinking
|
||||
assert visible_content_after_qwen_thinking(content) == 'Hello'
|
||||
|
||||
|
||||
def test_qwen_think_split_holds_back_a_partial_close_tag():
|
||||
from src.clean_agent_preview import qwen_think_split_events
|
||||
|
||||
state = {'think_emitted': 0, 'answer_emitted': 0, 'closed': False}
|
||||
events = []
|
||||
content = ''
|
||||
for piece in ['<think>a', '</thi', 'nk>', 'B']:
|
||||
content += piece
|
||||
events.extend(qwen_think_split_events(content, state))
|
||||
|
||||
thinking = ''.join(e['delta'] for e in events if e.get('thinking'))
|
||||
answer = ''.join(e['delta'] for e in events if not e.get('thinking'))
|
||||
assert thinking == 'a' # the partial '</thi' is never leaked
|
||||
assert answer == 'B'
|
||||
assert all(e.get('delta') for e in events)
|
||||
|
||||
|
||||
def test_qwen_think_split_is_inert_without_a_think_block():
|
||||
from src.clean_agent_preview import qwen_think_split_events
|
||||
|
||||
state = {'think_emitted': 0, 'answer_emitted': 0, 'closed': False}
|
||||
assert qwen_think_split_events('plain answer', state) == []
|
||||
assert qwen_think_split_events('', state) == []
|
||||
assert state == {'think_emitted': 0, 'answer_emitted': 0, 'closed': False}
|
||||
|
||||
# source-level: stream_preview wires the incremental split
|
||||
source = (Path(__file__).resolve().parents[1] / 'src/clean_agent_preview.py').read_text()
|
||||
assert 'qwen_think_split_events(' in source
|
||||
assert "think_answer_streamed" in source
|
||||
|
||||
@@ -17,6 +17,42 @@ ERROR = 'event: error\ndata: {"status": 504, "error": {"message": "stream timeou
|
||||
DONE = 'data: [DONE]\n\n'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_answer_deltas_are_visible_before_terminal_completion_check():
|
||||
finish = asyncio.Event()
|
||||
|
||||
@with_completion_gate
|
||||
async def stream(messages):
|
||||
yield _event({'delta': 'Live answer starts. '})
|
||||
await finish.wait()
|
||||
yield _event({'type': 'metrics', 'data': {}})
|
||||
yield DONE
|
||||
|
||||
output = stream([])
|
||||
first = await asyncio.wait_for(anext(output), timeout=0.2)
|
||||
assert json.loads(first[6:])['delta'] == 'Live answer starts. '
|
||||
finish.set()
|
||||
remaining = [chunk async for chunk in output]
|
||||
assert _decision([first, *remaining])['can_complete'] is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_completion_replacement_replaces_already_streamed_answer_for_turn():
|
||||
@with_completion_gate
|
||||
async def stream(messages):
|
||||
yield _event({'delta': 'The function returns a boolean. I ran pytest and all tests passed.'})
|
||||
yield _event({'type': 'metrics', 'data': {}})
|
||||
yield DONE
|
||||
|
||||
chunks = [chunk async for chunk in stream([{'role': 'user', 'content': 'Explain this code.'}])]
|
||||
frames = [data for event, data in _frames(chunks) if event == 'message' and isinstance(data, dict)]
|
||||
assert any(item.get('delta') == 'The function returns a boolean.' for item in frames)
|
||||
assert not any('I ran pytest' in str(item.get('delta', '')) for item in frames)
|
||||
replacement = next(item for item in frames if item.get('type') == 'final_response')
|
||||
assert 'I ran pytest' not in replacement['content']
|
||||
assert replacement['replacement_scope'] == 'turn'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize('terminal_texts', [[], ['', ''], ['Recovered answer with literal [DONE] text.']])
|
||||
async def test_terminal_round_retraction_does_not_resurrect_buffered_drafts(terminal_texts):
|
||||
|
||||
@@ -338,8 +338,8 @@ def test_hardlinked_effect_state_is_control_plane_without_scanning_the_store(tmp
|
||||
assert str(store) not in listed
|
||||
assert resources._control_plane_path(str(alias), snapshot=snapshot) is True
|
||||
assert str(store) not in globbed, "the effect store is listed one level, never recursively inventoried"
|
||||
# Multi-linked regular files are treated as control-plane resources to prevent hardlink race/escape behavior.
|
||||
# Multiply linked files elsewhere stay ordinary.
|
||||
elsewhere = tmp_path / "other.txt"
|
||||
elsewhere.write_text("y")
|
||||
os.link(elsewhere, workspace / "pnpm-style.txt")
|
||||
assert resources._control_plane_path(str(workspace / "pnpm-style.txt")) is True
|
||||
assert resources._control_plane_path(str(workspace / "pnpm-style.txt")) is False
|
||||
|
||||
@@ -148,6 +148,7 @@ def _chat_stream_endpoint(
|
||||
endpoint_url="https://selected.example/v1",
|
||||
session_model="selected-model",
|
||||
session_history=(),
|
||||
attachment_meta=(),
|
||||
):
|
||||
def add_message(message):
|
||||
captured.setdefault("added_messages", []).append(message)
|
||||
@@ -172,7 +173,7 @@ def _chat_stream_endpoint(
|
||||
{"role": "assistant", "content": "old answer"},
|
||||
{"role": "user", "content": "hello"},
|
||||
],
|
||||
preprocessed=SimpleNamespace(attachment_meta=[]),
|
||||
preprocessed=SimpleNamespace(attachment_meta=list(attachment_meta)),
|
||||
auto_opened_docs=[],
|
||||
rag_sources=[],
|
||||
web_sources=[],
|
||||
|
||||
@@ -7,6 +7,7 @@ os.environ.setdefault("DATABASE_URL", "sqlite:///:memory:")
|
||||
import pytest
|
||||
|
||||
from src.llm_core import _supports_thinking
|
||||
from src.model_profiles import supports_user_thinking_toggle
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
@@ -25,3 +26,16 @@ def test_deepseek_v4_models_support_thinking(model):
|
||||
@pytest.mark.parametrize("model", ["deepseek-v3", "deepseek-chat"])
|
||||
def test_other_deepseek_models_are_not_promoted_to_thinking(model):
|
||||
assert _supports_thinking(model) is False
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"model",
|
||||
[
|
||||
"deepseek-v4",
|
||||
"deepseek-v4-flash",
|
||||
"DeepSeek-V4-Flash",
|
||||
"deepseek/deepseek-v4-flash",
|
||||
],
|
||||
)
|
||||
def test_deepseek_v4_models_expose_the_user_thinking_toggle(model):
|
||||
assert supports_user_thinking_toggle(model) is True
|
||||
|
||||
@@ -368,7 +368,7 @@ def test_detached_container_math_typesets_with_the_real_renderer(node_available)
|
||||
|
||||
mdToHtml defers math to a document-scoped flush, which cannot reach a
|
||||
detached node, so the export has to typeset its own container before
|
||||
handing it to browser printing. This is that container: pending spans in, real
|
||||
handing it to html2pdf. This is that container: pending spans in, real
|
||||
KaTeX markup out, no .katex-error and nothing left pending.
|
||||
"""
|
||||
out = _run_node(
|
||||
@@ -400,7 +400,7 @@ def test_detached_container_math_typesets_with_the_real_renderer(node_available)
|
||||
assert "ody-math-pending" not in out["written"]
|
||||
|
||||
|
||||
def test_pdf_export_typesets_its_container_before_print():
|
||||
def test_pdf_export_typesets_its_container_before_html2pdf():
|
||||
"""Ordering in a call site, so pin the call site. No node needed."""
|
||||
source = document_source()
|
||||
match = re.search(r"\n async function exportAsPdf\(\) \{(.*?)\n \}\n", source, re.S)
|
||||
@@ -410,7 +410,7 @@ def test_pdf_export_typesets_its_container_before_print():
|
||||
render = "await markdownModule.renderMath(container);"
|
||||
assert render in body, "the export never typesets its detached container"
|
||||
assert body.index("container.innerHTML = html;") < body.index(render)
|
||||
assert body.index(render) < body.index("frame.contentWindow.print()")
|
||||
assert body.index(render) < body.index("window.html2pdf()")
|
||||
|
||||
|
||||
def test_md_to_html_renders_inline_once_katex_is_loaded(node_available):
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
"""Model-picker backend: a slow/dead endpoint must not stall /api/models refresh.
|
||||
|
||||
Covers the fixes in routes/model_routes.py:
|
||||
- the local endpoint refresh timeout default is 2s (was 10s);
|
||||
- a recently failed/timed-out endpoint keeps a ~60s cooldown even on a
|
||||
user-forced refresh, so a forced refresh does not re-probe a dead endpoint
|
||||
on every picker open;
|
||||
- probing is parallel, so one slow endpoint does not push the whole refresh
|
||||
past its own timeout.
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
import types
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from tests.helpers.import_state import (
|
||||
clear_fake_endpoint_resolver_modules,
|
||||
preserve_import_state,
|
||||
)
|
||||
|
||||
|
||||
def _load_model_routes():
|
||||
with preserve_import_state(
|
||||
"core.database", "src.database", "core.session_manager", "routes.model_routes"
|
||||
):
|
||||
clear_fake_endpoint_resolver_modules()
|
||||
if "core.database" not in sys.modules:
|
||||
stub = types.ModuleType("core.database")
|
||||
for name in (
|
||||
"SessionLocal", "ModelEndpoint", "Session", "ChatMessage", "Document",
|
||||
"DocumentVersion", "GalleryImage", "GalleryAlbum", "Note", "CalendarCal",
|
||||
"CalendarEvent", "ScheduledTask", "TaskRun", "McpServer",
|
||||
"ProviderAuthSession", "Base",
|
||||
):
|
||||
setattr(stub, name, MagicMock())
|
||||
stub.utcnow_naive = MagicMock()
|
||||
sys.modules["core.database"] = stub
|
||||
import routes.model_routes as model_routes
|
||||
return model_routes
|
||||
|
||||
|
||||
def _endpoint(**kw):
|
||||
base = dict(
|
||||
id="ep", base_url="http://127.0.0.1:9/v1", api_key="k",
|
||||
provider_auth_id=None, cached_models="[]", pinned_models=[],
|
||||
is_enabled=True, model_refresh_mode="auto",
|
||||
)
|
||||
base.update(kw)
|
||||
return SimpleNamespace(**base)
|
||||
|
||||
|
||||
def test_local_refresh_timeout_is_two_seconds():
|
||||
model_routes = _load_model_routes()
|
||||
ep = _endpoint(base_url="http://100.67.207.85:8071/v1")
|
||||
assert model_routes._endpoint_refresh_timeout(ep, "local") == 2.0
|
||||
assert model_routes._endpoint_refresh_timeout(ep, "remote") == 2.0
|
||||
|
||||
|
||||
def test_recent_failure_is_not_reprobed_even_when_forced():
|
||||
model_routes = _load_model_routes()
|
||||
router = model_routes.setup_model_routes(model_discovery=None)
|
||||
should = router._should_refresh_endpoint
|
||||
ep = _endpoint(id="remote-dead", base_url="http://100.67.207.85:8071/v1")
|
||||
|
||||
now = time.time()
|
||||
ok, info = should(ep, now, force=True)
|
||||
assert ok is True and info.get("base")
|
||||
key = info["key"]
|
||||
|
||||
# A failure one second ago keeps a 60s cooldown even on a forced refresh.
|
||||
router._refresh_state[key] = {
|
||||
"fail_count": 1, "last_failure": now - 1.0, "inflight": False,
|
||||
}
|
||||
ok_recent, _ = should(ep, time.time(), force=True)
|
||||
assert ok_recent is False
|
||||
|
||||
# A failure long past the 60s floor is re-probed again.
|
||||
router._refresh_state[key] = {
|
||||
"fail_count": 1, "last_failure": now - 120.0, "inflight": False,
|
||||
}
|
||||
ok_old, _ = should(ep, time.time(), force=True)
|
||||
assert ok_old is True
|
||||
|
||||
|
||||
class _FakeQuery:
|
||||
def __init__(self, rows, first_value):
|
||||
self._rows = rows
|
||||
self._first = first_value
|
||||
|
||||
def filter(self, *args, **kwargs):
|
||||
return self
|
||||
|
||||
def all(self):
|
||||
return list(self._rows)
|
||||
|
||||
def first(self):
|
||||
return self._first
|
||||
|
||||
|
||||
class _FakeDB:
|
||||
def __init__(self, rows, first_value):
|
||||
self._rows = rows
|
||||
self._first = first_value
|
||||
self.committed = False
|
||||
|
||||
def query(self, model):
|
||||
return _FakeQuery(self._rows, self._first)
|
||||
|
||||
def commit(self):
|
||||
self.committed = True
|
||||
|
||||
def close(self):
|
||||
pass
|
||||
|
||||
|
||||
def test_one_slow_endpoint_does_not_stall_the_forced_refresh(monkeypatch):
|
||||
model_routes = _load_model_routes()
|
||||
router = model_routes.setup_model_routes(model_discovery=None)
|
||||
|
||||
healthy = _endpoint(id="healthy", base_url="http://127.0.0.1:7021/v1")
|
||||
dead = _endpoint(id="dead", base_url="http://127.0.0.1:9/v1")
|
||||
|
||||
monkeypatch.setattr(model_routes, "SessionLocal", lambda: _FakeDB([healthy, dead], healthy))
|
||||
monkeypatch.setattr(
|
||||
model_routes, "_disable_stale_cookbook_local_endpoints", lambda db: False
|
||||
)
|
||||
|
||||
def _fake_probe(base, api_key=None, timeout=5):
|
||||
if "127.0.0.1:9" in base:
|
||||
time.sleep(min(float(timeout or 2), 2.0))
|
||||
raise RuntimeError("timed out")
|
||||
return ["healthy-model"]
|
||||
|
||||
monkeypatch.setattr(model_routes, "_probe_endpoint", _fake_probe)
|
||||
|
||||
started = time.time()
|
||||
done = router._refresh_caches_bg(force=True)
|
||||
assert done is not None
|
||||
assert done.wait(5) is True
|
||||
elapsed = time.time() - started
|
||||
|
||||
assert elapsed <= 3.0, f"forced refresh took {elapsed:.2f}s"
|
||||
assert json.loads(healthy.cached_models) == ["healthy-model"]
|
||||
@@ -8,6 +8,6 @@ ROOT = Path(__file__).resolve().parents[1]
|
||||
def test_pdf_backed_documents_do_not_offer_destructive_html_pdf_export():
|
||||
source = document_source()
|
||||
|
||||
assert "if (!isForm && (_isDocxLang(lang) || typeof window.print === 'function')) {" in source
|
||||
assert "label: _isDocxLang(lang) ? 'Convert to PDF' : 'Print / save PDF'" in source
|
||||
assert "if (!isForm) {" in source
|
||||
assert "label: _isDocxLang(lang) ? 'Convert to PDF' : 'Print as PDF'" in source
|
||||
assert "destroy the original page layout, images, and form structure" in source
|
||||
|
||||
@@ -240,7 +240,8 @@ def test_non_qwen_fallback_keeps_requested_temperature(monkeypatch):
|
||||
|
||||
|
||||
def test_qwen_notes_fallback_reenables_personal_managers(monkeypatch):
|
||||
"""Model/route fallback cannot bypass caller-disabled tools; denials persist."""
|
||||
"""The answering candidate's notes mode must unblock the managers for
|
||||
execution, not just enable them in its own route schemas."""
|
||||
|
||||
_install_route_probe(monkeypatch)
|
||||
stream_round = 0
|
||||
@@ -298,9 +299,8 @@ def test_qwen_notes_fallback_reenables_personal_managers(monkeypatch):
|
||||
)
|
||||
)
|
||||
|
||||
# Caller hard denials are absolute: route fallback cannot re-enable disabled tools.
|
||||
assert {"manage_notes", "manage_calendar", "manage_tasks"}.issubset(
|
||||
seen_exec["disabled_tools"]
|
||||
assert seen_exec["disabled_tools"].isdisjoint(
|
||||
{"manage_notes", "manage_calendar", "manage_tasks"}
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -1,7 +1,11 @@
|
||||
"""Regression guard for the README title presentation.
|
||||
|
||||
The original ASCII-art banner needed a code fence to preserve whitespace.
|
||||
Keep the wordmark title identity and the historical ASCII-fencing guard.
|
||||
Originally (#1390) the README opened with an ASCII-art banner that had to live
|
||||
inside a ``` code fence, otherwise GitHub's markdown collapsed its leading
|
||||
whitespace and box-drawing rules and rendered it misaligned. The README refresh
|
||||
(#4306) dropped that banner in favour of a centered wordmark image, so the guard
|
||||
now pins the wordmark identity instead, while still catching the original failure
|
||||
mode if an un-fenced ASCII banner is ever reintroduced.
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
@@ -19,6 +23,8 @@ def _fenced_segments(text: str):
|
||||
|
||||
|
||||
def test_readme_opens_with_wordmark_title():
|
||||
# The README must still open with a recognizable Odysseus title: now the
|
||||
# centered wordmark image rather than an H1 / ASCII banner.
|
||||
head = "\n".join(README.read_text(encoding="utf-8").splitlines()[:15])
|
||||
assert 'alt="Odysseus"' in head, "README must open with the Odysseus wordmark image"
|
||||
|
||||
|
||||
@@ -64,12 +64,57 @@ def test_explicit_request_classes_remain_narrow(user_text, allowed, denied):
|
||||
assert not grant.permits(ExactOperation.normalize(denied, '{}'))
|
||||
|
||||
|
||||
def test_search_selection_keeps_the_sibling_web_reader_authorized():
|
||||
# "Search the web ..." narrows selection to web_search alone, but the turn
|
||||
# must be able to read a URL the search surfaces; otherwise every result is
|
||||
# a dead end and the turn fails closed with request_authority_denied.
|
||||
grant = create_request_authority(
|
||||
"Search the web for the latest 2025 consumer EEG headset prices "
|
||||
"under $500 and cite sources")
|
||||
assert grant.permits(ExactOperation.normalize("web_search", '{"query":"eeg"}'))
|
||||
assert grant.permits(
|
||||
ExactOperation.normalize("web_fetch", '{"url":"https://example.com/report"}'))
|
||||
|
||||
|
||||
def test_fetch_only_selection_does_not_grant_search():
|
||||
grant = create_request_authority("Fetch https://example.com and summarize it")
|
||||
assert grant.permits(
|
||||
ExactOperation.normalize("web_fetch", '{"url":"https://example.com"}'))
|
||||
assert not grant.permits(ExactOperation.normalize("web_search", '{"query":"x"}'))
|
||||
|
||||
|
||||
def test_implicit_editor_revision_authority_requires_active_document():
|
||||
prompt = ('ga mer ut pa att vi tanker pa framtid for bjorn och att vi faktist '
|
||||
'trivs sa bra har med framtid planerad')
|
||||
edit = ExactOperation.normalize("edit_document", '{"edits":[]}')
|
||||
|
||||
with_editor = create_request_authority(prompt, active_document=True)
|
||||
without_editor = create_request_authority(prompt, active_document=False)
|
||||
|
||||
assert with_editor.permits(edit)
|
||||
assert not without_editor.permits(edit)
|
||||
|
||||
|
||||
def test_safe_task_read_does_not_authorize_same_tool_mutation():
|
||||
grant = create_request_authority("List my tasks")
|
||||
assert grant.permits(ExactOperation.normalize("manage_tasks", '{"action":"list"}'))
|
||||
assert not grant.permits(ExactOperation.normalize("manage_tasks", '{"action":"create","prompt":"run bash"}'))
|
||||
|
||||
|
||||
def test_action_only_read_tolerates_benign_model_arguments():
|
||||
# "What do you remember about me?" pins manage_memory to action=list. The
|
||||
# model must not be denied merely for adding an innocuous limit/cursor, while
|
||||
# a different (mutating) action stays outside the read grant.
|
||||
grant = create_request_authority("What do you remember about me?")
|
||||
memory = next(g for g in grant.grants if g.tool == "manage_memory")
|
||||
assert memory.inputs is None
|
||||
assert grant.permits(ExactOperation.normalize("manage_memory", '{"action":"list"}'))
|
||||
assert grant.permits(
|
||||
ExactOperation.normalize("manage_memory", '{"action":"list","limit":10}'))
|
||||
assert not grant.permits(
|
||||
ExactOperation.normalize("manage_memory", '{"action":"delete","id":"x"}'))
|
||||
|
||||
|
||||
def test_exact_read_identifiers_cannot_be_changed_by_model():
|
||||
grant = create_request_authority("Read note id abc123")
|
||||
read = next(g for g in grant.grants if g.tool == "manage_notes")
|
||||
@@ -413,6 +458,42 @@ async def test_server_seeded_defaults_have_authority_but_legacy_rows_do_not_gain
|
||||
engine.dispose()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unsealed_housekeeping_row_is_replaced_with_an_authorized_copy(monkeypatch, tmp_path):
|
||||
"""A built-in row that predates the authority seal (NULL snapshot) must be
|
||||
replaced by a freshly seeded, authorized copy, not left unrunnable. This is
|
||||
the 'Email Tags' -> 'no matching server request authority' failure."""
|
||||
import core.database as database
|
||||
import routes.prefs_routes as preferences
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
from src.task_scheduler import TaskScheduler
|
||||
engine = create_engine(f"sqlite:///{tmp_path / 'tasks.db'}")
|
||||
database.Base.metadata.create_all(engine)
|
||||
sessions = sessionmaker(bind=engine)
|
||||
monkeypatch.setattr(database, "SessionLocal", sessions)
|
||||
monkeypatch.setattr(preferences, "_load_for_user", lambda owner: {})
|
||||
scheduler = TaskScheduler(session_manager=None)
|
||||
monkeypatch.setattr(scheduler, "ensure_assistant_defaults", AsyncMock())
|
||||
with sessions() as db:
|
||||
db.add(database.ScheduledTask(id="broken", owner="alice", name="Email Tags",
|
||||
task_type="action", action="check_email_urgency",
|
||||
status="active", request_authority_json=None))
|
||||
db.commit()
|
||||
await scheduler.ensure_defaults("alice")
|
||||
with sessions() as db:
|
||||
rows = db.query(database.ScheduledTask).filter(
|
||||
database.ScheduledTask.action == "check_email_urgency").all()
|
||||
assert len(rows) == 1, "the unsealed row must be replaced, not duplicated"
|
||||
task = rows[0]
|
||||
assert task.id != "broken"
|
||||
assert task.status == "active", "the user's enable choice must survive the reseed"
|
||||
grant = restore_task_authority(task.request_authority_json, task.prompt,
|
||||
task.task_type, task.action, owner=task.owner)
|
||||
assert grant.permits(task_operation(task.task_type, task.action, task.prompt))
|
||||
engine.dispose()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dispatch_binds_explicit_authority_for_nested_handler(monkeypatch):
|
||||
from src import tool_execution as execution
|
||||
|
||||
@@ -139,7 +139,7 @@ def run_terminal_branch(monkeypatch, native_calls, result, operation=None, *, ma
|
||||
_required_read_native_id=al._required_read_native_id,
|
||||
_dispatch_required_safe_read=al._dispatch_required_safe_read,
|
||||
_compute_final_metrics=lambda *a, **kw: {"tool_events": a[8]},
|
||||
session_id="fixture", disabled_tools=set(), _caller_hard_denials=frozenset(), tool_policy=policy,
|
||||
session_id="fixture", disabled_tools=set(), tool_policy=policy,
|
||||
owner="fixture", workspace=None, run_security=context, active_document=None,
|
||||
client_runtime_context=None, round_num=1, model="fixture-model",
|
||||
actual_endpoint_id="fixture-endpoint", actual_endpoint_label="fixture",
|
||||
|
||||
@@ -549,3 +549,34 @@ def test_host_shell_native_arguments_use_the_same_verifier_classification(comman
|
||||
encoded = json.dumps({'command': command, 'timeout': 120})
|
||||
assert command_is_test(encoded) is expected
|
||||
assert command_is_validation(encoded) is expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize('tool', ['create_document', 'update_document', 'edit_document'])
|
||||
def test_successful_document_write_supports_unnamed_mutation_claim(tool):
|
||||
ledger = EvidenceLedger.from_tool_events([
|
||||
{'tool': tool, 'command': 'fix grammar', 'exit_code': 0},
|
||||
])
|
||||
claim = 'I updated the document with the grammar fix.'
|
||||
answer, reason = completion_answer(claim, ledger, ledger.evaluate())
|
||||
assert claim in answer
|
||||
assert not reason
|
||||
|
||||
|
||||
def test_document_write_without_authoritative_receipt_cannot_support_claim():
|
||||
ledger = EvidenceLedger.from_tool_events([
|
||||
{'tool': 'update_document', 'command': 'fix grammar'},
|
||||
])
|
||||
claim = 'I updated the document with the grammar fix.'
|
||||
answer, reason = completion_answer(claim, ledger, ledger.evaluate())
|
||||
assert reason
|
||||
assert claim not in answer
|
||||
|
||||
|
||||
def test_document_evidence_does_not_support_named_file_claim():
|
||||
ledger = EvidenceLedger.from_tool_events([
|
||||
{'tool': 'update_document', 'command': 'fix grammar', 'exit_code': 0},
|
||||
])
|
||||
claim = 'I updated config.py.'
|
||||
answer, reason = completion_answer(claim, ledger, ledger.evaluate())
|
||||
assert reason
|
||||
assert claim not in answer
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
"""Session-load 404 regression (passover item #9).
|
||||
|
||||
Loading a chat session used to fire two benign 404s in the browser console:
|
||||
GET /api/research/status/<id> (no research for this session)
|
||||
GET /api/chat/stream_status/<id> (no active stream for this session)
|
||||
|
||||
Both are now a normal "empty" state: HTTP 200 with an explicit status
|
||||
("none" / "idle"). The clients already branch on the body status, so this is
|
||||
backward compatible (an older server's 404 is still tolerated).
|
||||
"""
|
||||
import asyncio
|
||||
import inspect
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from routes import chat_routes
|
||||
from routes import research_routes
|
||||
|
||||
|
||||
def _route_endpoint(router, path, method="GET"):
|
||||
for route in router.routes:
|
||||
if getattr(route, "path", "") != path:
|
||||
continue
|
||||
if method in getattr(route, "methods", set()):
|
||||
return route.endpoint
|
||||
raise AssertionError(f"{method} {path} not registered")
|
||||
|
||||
|
||||
def _req(user="alice"):
|
||||
return SimpleNamespace(state=SimpleNamespace(current_user=user))
|
||||
|
||||
|
||||
# --- /api/research/status/<id> -------------------------------------------
|
||||
|
||||
def test_research_status_returns_200_none_when_absent(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(research_routes, "DEEP_RESEARCH_DIR", str(tmp_path / "deep_research"))
|
||||
handler = MagicMock()
|
||||
handler._active_tasks = {}
|
||||
router = research_routes.setup_research_routes(handler)
|
||||
endpoint = _route_endpoint(router, "/api/research/status/{session_id}")
|
||||
|
||||
out = asyncio.run(endpoint(session_id="sess-abc", request=_req()))
|
||||
|
||||
assert out == {"status": "none"}
|
||||
|
||||
|
||||
def test_research_status_returns_status_when_present(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(research_routes, "DEEP_RESEARCH_DIR", str(tmp_path / "deep_research"))
|
||||
handler = MagicMock()
|
||||
handler._active_tasks = {"sess-abc": {"owner": "alice"}}
|
||||
handler.get_status.return_value = {"status": "running", "progress": {"phase": "searching"}}
|
||||
handler.get_source_state.return_value = None
|
||||
handler.get_source_coverage.return_value = None
|
||||
handler.get_navigation_trace.return_value = None
|
||||
handler.get_action_trace.return_value = None
|
||||
router = research_routes.setup_research_routes(handler)
|
||||
endpoint = _route_endpoint(router, "/api/research/status/{session_id}")
|
||||
|
||||
out = asyncio.run(endpoint(session_id="sess-abc", request=_req()))
|
||||
|
||||
assert out["status"] == "running"
|
||||
|
||||
|
||||
# --- /api/chat/stream_status/<id> ----------------------------------------
|
||||
|
||||
def _chat_stream_status_endpoint(monkeypatch):
|
||||
monkeypatch.setattr(chat_routes, "_verify_session_owner", lambda request, session_id: None)
|
||||
router = chat_routes.setup_chat_routes(
|
||||
MagicMock(), MagicMock(), MagicMock(), MagicMock(), MagicMock(), MagicMock()
|
||||
)
|
||||
return _route_endpoint(router, "/api/chat/stream_status/{session_id}")
|
||||
|
||||
|
||||
def test_stream_status_returns_200_idle_when_no_stream(monkeypatch):
|
||||
endpoint = _chat_stream_status_endpoint(monkeypatch)
|
||||
chat_routes._active_streams.pop("sess-idle", None)
|
||||
|
||||
out = asyncio.run(endpoint(SimpleNamespace(), "sess-idle"))
|
||||
|
||||
assert out == {"status": "idle"}
|
||||
|
||||
|
||||
def test_stream_status_returns_record_when_streaming(monkeypatch):
|
||||
endpoint = _chat_stream_status_endpoint(monkeypatch)
|
||||
chat_routes._active_streams["sess-live"] = {"status": "streaming", "mode": "agent"}
|
||||
try:
|
||||
out = asyncio.run(endpoint(SimpleNamespace(), "sess-live"))
|
||||
finally:
|
||||
chat_routes._active_streams.pop("sess-live", None)
|
||||
|
||||
assert out["status"] == "streaming"
|
||||
|
||||
|
||||
# --- client compatibility (source-level) ---------------------------------
|
||||
|
||||
def test_chat_watchdog_accepts_200_idle_body():
|
||||
src = inspect.getsource(chat_routes) # ensure chat_routes imported
|
||||
chatjs = open("static/js/chat.js", encoding="utf-8").read()
|
||||
# The stale-stream watchdog must key off the body status, not only a 404.
|
||||
assert "_noActiveStream" in chatjs
|
||||
assert "_probe.status !== 'streaming'" in chatjs
|
||||
assert src # keep flake happy
|
||||
|
||||
|
||||
def test_sessions_check_uses_body_status_not_ok_only():
|
||||
sj = open("static/js/sessions.js", encoding="utf-8").read()
|
||||
assert "info.status !== 'streaming'" in sj
|
||||
assert "status !== 'streaming'" in sj
|
||||
@@ -0,0 +1,33 @@
|
||||
"""Health audit: SSE client disconnects (e.g. /api/editor/bridge/events) surface
|
||||
as anyio.EndOfStream through Starlette's BaseHTTPMiddleware. The slow-request
|
||||
middleware must map that to a client-side 499, not a logged 500 traceback."""
|
||||
import asyncio
|
||||
|
||||
import anyio
|
||||
from starlette.requests import Request
|
||||
|
||||
from app import _SlowRequestLogMiddleware
|
||||
|
||||
|
||||
def _request(path):
|
||||
return Request({"type": "http", "method": "GET", "path": path, "headers": []})
|
||||
|
||||
|
||||
def test_stream_disconnect_is_mapped_to_499():
|
||||
async def call_next(_request):
|
||||
raise anyio.EndOfStream
|
||||
|
||||
mw = _SlowRequestLogMiddleware(lambda *a: None)
|
||||
resp = asyncio.run(mw.dispatch(_request("/api/editor/bridge/events"), call_next))
|
||||
assert resp.status_code == 499
|
||||
|
||||
|
||||
def test_real_responses_pass_through():
|
||||
from starlette.responses import JSONResponse
|
||||
|
||||
async def call_next(_request):
|
||||
return JSONResponse({"ok": True}, status_code=200)
|
||||
|
||||
mw = _SlowRequestLogMiddleware(lambda *a: None)
|
||||
resp = asyncio.run(mw.dispatch(_request("/api/health"), call_next))
|
||||
assert resp.status_code == 200
|
||||
@@ -328,6 +328,30 @@ def test_conversational_request_wrappers_preserve_explicit_family(message, expec
|
||||
assert requested_capabilities(message) == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize("message", [
|
||||
"30-day task + protocol doc",
|
||||
"I wanna work on a project where I test meditation for 30 days. But I test my brain activity before and after so I can compare.",
|
||||
"Protocol doc only",
|
||||
"Make a 30-day meditation plan",
|
||||
"help me build a habit tracker for the project",
|
||||
])
|
||||
def test_bare_project_task_noun_does_not_inject_task_tools(message):
|
||||
# A project description that merely contains the noun "task" must not offer
|
||||
# the background-task scheduler or let the model auto-create a task.
|
||||
assert "tasks" not in requested_capabilities(message)
|
||||
assert selected_tools_for_request(message) != frozenset({"manage_tasks"})
|
||||
|
||||
|
||||
@pytest.mark.parametrize("message", [
|
||||
"Create a task to call the dentist",
|
||||
"List my scheduled tasks",
|
||||
"show me my automations",
|
||||
"what scheduled tasks do i have?",
|
||||
])
|
||||
def test_explicit_task_management_still_routes_to_tasks(message):
|
||||
assert "tasks" in requested_capabilities(message)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("message", [
|
||||
"can u also pop the notes panel open for me?",
|
||||
"k leave it for now - also pop open the calendar panel",
|
||||
@@ -4413,3 +4437,12 @@ def test_chinese_video_to_image_artifact_keeps_media_and_workspace_tools():
|
||||
"请完整浏览 /workspace/input/video.mp4，为每款零食截图并拼成清单图，"
|
||||
"输出到 /workspace/snack_checklist.jpg，交付前打开成品图检查。"
|
||||
) == {"media_inspection", "shell_files"}
|
||||
|
||||
|
||||
def test_named_shell_selects_shell_family_even_for_a_bare_command():
|
||||
# An explicitly named shell is a request to use the shell even when the
|
||||
# rest of the message is a bare/typo'd command; admission must match the
|
||||
# always-offered bash tool.
|
||||
assert requested_capabilities("Bash commabd Ls la") == {"shell_files"}
|
||||
assert requested_capabilities("bash ls") == {"shell_files"}
|
||||
assert requested_capabilities("use the terminal to show me the files") == {"shell_files"}
|
||||
|
||||
Reference in New Issue
Block a user