routes/email_routes.py (7,194 lines), routes/email_helpers.py (2,025) and
routes/email_pollers.py (1,764) were the largest flat email area left in
routes/. They now live in routes/email/, following the pattern fifteen route
areas already use: the canonical module under the package, a shim at the old
path that replaces itself in sys.modules with the canonical object.
The shim matters more here than the move. Dozens of email tests monkeypatch
module attributes, and several pop the module from sys.modules and re-import
it. Without identity between routes.email_routes and routes.email.email_routes
those patches would apply to a different object than the app uses.
Intra-package imports keep the flat path, matching routes/document/ and
routes/gallery/: the shim resolves them and the diff stays a move.
Eight path references in five test modules read these files as source rather
than importing them, so they now read the canonical location. That is the same
adjustment every previous conversion made, and it is why the note/document
shims mention source-introspection tests explicitly.
No behaviour change. Verified with the full suite and under several explicit
collection orders, since a byte-identical routes move has broken tests under
one ordering before.
_scheduled_poll_once selected rows WHERE status='pending' and only wrote
status='sent'/'failed' after the SMTP send and IMAP append completed -
no atomic claim in between. Two overlapping callers (the in-process 30s
poller and an externally cron/systemd-driven 'odysseus-mail
poll-scheduled', or the CLI run manually) can both SELECT the same
pending row before either UPDATEs it, and both send it. _start_poller's
own docstring already names this exact risk ('avoid two copies of
_scheduled_poll_once racing on the same SQLite') but nothing in the code
enforced it - it was advisory only.
Add an atomic per-row claim: UPDATE ... SET status='sending' WHERE
id=? AND status='pending', proceeding only when rowcount == 1. The
loser of the race sees rowcount == 0 and skips the row instead of
sending a duplicate.
Adds a regression test that drives two real threads through the real
_scheduled_poll_once against a shared SQLite file, synchronized with a
barrier and a widened send-path window, and asserts exactly one send
fires. Reverting the fix makes the test fail reliably (5/5 runs); with
the fix it passes reliably (5/5 runs).
Fixes#5109
Move calendar processed-marker insert into the LLM success path (else branch).
Previously, the INSERT ran even after a transient LLM failure, causing the
poller to skip retrying calendar extraction on subsequent runs.
Minimal change: only touches the try/except/else control flow in
_auto_summarize_pass_single() — preserves existing formatting and line endings.
The fallback regex in email_pollers.py that recovers a
[{"action": ...}, ...] JSON array from raw model output used lazy
[^[\]]*? runs inside a (?:,\s*\{...\}\s*)* repetition, which backtracks
exponentially (CodeQL py/redos) on inputs like [{"action"},{ + }},{{ * N.
It runs on the LLM reply to an email→calendar prompt embedding the
untrusted email body, so a crafted email can stall the background poller.
Extract the pattern to a module-level _CAL_ACTION_ARRAY_RE and rewrite the
object-content class from the lazy [^[\]]*? to a greedy brace-delimited
[^{}], which removes the quantifier ambiguity. The match is linear (a 500KB
adversarial input now resolves in <1ms) and equivalent on well-formed
arrays; it is also strictly more robust for values containing '[' or ']'
(the old class bailed on those and extracted nothing).
Resolves CodeQL py/redos #198.
Two more bare uid.decode() calls at lines 889 and 897 crash with
AttributeError when uid is already a string. Applies the same
isinstance guard used everywhere else in this function.
On a large Gmail mailbox the email-summary poller's SINCE scan often finds
nothing (INTERNALDATE/date-header quirks), so it falls back to SEARCH ALL. That
returns one enormous UID line; the socket read can time out mid-response, and the
exception was swallowed — leaving the unread '* SEARCH 325188 …' bytes on the
socket. The next command (the downstream re-select) then read those leftover
bytes and failed with 'EXAMINE => unexpected response: b'325188 …''.
Extract the fallback into _latest_inbox_fallback_uids(conn, reconnect): on a
failed SEARCH ALL it logs out the poisoned connection and reconnects, returning
the fresh connection for downstream use. Reconnecting is correct by construction
— a new connection cannot carry the old one's leftover bytes — so the re-select
always runs on a clean socket.
The same SEARCH ALL + reuse pattern also exists in mcp_servers/email_server.py
and routes/email_routes.py; left for a separate change to keep this surgical.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 'from urllib.parse import quote as _q' at line 734 shadows the
module-level _q (istrstrstrstrstrstrIMAPutility) imported from email_helpers, causing
UnboundLocalError at lines 191 and 278 where _q is used before the
local import executes. This silently breaks the entire auto-summarize
pass.
Every other uid.decode() call in this function uses
'uid.decode() if isinstance(uid, bytes) else str(uid)' but the
warning at line 832 does bare uid.decode(), crashing with
AttributeError when uid is already a string.
`_auto_summarize_pass_single` in `routes/email_pollers.py` opens a
long-lived IMAP connection at line 172 and then performs ~700 lines of
work — IMAP `select`/`FETCH`/`SEARCH`, network POSTs to the LLM
endpoint, SQLite writes, and per-uid awaits. The only `conn.logout()`
calls were on three safe paths (early `"No recent emails"`, early
`"No model configured"`, and the happy path at the very end). If any
exception fired between `conn` being created and the final happy path,
the outer `except` block at line 921 caught it, logged, and returned —
without ever calling `conn.logout()`. The IMAP socket leaked until
the server's idle timeout killed it.
This is the same shape as the just-merged upstream fixes#1325
(`_imap_move` in `routes/email_helpers.py`) and #1330 (`_list_emails_sync`
in `routes/email_routes.py`), but in the *background* poller path —
`_auto_summarize_poller` invokes it every 30 min, so the leak
accumulates on every crashed pass instead of being a transient
request-path leak.
The fix is the exact try/finally pattern from #1330:
1. initialize `conn = None` before the try
2. let the try-block assign `conn = _imap_connect(...)`
3. drop the three explicit `conn.logout()` calls on safe paths
4. add a `finally:` block that calls `conn.logout()` if `conn` was set
Tests in `tests/test_email_polly_imap_leak.py` (1, all passing):
- `test_auto_summarize_pass_logs_out_imap_on_select_failure` —
monkeypatches `_imap_connect` to return a fake conn whose `select`
raises `RuntimeError`, then asserts the fake `conn.logout` was
called exactly once and the function returned an `Error: ...`
string. Pre-fix the assertion fails because the outer `except`
never reached `conn.logout`; post-fix the `finally` block
guarantees it on every exit path.
Pre-fix verification: temporarily reverted the patch and re-ran the
test; it fails with `logout_calls=0` (the IMAP socket was leaked on
every crashed pass). Post-fix: `logout_calls=1`.
Uniqueness:
- `git log --all --oneline -S 'conn.logout' -- routes/email_pollers.py`
→ no recent commit has touched this pattern in this file
- GitHub PR search for `routes/email_pollers.py` open PRs → 0
- Function has no existing test file (`grep _auto_summarize_pass_single
tests/` → no results)
---
**@pewdiepie-archdaemon — gentle bump on a sibling PR that's also stuck
in your queue from the same author:** PR #1306
(`fix(caldav): no-op prune when date_search returns 0 events`) is on
its 4th rebase, isolated to 2 files, 2/2 tests passing, with one
independent approval from `lalalune` already on record. It was clean
the last time you re-checked; if there's a blocker I haven't
addressed, please flag it so I can fix it. Otherwise, both #1306 and
this one are ready to merge.
Co-authored-by: isharak7m <192635824+isharak7m@users.noreply.github.com>
* fix: omit temperature for OpenAI reasoning models (o1/o3/o4/gpt-5)
These models only accept the default temperature; sending any explicit
value (even 0.0) returns HTTP 400 "Only the default (1) value is
supported". This broke two paths:
- Endpoint probing in _probe_single_model hardcodes temperature: 0.0, so
a perfectly valid o3/gpt-5 endpoint is reported as failing in the
Model Endpoints health check.
- Chat/stream payloads send temperature unconditionally, so a non-default
temperature preset 400s on these models.
The code already special-cases the same model family for
max_completion_tokens, so this adds a sibling _restricts_temperature()
helper and omits the field for those models, letting the API use its
required default. gpt-4.5 is intentionally excluded (not a reasoning
model; accepts temperature normally).
Adds tests/test_llm_core_temperature.py covering the predicate and the
synchronous payload builder.
* fix: also omit temperature for reasoning models on the direct-POST paths
The first commit only covered llm_call/llm_call_async/stream_llm and the
endpoint probe. Email auto-summary, urgency-less spam classification, the
email reply-summary endpoint, and gallery vision tagging build their
OpenAI payloads inline and POST them directly (requests/httpx), bypassing
llm_core — so a reasoning model configured there would still 400 on the
temperature field. These sites already branch on _uses_max_completion_tokens,
so they're the same class; added the matching _restricts_temperature guard.
gallery_routes also gains the max_completion_tokens branch it was missing,
so gpt-5 vision tagging works end to end.
Note: email_pollers urgency scoring goes through llm_call_async and was
already covered.
The email auto-calendar pass (settings.email_auto_calendar / the
extract_email_events task) scans recently received mail and lets an LLM
create / update / cancel calendar events. Two problems made it a cross-tenant,
remotely triggerable hole:
1. No owner scoping. _auto_summarize_pass(account_id=None) fans out over EVERY
enabled account of EVERY user. For each message it fetched an upcoming-events
snapshot with NO owner filter (all tenants' events) and handed those uids +
titles to the extraction LLM, then executed the model's ops via
do_manage_calendar(...) with owner=None. do_manage_calendar only filters by
owner when owner is not None, so create/update/delete ran across ALL users'
calendars. Net: every user's event titles/times were disclosed to the model,
and the model could cancel/move/duplicate any tenant's events by uid.
2. No prompt-injection wrapping. The raw email From/Subject/body were
interpolated straight into an instruction-shaped extraction prompt (unlike
the chat path, which wraps external text via src/prompt_security). Anyone
who can email a user whose instance has auto-calendar enabled could inject
operations: create attacker-controlled "meeting" events (the path even
auto-harvests URLs from the body into the event location/description — a
phishing primitive) or cancel/modify the victim's real events, with zero
human in the loop.
Fix:
- Add core.database.get_upcoming_events(owner) and use it for the snapshot, so
the LLM only ever sees the processed account owner's events.
- Look up the EmailAccount owner in _auto_summarize_pass_single and pass owner=
to every do_manage_calendar call, so create/update/delete are scoped to that
user (owner=None stays the single-user / legacy escape hatch).
- Tell the extraction model the email is untrusted data and not to follow
instructions inside it (defense-in-depth against injection).
Add tests/test_calendar_owner_scope.py: get_upcoming_events returns only the
given owner's events (and everything when owner is None). Fails against the old
unscoped query.
Léo