_emit_scalar quotes a frontmatter scalar with json.dumps when it holds
punctuation that would change how the line reads back. _parse_scalar undid that
with a bare raw[1:-1]: it stripped the quotes but never decoded the escapes. So
a description containing ü was written as the escape sequence \u00fc, read
back with that escape still sitting literally in the value, and re-escaped on
the next save. The backslash run doubles every save, so a non-English skill
description degrades into backslash noise after a few edits, and the escapes are
shown verbatim in the skills list and the /skills catalog.
This is not limited to non-ASCII. Any description containing a quote takes the
same path, since the quote is itself what forces the quoted form.
Make the two halves symmetric: emit with ensure_ascii=False, since SKILL.md is
UTF-8 at both ends (skills.py reads it, atomic_write_text writes it) and the
ASCII-escaped form bought nothing; and parse double-quoted scalars with
json.loads, falling back to the previous literal reading when the value is not
valid JSON. Files already corrupted heal one level per load.
ensure_ascii=False on its own would open a smaller hole. json.dumps escapes
every C0 control character but passes NEL, LINE SEPARATOR and PARAGRAPH
SEPARATOR through literally, and parse_frontmatter reads one scalar per line via
str.splitlines(), which breaks on all three. Re-escape those three, and add them
plus the remaining splitlines characters to the set that forces a quoted scalar,
so none of them can reach the file bare.
Co-authored-by: Alexandre Teixeira <111787685+alteixeira20@users.noreply.github.com>
* fix(memory): don't let an unreadable store get overwritten with an empty one
load_all() answered a failed read the same way it answered an empty store:
with []. Every mutation path is a read-modify-write (load the whole file,
change it, save it back), so a failed read became
load_all() -> [] -> [].append(new) -> save([new])
and save() is atomic, so the replacement stuck.
The case that actually destroys data is a store that is READABLE but not
parseable - a truncated file, or one holding {} instead of []. Nothing
obstructs the write, so adding a memory returns HTTP 200 and every memory
already stored is gone. Verified end-to-end against a running instance: on the
current code a truncated memory.json plus one add leaves the file holding only
the new entry. Truncation is reachable - core/database.py rewrites memory.json
during migration with a plain open(.., "w") + json.dump, which is not atomic.
A live exclusive lock is not the dangerous case: it blocks the read and the
os.replace alike, so the save fails too and the store survives. That path
currently 500s and loses nothing.
_read_entries() now returns [] only when the file genuinely does not exist and
raises MemoryStoreUnreadable for every other failure, including a store that
parses but is not a JSON array. load_all() keeps the old lenient behaviour so
display, search and context injection still degrade quietly instead of
breaking chat. The read-modify-write callers switch to load_all_for_update(),
which propagates the error: the memory routes turn it into a 503 and change
nothing, backup import refuses rather than saving only the incoming rows, and
auto-extraction and the audit merge skip the write. The audit merge mattered
most - it rebuilds the whole file from one owner's slice plus everyone else's
rows, so an empty read there dropped every other tenant's memories.
The corrupt-JSON path still gets its one shot at the legacy memory.txt
migration before raising, so that recovery is unchanged.
The two updated fakes gained load_all_for_update because the real class has it;
MagicMock would otherwise hand the import path a Mock instead of the seeded list.
Fixes#5673
* fix(memory): fail closed on the remaining read-modify-write add paths
The strict loader landed with the routes, the backup import and the extractor
converted, but three read-modify-write sinks still called load_all(), which
degrades an unreadable store to []. Two of them are the paths users actually
reach, so the data loss in #5673 stayed reproducible:
- src/ai_interaction.py do_manage_memory, action "add" — reached from ordinary
chat via src/tool_execution.py:793 -> dispatch_ai_tool. "Remember that I
prefer X" against an unreadable store wrote a one-entry file over it and
reported success.
- mcp_servers/memory_server.py, action "add" — the same shape through
_scope_entries(), registered as a built-in in src/builtin_mcp.py.
- src/memory_provider.py NativeMemoryProvider.remember and .delete — wired
into app state in src/app_initializer.py but not consumed outside tests yet,
converted here so the pattern is uniform before it goes live.
The MCP server takes _scope_entries(for_update=True) so list keeps the lenient
read. The edit and delete branches on both tool paths were already fail-closed
by accident — an empty view matches nothing and returns before the save — so
they are left alone.
The three new tests drive the real entry points rather than replaying the
shape, and use a truncated store, which is the case that reads back fine so
nothing stops the save. Each asserts memory.json is byte-identical afterwards;
all three fail on the previous commit with the store overwritten.
* fix(integrations): pin api_call to the SSRF-validated IP
execute_api_call runs check_outbound_url on the target, but that guard only
resolves the host to answer (ok, reason) and hands back no address. The request
right after it opened a plain httpx.AsyncClient, which resolves the host again at
connect time. A base_url host on a low TTL can pass the guard as a public IP and
then flip to 169.254.169.254 for the connect, so the call lands on cloud metadata
with the integration's stored auth headers attached.
Resolve once, remember the IPs the guard actually validated, and pin the client's
socket to that set through a small AnyIO-backed transport. SNI and the Host header
still come from the URL, so TLS and vhost routing are unchanged; connect-time
fallback stays inside the approved address set over one shared deadline. This is
the same pinning the webhook sender and web-fetch paths already do -- api_call was
the last outbound path that skipped it.
Fixes#5513
* fix(integrations): de-duplicate the pinned IP list
_default_resolver calls getaddrinfo(host, None) with no socktype filter, so
glibc returns one record per socktype and a single-homed host comes back three
times over. _validated_ips kept every entry, so the transport pinned the same
address repeatedly and the connect fallback could spend its shared deadline
retrying one dead address instead of moving on to a genuinely different one.
Windows getaddrinfo collapses those duplicate records, which is why the
ip-literal pin test only failed on CI and not locally.
The edit/delete/pause/run actions of do_manage_tasks gated ownership with
`if owner and task.owner and task.owner != owner`. The middle term made the
check a no-op whenever task.owner was null/empty — the state a scheduled task
sits in when it was created in no-login mode (or via the localhost middleware
bypass) before the periodic legacy-owner sweep reassigns it to the admin user.
Any authenticated user's agent could then edit, delete, pause, or run another
tenant's owner-less task; edit+run lets an attacker rewrite the task prompt and
execute it in the scheduler's agent context.
The sibling `list` action already scopes with an exact `owner == owner` filter,
so the mutators were strictly more permissive than the reader. Drop the middle
term so the guard fails closed on owner-less rows for authenticated callers,
matching `list` and the calendar/notes/gallery/session null-owner gates. Auth
disabled (owner falsy) and same-owner access are unchanged.
The HTTP email route guard `_assert_owns_account` and the explicit-account_id
path in `_get_email_config` gated cross-tenant access with
`if row.owner and row.owner != owner` -- which skips the check entirely when the
account row is owner-less (owner NULL or ""). `email_accounts` is the one
owner-scoped table left out of the legacy-owner migration backfill
(core/database.py), so such rows persist on multi-user deploys: an account
configured while auth was disabled, or an imported legacy row. Any authenticated
user could then pass that account's id to read/send/update-credentials/delete
another tenant's mailbox and read its decrypted IMAP/SMTP creds.
Both sibling paths already enforce the intended contract -- the same-file
`_owner_or_matching_legacy_account` fallback and the MCP `_account_visible_to_owner`
gate (whose comment says it mirrors "the HTTP email route fallback") only expose
an owner-less account when its own mailbox (imap_user / from_address) is the
caller's. Factor that row-level predicate into `_account_visible_to_owner` and
use it in both guards, so owner-less accounts are visible only on a mailbox
match. Owned accounts, the legacy-claim path, and single-user mode (owner == "")
are unchanged.
Complements #5234 (which fixes the same class on the MCP tool layer); this is
the HTTP route layer it does not touch.
The grep tool's ripgrep fast-path excluded deny-listed key files with
`--glob "!*<pat>*"` for each entry in _SENSITIVE_FILE_PATTERNS. ripgrep's
--glob is case-sensitive, so on a case-insensitive filesystem (Windows,
default macOS) a key stored under a case variant of its name (ID_RSA,
Known_Hosts, Authorized_Keys) is the same file on disk but slips past the
lowercase exclusion, and ripgrep returns its contents. Those names are
non-dotfiles, so ripgrep's default hidden-file skipping does not cover them
either. The Python fallback already blocks them via the case-folded
_is_sensitive_path (#5097), so the two paths disagreed.
Switch the sensitive-pattern exclusions to --iglob so they match
case-insensitively, mirroring _is_sensitive_path. Add a regression test
that seeds ID_RSA and Known_Hosts and asserts grep returns ordinary
matches but not the key contents.
api/proxy endpoints (OpenRouter, other OpenAI-compatible aggregators)
short-circuit _query_context_length: they only consult the static
KNOWN_CONTEXT_WINDOWS table and otherwise return DEFAULT_CONTEXT (128000).
Any model not in that table — e.g. a freshly listed OpenRouter model like
Owl-alpha — was therefore capped at 128k even though the endpoint's catalog
reports its true window (1048576), so the rest of the model context never
got used.
The short-circuit exists so a context lookup doesn't download a large proxy
catalog on every call. Keep that property for the common case: known models
still resolve from the table with no network. For a model missing from the
table, read the window from the endpoint's /models catalog and cache the
whole id->context map per endpoint, so the catalog is fetched at most once
per endpoint (not once per model) and only for models that were broken
anyway. On any fetch/parse failure or a model absent from the catalog, fall
back to DEFAULT_CONTEXT exactly as before.
Factor the per-entry field extraction the non-proxy path already used into
_model_ctx_from_entry so both paths share it.
Fixes#4886
GlobTool resolves its search root through _resolve_search_root (which
confines it to the workspace or default allowlist), but the literal
fast-path joined the model-supplied pattern onto that root without
re-confining it. os.path.join lets an absolute pattern or one containing
../ escape the root, and normpath collapsed the .. segments, so glob
returned the absolute path of arbitrary host files once they existed --
an existence/path oracle that bypasses the confinement read_file,
write_file, grep, and ls all enforce.
Keep the literal lookup inside the root via a commonpath containment
check; an escaping literal falls through to the os.walk matcher, which
only ever yields paths under the root. Wildcard matching was already
confined.
Install libmagic1 and image-scoped python-magic in the Docker image so upload MIME detection can use content sniffing. Add regression coverage for the Dockerfile dependency pair and the libmagic-present sniffing path.
A single-day all-day event whose source writes DTEND equal to DTSTART
(treating DTEND as an inclusive bound rather than the RFC 5545 exclusive
one) was stored verbatim as a zero-duration row. list_events selects
events overlapping the window with `dtstart < end AND dtend > start`, so
that row is filtered out for any window starting at or after its date and
the event never appears, even though the import reported success.
Events created via the API never hit this because creation always
synthesizes a positive duration; only the two import paths can persist a
non-positive one. Clamp a non-positive end at import (import_ics and the
CalDAV pull) to the same default span used when DTEND is absent: one day
for all-day events, one hour otherwise.
Also repair the persisted state for users who already imported before this
clamp existed. Their stored zero-duration row is invisible, and re-importing
the same ICS hit the duplicate branch and skipped without touching it, so
the event stayed hidden. The duplicate branch now backfills the clamp onto
the matched row before skipping, and the response reports a `repaired` count.
(The CalDAV pull already rewrites dtend on re-sync, so it self-heals.)
Inline backtick spans were converted to <code> only at the end of
mdToHtml, after the bare-URL autolink and <a>/allowed-HTML passes. A URL
inside inline code is preceded by a space, so the autolink wrapped it in
an <a> tag and swapped it for an ___ALLOWED_HTML_ placeholder, corrupting
commands like `irm http://127.0.0.1:3000/x`.
Extract inline code into placeholders before the link passes, mirroring
the existing fenced-code-block handling, and restore them last so
placeholders carried inside restored <a> blocks resolve. Escape the code
at extraction time since it now bypasses the global escape pass.
The scheduled-task runner built the agent's tool set from RAG retrieval plus
ASSISTANT_ALWAYS_AVAILABLE. Neither includes bash/python (nor the file tools),
and no keyword hint force-includes them, so a task only saw the shell when the
tool-embedding index happened to surface it. On hosts where that index is empty
or degraded (e.g. a fresh Docker deploy), retrieval returns nothing and the task
agent never receives bash/python — telling the user the shell is disabled even
for an admin owner.
Offer the shell/file group to task agents by default, mirroring the chat agent
where these are on unless a privilege or global setting turns them off. The
existing blocked_tools_for_owner() gate in stream_agent_loop still strips the
whole group for non-admin multi-user owners and only admits it for admins and
single-user (AUTH_ENABLED=false) deployments, so this changes what is offered,
not who is allowed. A crew that defines an explicit enabled_tools allowlist
still has its restriction honored.
Also merge the operator's global disabled_tools setting into the scheduler's
disabled set before composing relevant_tools and before entering the agent
loop, matching what chat already does. Without it, the global tool-disable
contract did not reach unattended scheduled tasks: an admin or AUTH_ENABLED=false
task could still see and call shell/file tools the operator had turned off
globally, since the prompt/schema/execution gates only enforce the disabled
tools passed in.
_reminder_minutes matched the offset with (?:m|min|minute|minutes)\b and
(?:h|hr|hour|hours)\b. The trailing \b makes the common plural
abbreviations "mins"/"hrs" fail to match (after "min" the "s" is a word
char, so no boundary), so reminder_minutes "5 mins" or "2 hrs" returned
None and the event was created with no reminder, silently.
Widen the two unit regexes and the matching reminder_only description
regex to a strict superset that also accepts mins/hrs. The sibling
duration parser already accepts these forms (it has no \b), so this only
brings the reminder parser in line.
The two delete-ordering tests did monkeypatch.chdir(tmp_path) and wrote the
image under tmp_path/data/generated_images, but DATA_DIR (and therefore
gallery_routes.GALLERY_IMAGE_DIR) is always an absolute path, so the delete
resolver pointed at the repo's real data dir and ignored the chdir.
test_file_removed_on_successful_delete therefore failed on dev (the file at
the tmp path was never the one being removed), and test_file_kept_when_commit_fails
passed only by accident. Set GALLERY_IMAGE_DIR to the seeded tmp dir via
monkeypatch so both tests exercise the real path and pass deterministically.
#4159 (6b0eca9) made build_models_url insert /v1 for path-less bases, so
the TestBuildersRejectLookalikeHosts model assertions that expected
/models started failing and turned the pytest gate red on dev.
Both the generic OpenAI branch and the real Anthropic branch now end in
/v1/models, so a URL-only assertion no longer proves a lookalike host
dodged the Anthropic/Ollama branch. Assert _detect_provider == "openai"
directly and keep the /v1/models expectation.
When a download's tmux pane is gone, the status endpoint trusted only the
HF-cache probe to tell completed from stopped. The probe derives its cache
root from its own environment, but the download runner exports
HF_HOME=<local_dir> (the #2722 fix), so custom-dir downloads land in
<local_dir>/hub where the probe never looks - and ollama pulls don't touch
the HF cache at all. Finished downloads were reported as stopped forever,
and tasks already persisted as completed were demoted back to stopped on
the next poll. This is the backend half of #3897, deliberately left out of
the frontend fix in #4000.
- honor the conclusive runner markers first: DOWNLOAD_OK -> completed
(keeping the "Fetching 0 files" error guard), DOWNLOAD_FAILED -> error
- pass the task's local_dir through to the cache probes so they check the
cache the download actually wrote to, keeping the env-var fallback for
default-cache downloads
- move the probe scripts and marker classification into
routes/cookbook_output.py (dependency-free) with behavioral tests
Fixes#4017
PATCH and DELETE /api/tokens/{id} both called require_admin but never
checked that the token belonged to the requesting admin. Any admin could
rename, re-scope, or delete another admin's token by ID.
create_token already stamps owner on every token — update and delete
just never read it. Fixed by comparing token.owner against
get_current_user(request) after the 404 guard, same pattern the rest of
the auth routes use. Check is skipped when current_user is falsy
(AUTH_ENABLED=false / single-user mode).
Fixes#3898
SKILL.md files written with mixed-case owner (e.g. 'owner: Alice') were
skipped because the regex had no IGNORECASE flag. _usage.json keys like
'Alice::skill-name' were missed by the startswith prefix check for the
same reason.
Both comparisons now match the same way the deep_research and memory
blocks do — case-insensitively against old_username.
Fixes#3611
Direct DbSession.owner == user becomes WHERE owner IS NULL when user is None
(auth disabled), hiding all sessions that carry an explicit owner. Same flaw
on the Document and GalleryImage sub-queries (active-doc and gallery badges).
Replace all three with owner_filter(), which is a no-op when user is falsy.
Fixes#3620
The module derived its state file path as Path(os.environ.get("DATA_DIR", "data"))
/ "cookbook_state.json". The correct env var is ODYSSEUS_DATA_DIR, which is
already read by src/constants.py and exported as COOKBOOK_STATE_FILE. When
ODYSSEUS_DATA_DIR is set (Docker, custom installs), the old code read the wrong
env var and silently wrote state to data/cookbook_state.json relative to CWD
while every other file resolved under the custom data directory.
Fixes#3621
The DB owner-rename loop in rename_user patched every SQL column named
owner, but three non-SQL stores were left behind:
1. session_manager.sessions -- in-memory Session objects carry s.owner
set at server-boot time. get_sessions_for_user() does an exact
s.owner == username check, so the renamed user chat sidebar goes empty
until a server restart.
2. data/deep_research/*.json -- each completed research report is a
standalone JSON file with an owner field. research_routes filters
by d.get(owner) == user, making every report invisible to the
renamed user.
3. data/memory.json -- a flat JSON array; each entry carries an owner
field. memory_manager.load(owner=user) filters on it, so all memories
vanish from the memory panel.
Fix: after the SQL loop, patch all three:
- iterate sm.sessions and update owner in-place (exposed via app.state)
- walk data/deep_research/*.json and rewrite owner with atomic_write_json
- update matching entries in memory.json with atomic_write_json
All three use the same case-insensitive lower() comparison the SQL loop
already uses. Each step is independently wrapped so a single failure
does not abort the others or the rename itself.
Fixes#3362
parse_due_for_user only matched day-first format ('today at 3pm').
Time-first strings like '3pm today' or '11pm today' — which the tool
schema and tool_index both advertise as valid examples — fell through
all branches, hit dateutil or the legacy _parse_dt fallback, and in
many cases raised ValueError. do_manage_notes then stored the raw
string verbatim, and the ISO-only reminder scanner (action_ping_notes)
never fired the note.
Add a time-first regex branch immediately after the day-first branch
to handle '<time> today|tonight|tomorrow|tmrw|yesterday'. Existing
day-first parsing is unchanged.
Fixes#3302
