The arXiv and OpenAlex title resolvers called two hardcoded endpoints with a
hand-written "Odysseus/0.20" User-Agent, no outbound-URL policy, and a full
12s timeout each. APP_VERSION was already 1.0.3, so the agent string was wrong
the moment it was written, and a scholarly query could hold a user-facing
search open for the sum of all three hops.
Move the endpoints to named constants, build the User-Agent from APP_VERSION,
run both calls through check_outbound_url (they follow redirects, so the final
host is not the one in the constant), and give the whole SearXNG -> OpenAlex ->
arXiv chain one shared wall-clock budget.
The budget is a ContextVar rather than a parameter so the existing test doubles
for _openalex_title_results and _arxiv_title_results keep working unchanged.
* fix(skill-importer): validate URL scheme and improve skills.sh handling
* fix(skill-importer): enhance DNS resolution and SSRF protection in fetch URL handling
* fix(url-safety): add allowed_dist parameter to check_outbound_url for flexible private blocking
* test(skill-importer): add comprehensive tests for URL parsing and outbound checks
* ensure newline at end of file in test_check_outbound_url_allows_public_ip
* fix(skill-importer): improve TLS certificate handling in _get_checked function
* fix(skill-importer): enhance _check_fetch_url to handle both hostnames and full URLs
* fix(skill-importer): enhance parse_skill_source to support skills.sh URLs in path and netloc
* fix(skill-importer): simplify skills.sh hostname check in parse_skill_source
* fix(skill-importer): enhance parse_skill_source to identify skills.sh URLs in path and handle localhost/IP addresses
* fix(skill-importer): enhance _resolve_and_check_url to validate all resolved IP addresses and prevent TOCTOU vulnerabilities
* fix(skill-importer): enhance parse_skill_source to support schemeless GitHub and skills.sh URLs
* fix(memory): resolve CodeQL URL sanitization warning and restore _check_fetch_url test alias
* fix(memory): pin skill fetch sockets without rewriting URLs
* fix(memory): reject unsupported skill wrapper hosts
* refactor(url-safety): remove unused importer exception
* test(memory): keep redirect regression hermetic
* test(dns-rebinding): add test for _PinnedTransport to ensure connection to pinned IP
* fix(skill-importer): enhance skills.sh support to extract GitHub links from page content
* fix(skill-importer): improve URL scheme validation for GitHub and skills.sh links
* fix(skills): reject unusable skill URLs instead of guessing
Resolving a skills.sh link by scraping the first github.com URL out of
the page body cannot work. Skill pages only ever link the repository
root, never the skill's subdirectory, so every skill in a repo resolved
to the same bundle: importing skills.sh/anthropics/skills/pdf walked the
whole monorepo, saturated the 64-file cap, and installed algorithmic-art
behind an ok:true response. Restore the redirect-target unwrap and fail
with a message that says what to do instead.
Also report the real reason a URL is rejected. The scheme check keyed off
"://" appearing anywhere in the string, so a supplied-but-unusable URL
came back as "URL is required", and a schemeless URL carrying "://" in
its query was reported as an unsupported scheme. Key off the parsed
scheme and let opaque schemes (mailto:, javascript:) and a schemeless
host:port fall through to the host check.
* test(skills): tighten the real-socket pinning regression
The handler swallowed its own exceptions, so a failure inside it
surfaced as a confusing assertion on the captured client address.
Record the exception and assert on it, run the thread as a daemon, and
close the listening socket from the test so a hang cannot outlive the
run. Also drop the duplicate ipaddress import and the missing newline.
* fix(skills): require exact GitHub skill URLs
* test(skills): read complete pinned request headers
---------
Co-authored-by: RaresKeY <158580472+RaresKeY@users.noreply.github.com>
Co-authored-by: Léo <leograndcontact@gmail.com>
_emit_scalar quotes a frontmatter scalar with json.dumps when it holds
punctuation that would change how the line reads back. _parse_scalar undid that
with a bare raw[1:-1]: it stripped the quotes but never decoded the escapes. So
a description containing ü was written as the escape sequence \u00fc, read
back with that escape still sitting literally in the value, and re-escaped on
the next save. The backslash run doubles every save, so a non-English skill
description degrades into backslash noise after a few edits, and the escapes are
shown verbatim in the skills list and the /skills catalog.
This is not limited to non-ASCII. Any description containing a quote takes the
same path, since the quote is itself what forces the quoted form.
Make the two halves symmetric: emit with ensure_ascii=False, since SKILL.md is
UTF-8 at both ends (skills.py reads it, atomic_write_text writes it) and the
ASCII-escaped form bought nothing; and parse double-quoted scalars with
json.loads, falling back to the previous literal reading when the value is not
valid JSON. Files already corrupted heal one level per load.
ensure_ascii=False on its own would open a smaller hole. json.dumps escapes
every C0 control character but passes NEL, LINE SEPARATOR and PARAGRAPH
SEPARATOR through literally, and parse_frontmatter reads one scalar per line via
str.splitlines(), which breaks on all three. Re-escape those three, and add them
plus the remaining splitlines characters to the set that forces a quoted scalar,
so none of them can reach the file bare.
Co-authored-by: Alexandre Teixeira <111787685+alteixeira20@users.noreply.github.com>
* fix(memory): don't let an unreadable store get overwritten with an empty one
load_all() answered a failed read the same way it answered an empty store:
with []. Every mutation path is a read-modify-write (load the whole file,
change it, save it back), so a failed read became
load_all() -> [] -> [].append(new) -> save([new])
and save() is atomic, so the replacement stuck.
The case that actually destroys data is a store that is READABLE but not
parseable - a truncated file, or one holding {} instead of []. Nothing
obstructs the write, so adding a memory returns HTTP 200 and every memory
already stored is gone. Verified end-to-end against a running instance: on the
current code a truncated memory.json plus one add leaves the file holding only
the new entry. Truncation is reachable - core/database.py rewrites memory.json
during migration with a plain open(.., "w") + json.dump, which is not atomic.
A live exclusive lock is not the dangerous case: it blocks the read and the
os.replace alike, so the save fails too and the store survives. That path
currently 500s and loses nothing.
_read_entries() now returns [] only when the file genuinely does not exist and
raises MemoryStoreUnreadable for every other failure, including a store that
parses but is not a JSON array. load_all() keeps the old lenient behaviour so
display, search and context injection still degrade quietly instead of
breaking chat. The read-modify-write callers switch to load_all_for_update(),
which propagates the error: the memory routes turn it into a 503 and change
nothing, backup import refuses rather than saving only the incoming rows, and
auto-extraction and the audit merge skip the write. The audit merge mattered
most - it rebuilds the whole file from one owner's slice plus everyone else's
rows, so an empty read there dropped every other tenant's memories.
The corrupt-JSON path still gets its one shot at the legacy memory.txt
migration before raising, so that recovery is unchanged.
The two updated fakes gained load_all_for_update because the real class has it;
MagicMock would otherwise hand the import path a Mock instead of the seeded list.
Fixes#5673
* fix(memory): fail closed on the remaining read-modify-write add paths
The strict loader landed with the routes, the backup import and the extractor
converted, but three read-modify-write sinks still called load_all(), which
degrades an unreadable store to []. Two of them are the paths users actually
reach, so the data loss in #5673 stayed reproducible:
- src/ai_interaction.py do_manage_memory, action "add" — reached from ordinary
chat via src/tool_execution.py:793 -> dispatch_ai_tool. "Remember that I
prefer X" against an unreadable store wrote a one-entry file over it and
reported success.
- mcp_servers/memory_server.py, action "add" — the same shape through
_scope_entries(), registered as a built-in in src/builtin_mcp.py.
- src/memory_provider.py NativeMemoryProvider.remember and .delete — wired
into app state in src/app_initializer.py but not consumed outside tests yet,
converted here so the pattern is uniform before it goes live.
The MCP server takes _scope_entries(for_update=True) so list keeps the lenient
read. The edit and delete branches on both tool paths were already fail-closed
by accident — an empty view matches nothing and returns before the save — so
they are left alone.
The three new tests drive the real entry points rather than replaying the
shape, and use a truncated store, which is the case that reads back fine so
nothing stops the save. Each asserts memory.json is byte-identical afterwards;
all three fail on the previous commit with the store overwritten.
* fix(skills): replace deprecated utcnow in skill timestamp helper
_now_iso() builds the 'created' value in skill frontmatter. datetime.utcnow()
returns a naive datetime and has been deprecated since Python 3.12, scheduled
for removal. Switch to the timezone-aware datetime.now(timezone.utc), keeping
the serialized YYYY-MM-DDTHH:MM:SSZ shape unchanged so existing skill files
keep parsing.
timezone.utc is used rather than the datetime.UTC alias, which is 3.11+ only.
Adds regression tests covering the deprecation, the serialized shape, and
UTC correctness under a non-UTC local timezone -- the last guards against a
bare datetime.now(), which yields the same shape but local wall time.
Fixes#5697
* test(skills): skip timezone mutation where unsupported
---------
Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com>
* Harden skill importer against SSRF: block private targets + revalidate redirects per hop
The skill importer validated only the initial URL with the lenient SSRF guard
(block_private=False) and then fetched with follow_redirects=True, so a 3xx to
an internal/metadata address (169.254.169.254, 127.0.0.1, RFC-1918) was still
connected to — inconsistent with the hardened services/search/content.py
:_get_public_url path.
Add a _get_checked() helper that follows redirects manually and re-runs the
SSRF guard with block_private=True on every hop, and route all three fetch
sites (skills.sh unwrap, _fetch_bytes, _list_github_dir) through it. GitHub's
own redirects and the final-host _assert_github_url checks are preserved.
Adds hermetic regression tests (IP-literal hosts, faked HTTP layer) and updates
the existing mock signature for the new block_private kwarg.
Defense-in-depth: the endpoint is admin-gated (require_admin) and admins are
trusted per THREAT_MODEL.md, so this is not a cross-boundary vulnerability.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: enforce follow_redirects=False invariant in mock client
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
_extract_entities used the ASCII-only class [A-Z][a-zA-Z]+ to pull name
entities from a query, so non-ASCII names were dropped ("İstanbul",
"Zürich" yielded nothing) or shredded ("São Paulo" -> only "Paulo"),
degrading query enhancement for non-English/accented searches. Match
Unicode words and keep the alphabetic, uppercase-initial ones; ASCII
behaviour (the word boundary already excludes camelCase mid-word
capitals) is unchanged.
* fix(search): pin DNS-validated fetch connections
Rebase the DNS-rebinding SSRF fix onto current dev after search content moved behind the services.search.content canonical module.
Integrate the pinned httpcore NetworkBackend/BaseTransport approach with the current size-capped Client.stream fetch path, preserving Host/SNI semantics while forcing TCP connect to the already validated resolved IP.
Keep src.search.content as the compatibility wrapper and preserve existing OG-image http(s) behavior; this avoids reintroducing the unrelated scope changes that previously blocked review.
Add the explicit httpcore>=1.0,<2.0 requirement used by the public httpcore NetworkBackend and ConnectionPool APIs.
* test(search): restore and rebase DNS rebinding regressions
Keep the current security regression coverage that the stale PR branch had deleted, including auth-disabled localhost bypass and Ollama cookbook hardening tests.
Carry forward the DNS-rebinding coverage for private resolve blocking, pinned TCP connect behavior, Host header preservation, redirect revalidation, and the BaseTransport/public-httpcore static guard.
Update redirect tests to mock the current Client.stream-based capped fetch path rather than the older httpx.stream/get path.
* test(search): adapt size-cap fetch tests to pinned client stream
The DNS-rebinding repair moved _get_public_url from the module-level httpx.stream shortcut to httpx.Client(...).stream(...) so the fetch can use the pinned transport.
Keep the existing size-cap test fakes by routing Client.stream through the monkeypatched httpx.stream only when a test has installed that fake; otherwise fall back to a real Client.
This fixes the CI failures in tests/test_web_fetch_size_caps.py without touching unrelated upload-handler atomicity behavior, which is already flaky on clean origin/dev.
---------
Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com>
Add official Gemma 4 12B-it plus QAT-INT4/INT8 catalog entries (with their
GGUF sources), QAT quantization support across the quant tables and the
prequantized-prefix list, and the missing RTX 3050 / 3050 Ti memory
bandwidth so speed estimates stop falling back to the generic cuda value.
Remote Cookbook hwfit probes failed on Windows hosts because the PowerShell script was sent as nested -Command quoting through OpenSSH. Use -EncodedCommand for remote probes, auto-detect platform when omitted (including Darwin for Mac SSH hosts), and return a clearer error when SSH works but the probe fails.
Co-authored-by: Cursor <cursoragent@cursor.com>
Three classes of incorrect detection fixed:
(1) AMD GPU + no ROCm installed (e.g. Strix Halo) was reported as
backend=rocm everywhere, so launch commands emitted
HIP_VISIBLE_DEVICES (silent no-op on Vulkan) and the from-source
build path failed. Both _probe_amd_sysfs (routes/cookbook_routes)
and _detect_amd (services/hwfit/hardware) now probe rocminfo /
hipconfig / vulkaninfo at detection time and report vulkan when
only Vulkan is present.
(2) Build helper was picking the CUDA branch on AMD hosts whenever a
stray pip-installed nvcc was on PATH (vLLM wheels carry one
without libcudart). Added _odysseus_has_nvidia_hw() that checks
nvidia-smi / /dev/nvidia* / lspci, and gates both the nvcc PATH
augmentation and the CUDA elif branch on real hardware.
(3) Build chain reordered to ROCm/HIP > CUDA > Vulkan > CPU. Vulkan
tier added between CUDA and CPU as a portable fallback for hosts
with a GPU but no native toolchain (the common Strix Halo case).
Same _append_llama_cpp_linux_accel_build_lines also auto-attempts
sudo -n apt/pacman/dnf install of cmake/build-essential/git when
they are missing, surfacing a clear no-passwordless-sudo warning
otherwise.
* fix(hwfit): use CPU fallback for cpu_only speed estimates
* fix(hwfit): preserve ARM fallback for cpu_only estimates
---------
Co-authored-by: Cata <cata@bigjohn.local>
The outbound UA for web_fetch / web_search was inlined in four places with
two different values and nothing keeping them current: content.py pinned a
mid-2021 Chrome 91 build, and providers.py sent a bare Mozilla/5.0 in three
spots. Some sites serve a degraded or blocked page to a UA that old.
Add WEB_FETCH_USER_AGENT to src/constants.py (env-overridable, matching the
existing Copilot/Kimi UA-constant pattern) and import it in content.py and
providers.py. Default to a current, common desktop UA so pages return their
normal HTML: the market-leading desktop OS (Windows; NT 10.0 covers Windows
10 and 11) and browser (Chrome) on a current stable build. The version is now
bumped in one place.
Service-specific self-identifying agents (Copilot, Kimi, webhooks, cookbook)
are intentionally left separate. Adds a regression pinning the constant shape,
the env override, and a guard against a new inline Mozilla literal in the
search sources.
Closes#4324
* fix(search): add download budgets to web_fetch with truncation notice and hard ceiling
MAX_OUTPUT_CHARS only trims what the agent sees; fetch_webpage_content
buffered and cached the entire response body first, so a large or hostile
URL could pull arbitrarily many bytes into memory and the content cache.
The fetch is now a capped streaming GET (SSRF redirect guard unchanged):
a soft default budget (WEB_FETCH_SOFT_MAX_BYTES, 2 MB), a per-call
override via full/max_bytes on the web_fetch tool, and a hard ceiling
(WEB_FETCH_HARD_MAX_BYTES, 20 MB) that the override can never exceed.
When Content-Length already declares a body over the ceiling the fetch
is refused before any body bytes are buffered. Truncated results carry
truncated/fetched_bytes/total_bytes, the tool output leads with a
partial-content notice telling the model how to re-fetch with full=true,
and the tool schema documents the flag. A truncated PDF is reported as
a budget error since a cut PDF is unparseable. The effective cap is part
of the content-cache key so a truncated fetch is never served to a
full-budget request.
Existing tests that faked httpx.get or the old _get_public_url signature
are adapted to the streaming interface; behavior pins are unchanged.
Fixes#3812
* fix(search): close compressed-body cap bypass and protect the partial notice
Addresses RaresKeY's review on #3955:
- Force Accept-Encoding: identity for the capped fetch. With gzip/deflate the
wire bytes (and Content-Length) can be a fraction of the decoded body, so a
tiny compressed response could pass the hard-cap preflight and then expand
past the ceiling in a single decoded chunk before the streamed cap could
slice it. Identity makes Content-Length the true body size and keeps each
streamed chunk bounded by the network read, so the hard ceiling actually
bounds memory.
- Lead web_fetch output with the partial-content notice and cap the page
title. The notice is the user-facing contract for partial fetches, but the
title is untrusted, uncapped page content; placed ahead of the notice a giant
title could push it past MAX_OUTPUT_CHARS and drop it. The notice now leads
and the title is capped as a second guard.
Adds regressions: the fetch advertises identity encoding, and a truncated
result with an oversized title still surfaces the partial notice.
* fix(search): reject compressed responses that ignore the identity request
Requesting Accept-Encoding: identity is not enough on its own: a server can
ignore it and still return Content-Encoding: gzip, and httpx.iter_bytes would
decode that, so a tiny compressed body could balloon into one decoded chunk
far past the hard cap before the streamed loop slices it (and Content-Length,
the compressed wire length, makes the preflight and size metadata unreliable).
Refuse a non-identity Content-Encoding before reading the body. Adds a
regression where the server ignores the identity request and returns gzip;
the fetch is refused before any body is decoded.
providers.py redefined REQUEST_TIMEOUT = 20 locally, shadowing the same
value in src/constants.py and risking drift if the constant is bumped.
Import it from src.constants and drop the local copy; same value, one
source of truth.
Closes#4329
NVIDIA Grace Blackwell GB10 / DGX Spark was missing from GPU_BANDWIDTH, so
_lookup_bandwidth() returned None for it and _estimate_speed() fell through
to the crude FALLBACK_K path (k/active-params). That over-stated tok/s and
let speed scores saturate regardless of the box's real ~273 GB/s LPDDR5X
pool — distorting model ranking on these 128GB unified-memory rigs.
Add "gb10": 273 (GB/s). nvidia-smi reports the device name as "NVIDIA GB10",
which substring-matches the new key, so detected GB10 boxes now estimate
speed from the real bandwidth instead of the fallback.
Léo