odysseus/specs/search.md

Search

Last updated: dev@e71f8ce | 2026-08-25

Scope

This spec covers web search, URL fetching, and search-derived context in:

• canonical routes/search/search_routes.py, with routes/search_routes.py as a compatibility shim;
• reusable outbound transport primitives in src/outbound_fetch.py;
• services/search/* and exported services.search.SearchService;
• src/search/* compatibility aliases around canonical service modules;
• search call sites in src/chat_processor.py, src/tool_execution.py, src/session_search.py, src/research_handler.py, src/deep_research.py, and services/research/research_handler.py;
• search settings in src/settings.py, static/js/settings.js, and compare/research frontend search callers;
• YouTube context paths in src/youtube_handler.py and services/youtube/youtube_handler.py;
• research visual/report consumers in src/visual_report.py and routes/research/research_routes.py;
• tests under tests/test_search_*, tests/test_service_search_*, tests/test_services_search_*, tests/test_security_regressions.py, tests/test_agent_loop.py, tests/test_deep_research_*, tests/test_research_handler_*, tests/test_youtube_*, and tests/test_og_image_extraction.py.

routes/chat_routes.py also exposes GET /api/search, but that route searches chat messages and belongs to chat history behavior, not web search.

Route Flows

routes/search/search_routes.py owns the browser/API web-search routes:

• GET /api/search/config returns search configuration with provider key presence, not secret values;
• POST /api/search calls comprehensive_web_search(..., return_sources=True) and returns {context, sources, error?};
• GET /api/search/providers returns provider metadata and availability;
• POST /api/search/query calls one provider directly and returns {results, provider, time, error?} without ranking, fallback chains, cache formatting, or content fetch.

Compare mode uses both route shapes: shared presearch uses /api/search, while provider/search comparison panes use /api/search/query. Research panels can pass provider override settings through research routes into the deep-research search path.

Research provider naming is not fully normalized in the UI: some frontend selectors still use google, while provider dispatch expects google_pse.

Search Pipeline

services/search/core.py owns comprehensive_web_search(). It coordinates provider selection, fallback chains, ranking, optional fetch/content extraction, formatted prompt context, cache invalidation, and analytics.

services/search/service.py owns SearchService, the async facade exported by services.search and services. It wraps the synchronous comprehensive search path off the event loop and maps route-style output into service result rows.

services/search/providers.py owns provider-specific calls for SearXNG, Brave, DuckDuckGo, Google PSE, Tavily, and Serper. PROVIDER_INFO, provider availability, missing-key behavior, and provider dispatch live there.

services/search/query.py owns query enhancement and sanitization, including stripping markdown/code-fence noise from model- or user-supplied queries before provider calls and extracting Unicode/non-ASCII capitalized entity names. services/search/ranking.py owns result ranking, including word-boundary title/snippet/subject matching so short query terms do not match unrelated substrings.

Provider Settings And Fallback

src/settings.py owns default provider settings. The default provider is SearXNG, with DuckDuckGo as the default fallback chain. static/js/settings.js owns the admin search settings UI, provider key presence display, provider selection, and fallback ordering. SafeSearch is a backend/provider setting today, not a visible Settings control.

Provider API keys come from settings or environment at call time. Web config routes expose availability/presence only, non-admin settings reads are scrubbed, and chat settings tools cannot set provider credentials.

Runtime behavior:

• disabled search returns disabled/unavailable text in the comprehensive path;
• missing keyed-provider secrets return empty provider results instead of exposing secrets;
• SearXNG retries through JSON variants before HTML fallback, pins English/general-engine defaults where needed, and maps news/recency settings into provider time filters;
• comprehensive search retries providers and then walks the fallback chain;
• /api/search/query is a direct provider test/query path and does not use the comprehensive fallback chain. Direct provider result limits can be controlled dynamically by the caller.

Content Fetching

src.outbound_fetch.py owns reusable synchronous public-URL classification, one-resolution-per-hop DNS pinning, redirect handling, and response-body budgets without search/content-extraction dependencies. services/search/content.py adapts those primitives and owns webpage extraction/cache/result shaping for the services path:

• public HTTP/HTTPS URL checks;
• DNS fail-closed behavior;
• rejection of localhost, metadata, private, reserved, multicast, and link-local targets;
• redirect revalidation on each hop;
• one-time public DNS resolution per hop plus an httpcore/httpx pinned transport that connects to the validated public IP while preserving the original URL, Host header, and TLS SNI, closing DNS-rebinding time-of-check drift;
• metadata, Open Graph image, list, table, code block, PDF, and text extraction;
• readable text extraction for text/*, Markdown, .txt, .json, .jsonl, and JSON content types;
• central User-Agent behavior through WEB_FETCH_USER_AGENT;
• soft and hard download byte caps through WEB_FETCH_SOFT_MAX_BYTES and WEB_FETCH_HARD_MAX_BYTES, with declared-length and streaming-budget checks; requests prefer identity transfer encoding so compressed bodies cannot bypass the effective body cap;
• JS-heavy empty result hints;
• cache writes;
• empty/error result shape, including explicit HTTP-status failures instead of raising through callers.

src/search/content.py is now a compatibility alias to services.search.content; chat URL auto-fetch, agent web_fetch, and deep research keep the src.search import path but share the services implementation.

Agent web_fetch raises the per-call budget only within the global hard cap, leads tool output with a partial-content notice when the download budget truncated the page, and then applies normal tool-output truncation so the notice survives.

Content failures are caller-shaped:

• comprehensive search drops failed page fetches and keeps usable search context;
• web_fetch returns tool errors, including bot-protection and HTTP-status failures;
• direct URL chat prefetch turns failures into compact untrusted unavailable-page context without exposing raw URL/exception/response diagnostics;
• deep research records search/provider failures separately from extraction failures.

Result Shapes

Search does not have one canonical result shape yet. Current shapes include:

• /api/search: {context, sources, error?};
• /api/search/query: {results, provider, time, error?};
• comprehensive_web_search(return_sources=True): formatted context plus {url, title} sources;
• SearchService.search(): service result rows;
• agent web_search: tool output text plus a hidden sources marker stripped by the agent loop;
• agent web_fetch: fetched page text or tool error;
• deep research: findings, cited sources, optional source images, and _last_search_error state.

Chat/session transcript search is separate from web search but now uses chat_messages_fts when available, sanitizes FTS queries, and batches message lookup after FTS hits to avoid per-hit database reads.

Search owns Open Graph image extraction for fetched pages. Research owns promotion of those images into research sources and visual reports. This is not a standalone web image-search provider or gallery image proxy.

YouTube

services/youtube/youtube_handler.py owns YouTube URL detection, id extraction, transcript, comment, and formatting behavior. src/youtube_handler.py is a compatibility alias to the canonical services module so startup init_youtube() state and chat imports share one implementation.

YouTube transcript and comment content is search-like external context. URL parsing covers common watch, mobile/music, embed, /v/, shorts, live, and youtu.be forms and must tolerate non-string input.

Compatibility State

src/search/core.py, src/search/providers.py, src/search/ranking.py, src/search/cache.py, src/search/content.py, src/search/query.py, and src/search/analytics.py are compatibility shims or module aliases around services.search. Ranking helpers exposed through src.search.ranking include recency scoring, result ranking, naive-UTC handling, _SPORTS_HINT_RE, and age formats.

src.youtube_handler remains a compatibility import path, but it should resolve to the same module object as services.youtube.youtube_handler.

Context Policy

Search results, fetched pages, Open Graph metadata, and YouTube transcript/comment content are untrusted context.

Chat search, chat URL prefetch, compare presearch, and YouTube context wrap inserted content through the shared untrusted-context message helpers. Agent web_search/web_fetch results are read-only tool outputs and must not be treated as instructions.

Deep research wraps fetched webpage content through untrusted_context_message("webpage", content) before extractor calls, though search result/failure shapes still differ from chat and agent tools.

Optional And Platform Behavior

ddgs is optional; provider code has an HTML fallback. Search cache and analytics state live under the shared data dir and mkdir failures in read-only image layers are tolerated where possible. PDF extraction uses pdfminer.six only when installed. Native SearXNG defaults to http://localhost:8080; Docker uses the compose searxng service URL and pins the SearXNG image with a healthcheck.

Compose preserves retained SearXNG settings but runs scripts/migrate_searxng_settings.py before startup to add missing use_default_settings: true inheritance. The migration accepts only a regular single-document YAML mapping, preserves BOM/newline/style/ownership/mode, writes and directory-fsyncs atomically, and no-ops when the key exists. Compose treats migration failure as non-fatal so SearXNG health reports the retained-file problem instead of the wrapper command preventing startup.

httpx and BeautifulSoup are required runtime dependencies for the active search/fetch path.

Current Gaps

• Search route handlers need direct tests for request body formats, provider validation, provider availability, and route error/empty-result shapes.
• Agent search, chat search prefetch, and research search do not yet share a single result/failure shape.
• src/search and services/search are mostly consolidated through shims, but import-path parity tests remain important.
• Deep-research webpage-content extraction uses the shared untrusted wrapper, but synthesis/reuse boundaries still need route/tool tests.
• Search-sourced og_image URLs need an explicit privacy/security decision: documented direct browser loads, public-URL validation, or a same-origin proxy.
• Route and integration tests do not fully pin chat/compare/YouTube untrusted-context insertion.