odysseus/specs/search.md

Search

Last updated: dev@86b886c | 2026-10-15

Scope

This spec covers web search, URL fetching, and search-derived context in:

• canonical routes/search/search_routes.py, with routes/search_routes.py as a compatibility shim;
• reusable outbound transport primitives in src/outbound_fetch.py;
• services/search/* and exported services.search.SearchService;
• src/search/* compatibility aliases around canonical service modules;
• search call sites in src/chat_processor.py, src/tool_execution.py, src/session_search.py, src/research_handler.py, src/deep_research.py, and services/research/research_handler.py;
• search settings in src/settings.py, static/js/settings.js, and compare/research frontend search callers;
• YouTube context paths in src/youtube_handler.py and services/youtube/youtube_handler.py;
• research visual/report consumers in src/visual_report.py and routes/research/research_routes.py;
• tests under tests/test_search_*, tests/test_service_search_*, tests/test_services_search_*, tests/test_security_regressions.py, tests/test_agent_loop.py, tests/test_deep_research_*, tests/test_research_handler_*, tests/test_youtube_*, and tests/test_og_image_extraction.py.

routes/chat_routes.py also exposes GET /api/search, but that route searches chat messages and belongs to chat history behavior, not web search.

Route Flows

routes/search/search_routes.py owns the browser/API web-search routes:

• GET /api/search/config returns search configuration with provider key presence, not secret values;
• POST /api/search calls comprehensive_web_search(..., return_sources=True) and returns {context, sources, error?};
• GET /api/search/providers returns provider metadata and availability;
• POST /api/search/query calls one provider directly and returns {results, provider, time, error?} without ranking, fallback chains, cache formatting, or content fetch.

Compare mode uses both route shapes: shared presearch uses /api/search, while provider/search comparison panes use /api/search/query. Research panels can pass provider override settings through research routes into the deep-research search path.

Research provider naming is not fully normalized in the UI: some frontend selectors still use google, while provider dispatch expects google_pse.

Search Pipeline

services/search/core.py owns comprehensive_web_search(). It coordinates provider selection, fallback chains, ranking, optional fetch/content extraction, formatted prompt context, cache invalidation, and analytics.

services/search/service.py owns SearchService, the async facade exported by services.search and services. It wraps the synchronous comprehensive search path off the event loop and maps route-style output into service result rows.

services/search/providers.py owns provider-specific calls for SearXNG, Brave, DuckDuckGo, Google PSE, Tavily, and Serper. PROVIDER_INFO, provider availability, missing-key behavior, and provider dispatch live there.

services/search/query.py owns query enhancement and sanitization, including stripping markdown/code-fence noise from model- or user-supplied queries before provider calls and extracting Unicode/non-ASCII capitalized entity names. services/search/ranking.py owns result ranking, including word-boundary title/snippet/subject matching so short query terms do not match unrelated substrings.

Provider Settings And Fallback

src/settings.py owns default provider settings. The default provider is SearXNG, with DuckDuckGo as the default fallback chain. static/js/settings.js owns the admin search settings UI, provider key presence display, provider selection, and fallback ordering. SafeSearch is a backend/provider setting today, not a visible Settings control.

Provider API keys come from settings or environment at call time. Web config routes expose availability/presence only, non-admin settings reads are scrubbed, and chat settings tools cannot set provider credentials.

Runtime behavior:

• disabled search returns disabled/unavailable text in the comprehensive path;
• missing keyed-provider secrets return empty provider results instead of exposing secrets;
• SearXNG retries through JSON variants before HTML fallback, pins English/general-engine defaults where needed, and maps news/recency settings into provider time filters;
• comprehensive search retries providers and then walks the fallback chain;
• /api/search/query is a direct provider test/query path and does not use the comprehensive fallback chain. Direct provider result limits can be controlled dynamically by the caller.

Content Fetching

src.outbound_fetch.py owns reusable synchronous public-URL classification, one-resolution-per-hop DNS pinning, redirect handling, and response-body budgets without search/content-extraction dependencies. services/search/content.py adapts those primitives and owns webpage extraction/cache/result shaping for the services path:

• public HTTP/HTTPS URL checks;
• DNS fail-closed behavior;
• rejection of localhost, metadata, private, reserved, multicast, and link-local targets;
• redirect revalidation on each hop;
• one-time public DNS resolution per hop plus an httpcore/httpx pinned transport that connects to the validated public IP while preserving the original URL, Host header, and TLS SNI, closing DNS-rebinding time-of-check drift;
• metadata, Open Graph image, list, table, code block, PDF, and text extraction;
• readable text extraction for text/*, Markdown, .txt, .json, .jsonl, and JSON content types;
• central User-Agent behavior through WEB_FETCH_USER_AGENT;
• soft and hard download byte caps through WEB_FETCH_SOFT_MAX_BYTES and WEB_FETCH_HARD_MAX_BYTES, with declared-length and streaming-budget checks; decoded response bytes are capped even for compressed transfer encodings, and truncation is reported;
• JS-heavy empty result hints;
• cache writes;
• empty/error result shape, including explicit HTTP-status failures instead of raising through callers.

src/search/content.py is now a compatibility alias to services.search.content; chat URL auto-fetch, agent web_fetch, and deep research keep the src.search import path but share the services implementation.

Agent web_fetch raises the per-call budget only within the global hard cap, leads tool output with a partial-content notice when the download budget truncated the page, and then applies normal tool-output truncation so the notice survives.

Content failures are caller-shaped:

• comprehensive search drops failed page fetches and keeps usable search context;
• web_fetch returns tool errors, including bot-protection and HTTP-status failures;
• direct URL chat prefetch turns failures into compact untrusted unavailable-page context without exposing raw URL/exception/response diagnostics;
• deep research records search/provider failures separately from extraction failures.

Result Shapes

Search does not have one canonical result shape yet. Current shapes include:

• /api/search: {context, sources, error?};
• /api/search/query: {results, provider, time, error?};
• comprehensive_web_search(return_sources=True): formatted context plus {url, title} sources;
• SearchService.search(): service result rows;
• agent web_search: tool output text plus a hidden sources marker stripped by the agent loop;
• agent web_fetch: fetched page text or tool error;
• deep research: findings, cited sources, optional source images, and _last_search_error state.

Chat/session transcript search is separate from web search but now uses chat_messages_fts when available, sanitizes FTS queries, and batches message lookup after FTS hits to avoid per-hit database reads.

Search owns Open Graph image extraction for fetched pages. Research owns promotion of those images into research sources and visual reports. This is not a standalone web image-search provider or gallery image proxy.

YouTube

services/youtube/youtube_handler.py owns YouTube URL detection, id extraction, transcript, comment, and formatting behavior. src/youtube_handler.py is a compatibility alias to the canonical services module so startup init_youtube() state and chat imports share one implementation.

YouTube transcript and comment content is search-like external context. URL parsing covers common watch, mobile/music, embed, /v/, shorts, live, and youtu.be forms and must tolerate non-string input.

Compatibility State

src/search/core.py, src/search/providers.py, src/search/ranking.py, src/search/cache.py, src/search/content.py, src/search/query.py, and src/search/analytics.py are compatibility shims or module aliases around services.search. Ranking helpers exposed through src.search.ranking include recency scoring, result ranking, naive-UTC handling, _SPORTS_HINT_RE, and age formats.

src.youtube_handler remains a compatibility import path, but it should resolve to the same module object as services.youtube.youtube_handler.

Context Policy

Search results, fetched pages, Open Graph metadata, and YouTube transcript/comment content are untrusted context.

Chat search, chat URL prefetch, compare presearch, and YouTube context wrap inserted content through the shared untrusted-context message helpers. Agent web_search/web_fetch results are read-only tool outputs and must not be treated as instructions.

Deep research wraps fetched webpage content through untrusted_context_message("webpage", content) before extractor calls, though search result/failure shapes still differ from chat and agent tools.

Optional And Platform Behavior

ddgs is optional; provider code has an HTML fallback. Search cache and analytics state live under the shared data dir and mkdir failures in read-only image layers are tolerated where possible. PDF extraction prefers pypdf layout text with a plain-text compatibility fallback and page labels, then optional pdfminer; a truncated soft-budget PDF can retry within the hard cap when its declared size allows. Native SearXNG defaults to http://localhost:8080; Docker uses the compose searxng service URL and pins the SearXNG image with a healthcheck.

Compose preserves retained SearXNG settings but runs scripts/migrate_searxng_settings.py before startup to add missing use_default_settings: true inheritance. The migration accepts only a regular single-document YAML mapping, preserves BOM/newline/style/ownership/mode, writes and directory-fsyncs atomically, and no-ops when the key exists. Compose treats migration failure as non-fatal so SearXNG health reports the retained-file problem instead of the wrapper command preventing startup.

httpx and BeautifulSoup are required runtime dependencies for the active search/fetch path.

Query Repair And Extractive Observations

services/search/core.py normalizes conversational provider queries, filters low-relevance results, respects explicit site scope, retries bounded broadened queries after empty results, and reapplies relevance/ranking on cache hits. Completed empty provider responses advance through the chain instead of repeating the same request as if it were a transport error. Named scholarly work lookup can use arXiv/OpenAlex metadata before generic search, and exact arXiv identifiers have a direct source path.

src/search_intent.py distinguishes requested publication windows from current reference information: current documentation/configuration alone does not imply recently published pages. src/search_passages.py owns literal, query-relevant excerpts and balanced bounded search observations across fetched sources, preserving explicit omission markers and source headers rather than prefix-truncating away later sources. These are extractive observations, not generated source claims. Deep research shares provider primitives through its navigator but retains its own result and failure shapes.

Retrieval Limits And Source Selection

When SearXNG is primary and searxng_yep is absent from the configured chain, _build_provider_chain() inserts that separate candidate at index one, including when SearXNG is the only configured provider. An already configured YEP position is retained; other primary providers do not get the automatic insertion. Missing keyed providers are excluded. Exact-empty query relaxation is bounded and disabled for explicit operators, quotes or exclusions. Low-relevance filtering can retain scoped results rather than invent a confident match. Temporal qualifiers can exclude a year from product-number overlap checks, while genuine numeric model names remain significant.

News category and publication window are independent: current events infer a day and news a week where no explicit window exists, while current reference documentation does not imply recent publication. Software release hints can retain general category; explicit windows survive JSON-to-HTML and news-to-general fallback along with language/engines/SafeSearch. The optional DuckDuckGo HTML request has a maximum five-second request timeout, not a bound on the complete multi-provider search.

HTML extraction removes chrome from a parsed copy, prefers semantic main content and a sole substantial article (at least 200 stripped characters), and avoids ancestor duplication; multiple article cards retain main context. Thin-body and access-challenge rules are heuristics. An exact challenge title with corroborating short body is an unavailable page even under HTTP200 and is not cached as article evidence. XML/Atom preserves bounded raw text; failed arXiv category/date listings can return a suggested start-zero/max100 Atom URL without fetching it or claiming completeness. Extractor caches use the current semantic-links namespace rather than old parser results.

Scholarly metadata uses a versioned User-Agent, a 12-second lookup timeout and shared 20-second monotonic budget, with at most three manually checked redirects and cleared original query parameters after redirect. This helper checks destinations but uses ordinary HTTP transport rather than claiming the separate outbound-fetch DNS connection pinning contract. Title/identifier matching remains heuristic metadata lookup, not independent verification of a paper. Direct browser-source citations and search snippets remain untrusted and insufficient to certify full-report evidence.

Current Gaps

• Search route handlers need direct tests for request body formats, provider validation, provider availability, and route error/empty-result shapes.
• Agent search, chat search prefetch, and research search do not yet share a single result/failure shape.
• src/search and services/search are mostly consolidated through shims, but import-path parity tests remain important.
• Deep-research webpage-content extraction uses the shared untrusted wrapper, but synthesis/reuse boundaries still need route/tool tests.
• Search-sourced og_image URLs need an explicit privacy/security decision: documented direct browser loads, public-URL validation, or a same-origin proxy.
• Route and integration tests do not fully pin chat/compare/YouTube untrusted-context insertion.