mirror of
https://github.com/pewdiepie-archdaemon/odysseus.git
synced 2026-09-13 19:52:20 +02:00
RaresKeYandStressTestor
7026cf40b5
* docs(specs): restore bootstrap after dev rewrite
* docs(specs): remove runtime inventory snapshot
* docs(specs): reconcile current dev truth
* docs(specs): document scheduled task actions as an owner-attribution source
Owner Attribution covered cookie, bearer-token and internal-loopback
requests. Scheduled task actions are a fourth source and behave
differently: _execute_action passes owner=task.owner off the stored
ScheduledTask row, so no request and no resolved principal are in
flight, and route-level require_user() never runs.
Webhook triggers are the sharp case. They are unauthenticated by
design with the token as the only credential and execute under the
stored task.owner.
Paths cite routes/task/task_routes.py, the canonical location after
the task subpackage move (#6081); routes/task_routes.py on current dev
is the backward-compat shim.
* docs(specs): add chained tasks to the trigger list, refresh dev stamp
Review feedback from RaresKeY on the previous commit.
"Every trigger path" was too broad: success-chained tasks are another
path into _execute_action. Added them with their own citation, and
noted that chaining additionally requires the target task to share
task.owner and rejects cycles, which is stricter than the trigger-side
checks. Softened the lead-in to "these trigger paths".
Line 56 still pointed at routes/task_routes.py for webhook credential
validation. That path is the backward-compat shim on current dev after
the task subpackage move (#6081); repointed to the canonical
routes/task/task_routes.py.
Stamp moved to dev@2a6b09b. Inspection backing that bump was scoped:
every file path cited in this spec was mechanically checked to resolve
on 2a6b09b, and every file:line in the Owner Attribution additions was
read against it. Behavioral claims elsewhere in the file were not
re-audited.
* docs(specs): correct SECURE_COOKIES description to match current behavior
Third of the stale details RaresKeY enumerated. The cookie section
described SECURE_COOKIES as purely opt-in, which stopped being true.
_secure_cookie() (routes/auth_routes.py:89) treats an explicit true or
false as authoritative and derives the Secure attribute from the
request otherwise, including when the variable is unset and when
docker-compose injects it present-but-empty. Either the connection
scheme or the first X-Forwarded-Proto hop being https is enough.
* docs(specs): refresh current dev truth
---------
Co-authored-by: StressTestor <212606152+StressTestor@users.noreply.github.com>
141 lines
11 KiB
Markdown
141 lines
11 KiB
Markdown
# Search
|
|
|
|
Last updated: dev@e71f8ce | 2026-08-25
|
|
|
|
## Scope
|
|
|
|
This spec covers web search, URL fetching, and search-derived context in:
|
|
|
|
- canonical `routes/search/search_routes.py`, with `routes/search_routes.py` as a compatibility shim;
|
|
- reusable outbound transport primitives in `src/outbound_fetch.py`;
|
|
- `services/search/*` and exported `services.search.SearchService`;
|
|
- `src/search/*` compatibility aliases around canonical service modules;
|
|
- search call sites in `src/chat_processor.py`, `src/tool_execution.py`, `src/session_search.py`, `src/research_handler.py`, `src/deep_research.py`, and `services/research/research_handler.py`;
|
|
- search settings in `src/settings.py`, `static/js/settings.js`, and compare/research frontend search callers;
|
|
- YouTube context paths in `src/youtube_handler.py` and `services/youtube/youtube_handler.py`;
|
|
- research visual/report consumers in `src/visual_report.py` and `routes/research/research_routes.py`;
|
|
- tests under `tests/test_search_*`, `tests/test_service_search_*`, `tests/test_services_search_*`, `tests/test_security_regressions.py`, `tests/test_agent_loop.py`, `tests/test_deep_research_*`, `tests/test_research_handler_*`, `tests/test_youtube_*`, and `tests/test_og_image_extraction.py`.
|
|
|
|
`routes/chat_routes.py` also exposes `GET /api/search`, but that route searches chat messages and belongs to chat history behavior, not web search.
|
|
|
|
## Route Flows
|
|
|
|
`routes/search/search_routes.py` owns the browser/API web-search routes:
|
|
|
|
- `GET /api/search/config` returns search configuration with provider key presence, not secret values;
|
|
- `POST /api/search` calls `comprehensive_web_search(..., return_sources=True)` and returns `{context, sources, error?}`;
|
|
- `GET /api/search/providers` returns provider metadata and availability;
|
|
- `POST /api/search/query` calls one provider directly and returns `{results, provider, time, error?}` without ranking, fallback chains, cache formatting, or content fetch.
|
|
|
|
Compare mode uses both route shapes: shared presearch uses `/api/search`, while provider/search comparison panes use `/api/search/query`. Research panels can pass provider override settings through research routes into the deep-research search path.
|
|
|
|
Research provider naming is not fully normalized in the UI: some frontend selectors still use `google`, while provider dispatch expects `google_pse`.
|
|
|
|
## Search Pipeline
|
|
|
|
`services/search/core.py` owns `comprehensive_web_search()`. It coordinates provider selection, fallback chains, ranking, optional fetch/content extraction, formatted prompt context, cache invalidation, and analytics.
|
|
|
|
`services/search/service.py` owns `SearchService`, the async facade exported by `services.search` and `services`. It wraps the synchronous comprehensive search path off the event loop and maps route-style output into service result rows.
|
|
|
|
`services/search/providers.py` owns provider-specific calls for SearXNG, Brave, DuckDuckGo, Google PSE, Tavily, and Serper. `PROVIDER_INFO`, provider availability, missing-key behavior, and provider dispatch live there.
|
|
|
|
`services/search/query.py` owns query enhancement and sanitization, including stripping markdown/code-fence noise from model- or user-supplied queries before provider calls and extracting Unicode/non-ASCII capitalized entity names. `services/search/ranking.py` owns result ranking, including word-boundary title/snippet/subject matching so short query terms do not match unrelated substrings.
|
|
|
|
## Provider Settings And Fallback
|
|
|
|
`src/settings.py` owns default provider settings. The default provider is SearXNG, with DuckDuckGo as the default fallback chain. `static/js/settings.js` owns the admin search settings UI, provider key presence display, provider selection, and fallback ordering. SafeSearch is a backend/provider setting today, not a visible Settings control.
|
|
|
|
Provider API keys come from settings or environment at call time. Web config routes expose availability/presence only, non-admin settings reads are scrubbed, and chat settings tools cannot set provider credentials.
|
|
|
|
Runtime behavior:
|
|
|
|
- disabled search returns disabled/unavailable text in the comprehensive path;
|
|
- missing keyed-provider secrets return empty provider results instead of exposing secrets;
|
|
- SearXNG retries through JSON variants before HTML fallback, pins English/general-engine defaults where needed, and maps news/recency settings into provider time filters;
|
|
- comprehensive search retries providers and then walks the fallback chain;
|
|
- `/api/search/query` is a direct provider test/query path and does not use the comprehensive fallback chain. Direct provider result limits can be controlled dynamically by the caller.
|
|
|
|
## Content Fetching
|
|
|
|
`src.outbound_fetch.py` owns reusable synchronous public-URL classification, one-resolution-per-hop DNS pinning, redirect handling, and response-body budgets without search/content-extraction dependencies. `services/search/content.py` adapts those primitives and owns webpage extraction/cache/result shaping for the services path:
|
|
|
|
- public HTTP/HTTPS URL checks;
|
|
- DNS fail-closed behavior;
|
|
- rejection of localhost, metadata, private, reserved, multicast, and link-local targets;
|
|
- redirect revalidation on each hop;
|
|
- one-time public DNS resolution per hop plus an `httpcore`/`httpx` pinned
|
|
transport that connects to the validated public IP while preserving the
|
|
original URL, Host header, and TLS SNI, closing DNS-rebinding time-of-check
|
|
drift;
|
|
- metadata, Open Graph image, list, table, code block, PDF, and text extraction;
|
|
- readable text extraction for `text/*`, Markdown, `.txt`, `.json`, `.jsonl`, and JSON content types;
|
|
- central User-Agent behavior through `WEB_FETCH_USER_AGENT`;
|
|
- soft and hard download byte caps through `WEB_FETCH_SOFT_MAX_BYTES` and `WEB_FETCH_HARD_MAX_BYTES`, with declared-length and streaming-budget checks; requests prefer identity transfer encoding so compressed bodies cannot bypass the effective body cap;
|
|
- JS-heavy empty result hints;
|
|
- cache writes;
|
|
- empty/error result shape, including explicit HTTP-status failures instead of raising through callers.
|
|
|
|
`src/search/content.py` is now a compatibility alias to `services.search.content`; chat URL auto-fetch, agent `web_fetch`, and deep research keep the `src.search` import path but share the services implementation.
|
|
|
|
Agent `web_fetch` raises the per-call budget only within the global hard cap, leads tool output with a partial-content notice when the download budget truncated the page, and then applies normal tool-output truncation so the notice survives.
|
|
|
|
Content failures are caller-shaped:
|
|
|
|
- comprehensive search drops failed page fetches and keeps usable search context;
|
|
- `web_fetch` returns tool errors, including bot-protection and HTTP-status failures;
|
|
- direct URL chat prefetch turns failures into compact untrusted unavailable-page context without exposing raw URL/exception/response diagnostics;
|
|
- deep research records search/provider failures separately from extraction failures.
|
|
|
|
## Result Shapes
|
|
|
|
Search does not have one canonical result shape yet. Current shapes include:
|
|
|
|
- `/api/search`: `{context, sources, error?}`;
|
|
- `/api/search/query`: `{results, provider, time, error?}`;
|
|
- `comprehensive_web_search(return_sources=True)`: formatted context plus `{url, title}` sources;
|
|
- `SearchService.search()`: service result rows;
|
|
- agent `web_search`: tool output text plus a hidden sources marker stripped by the agent loop;
|
|
- agent `web_fetch`: fetched page text or tool error;
|
|
- deep research: findings, cited sources, optional source images, and `_last_search_error` state.
|
|
|
|
Chat/session transcript search is separate from web search but now uses `chat_messages_fts` when available, sanitizes FTS queries, and batches message lookup after FTS hits to avoid per-hit database reads.
|
|
|
|
Search owns Open Graph image extraction for fetched pages. Research owns promotion of those images into research sources and visual reports. This is not a standalone web image-search provider or gallery image proxy.
|
|
|
|
## YouTube
|
|
|
|
`services/youtube/youtube_handler.py` owns YouTube URL detection, id extraction, transcript, comment, and formatting behavior. `src/youtube_handler.py` is a compatibility alias to the canonical services module so startup `init_youtube()` state and chat imports share one implementation.
|
|
|
|
YouTube transcript and comment content is search-like external context. URL parsing covers common watch, mobile/music, embed, `/v/`, shorts, live, and `youtu.be` forms and must tolerate non-string input.
|
|
|
|
## Compatibility State
|
|
|
|
`src/search/core.py`, `src/search/providers.py`, `src/search/ranking.py`, `src/search/cache.py`, `src/search/content.py`, `src/search/query.py`, and `src/search/analytics.py` are compatibility shims or module aliases around `services.search`. Ranking helpers exposed through `src.search.ranking` include recency scoring, result ranking, naive-UTC handling, `_SPORTS_HINT_RE`, and age formats.
|
|
|
|
`src.youtube_handler` remains a compatibility import path, but it should resolve to the same module object as `services.youtube.youtube_handler`.
|
|
|
|
## Context Policy
|
|
|
|
Search results, fetched pages, Open Graph metadata, and YouTube transcript/comment content are untrusted context.
|
|
|
|
Chat search, chat URL prefetch, compare presearch, and YouTube context wrap inserted content through the shared untrusted-context message helpers. Agent `web_search`/`web_fetch` results are read-only tool outputs and must not be treated as instructions.
|
|
|
|
Deep research wraps fetched webpage content through `untrusted_context_message("webpage", content)` before extractor calls, though search result/failure shapes still differ from chat and agent tools.
|
|
|
|
## Optional And Platform Behavior
|
|
|
|
`ddgs` is optional; provider code has an HTML fallback. Search cache and analytics state live under the shared data dir and mkdir failures in read-only image layers are tolerated where possible. PDF extraction uses `pdfminer.six` only when installed. Native SearXNG defaults to `http://localhost:8080`; Docker uses the compose `searxng` service URL and pins the SearXNG image with a healthcheck.
|
|
|
|
Compose preserves retained SearXNG settings but runs `scripts/migrate_searxng_settings.py` before startup to add missing `use_default_settings: true` inheritance. The migration accepts only a regular single-document YAML mapping, preserves BOM/newline/style/ownership/mode, writes and directory-fsyncs atomically, and no-ops when the key exists. Compose treats migration failure as non-fatal so SearXNG health reports the retained-file problem instead of the wrapper command preventing startup.
|
|
|
|
`httpx` and BeautifulSoup are required runtime dependencies for the active search/fetch path.
|
|
|
|
## Current Gaps
|
|
|
|
- Search route handlers need direct tests for request body formats, provider validation, provider availability, and route error/empty-result shapes.
|
|
- Agent search, chat search prefetch, and research search do not yet share a single result/failure shape.
|
|
- `src/search` and `services/search` are mostly consolidated through shims, but import-path parity tests remain important.
|
|
- Deep-research webpage-content extraction uses the shared untrusted wrapper, but synthesis/reuse boundaries still need route/tool tests.
|
|
- Search-sourced `og_image` URLs need an explicit privacy/security decision: documented direct browser loads, public-URL validation, or a same-origin proxy.
|
|
- Route and integration tests do not fully pin chat/compare/YouTube untrusted-context insertion.
|