3 Commits

Include renames

Author	SHA1	Message	Date
jpmschweitzerandClaude Fable 5	bd1699115a	fix: stream content fetches so the 5MB cap aborts the download ... ContentExtractor._fetch used client.get(), buffering the whole body in memory before the MAX_RESPONSE_BYTES check truncated it - the cap protected Trafilatura but not memory/bandwidth (a multi-hundred-MB URL was still fully downloaded, on up to max_urls_per_batch concurrent fetches, bounded only by the read timeout). Fetches now stream via client.stream + aiter_bytes and close the connection as soon as the cap is reached; charset still comes from the Content-Type header, available before the body is read. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L	2026-07-14 15:21:05 +02:00
jpmschweitzerandClaude Fable 5	69e6a01e65	perf: harden content extractor - async fetch, single parse, batch cap ... - Pages are fetched with httpx.AsyncClient under real connect (3s) and read timeouts on the event loop; only the CPU-bound Trafilatura parse runs in the thread pool. trafilatura.fetch_url previously ran inside the worker thread with no caller-side timeout control, so an asyncio.wait_for timeout abandoned the thread while it kept downloading for up to ~30s. - Trafilatura now runs ONCE per document via bare_extraction (text and metadata together). The old path parsed three times: extract() for text, extract(output_format='xml') whose result was discarded, and bare_extraction for metadata. - extract_batch caps full-page extractions per call (default 8, configurable); overflow URLs return unsuccessful results so the web leg falls back to the search snippet instead of fanning out unbounded downloads per search. - Responses over 5MB are truncated before parsing; thread-pool queue depth is logged for backpressure visibility. Verified live against a real URL (fetch + single-parse extraction OK). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L	2026-07-14 14:29:19 +02:00
jpmschweitzerandClaude Opus 4.5	61863ff597	feat: add RAG search endpoint with content extraction ... - Add /rag/search endpoint for web, news, and image search via SearXNG - Add /content/extract and /content/extract/batch endpoints - Add ContentExtractor client using Trafilatura for content extraction - Enhance HybridRAG web search with full content extraction - Add Redis caching for search results - Add new configuration options for search and extraction timeouts 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>	2025-12-15 15:50:48 +01:00