- Pages are fetched with httpx.AsyncClient under real connect (3s) and
read timeouts on the event loop; only the CPU-bound Trafilatura parse
runs in the thread pool. trafilatura.fetch_url previously ran inside
the worker thread with no caller-side timeout control, so an
asyncio.wait_for timeout abandoned the thread while it kept
downloading for up to ~30s.
- Trafilatura now runs ONCE per document via bare_extraction (text and
metadata together). The old path parsed three times: extract() for
text, extract(output_format='xml') whose result was discarded, and
bare_extraction for metadata.
- extract_batch caps full-page extractions per call (default 8,
configurable); overflow URLs return unsuccessful results so the web
leg falls back to the search snippet instead of fanning out unbounded
downloads per search.
- Responses over 5MB are truncated before parsing; thread-pool queue
depth is logged for backpressure visibility.
Verified live against a real URL (fetch + single-parse extraction OK).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
jpmschweitzerandClaude Fable 5
69e6a01e65