ContentExtractor._fetch used client.get(), buffering the whole body in
memory before the MAX_RESPONSE_BYTES check truncated it - the cap
protected Trafilatura but not memory/bandwidth (a multi-hundred-MB URL
was still fully downloaded, on up to max_urls_per_batch concurrent
fetches, bounded only by the read timeout).
Fetches now stream via client.stream + aiter_bytes and close the
connection as soon as the cap is reached; charset still comes from the
Content-Type header, available before the body is read.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
jpmschweitzerandClaude Fable 5
bd1699115a