mirror of
https://github.com/crewAIInc/crewAI.git
synced 2026-08-10 16:32:28 +00:00
26ff36a9ae
FileReadTool is confined to the local filesystem, so there was no way for an agent to read a document that lives behind an http(s) URL. Rather than adding a flag to FileReadTool, this adds a separate tool: granting it grants network egress to addresses an LLM picks at runtime, and that should be a deliberate choice rather than a toggle on a filesystem tool. URLReadTool fetches a URL and returns its content as text. PDF and DOCX bodies have their text extracted, HTML is stripped to visible text, and text-shaped types (plain text, Markdown, JSON, XML, YAML, CSV) are decoded using the charset the server declares. Any other content type is refused rather than returned as base64, keeping the output text-only. Requests reuse the existing SSRF protections in security/safe_requests: validate_url resolves every hostname and rejects private, loopback, link-local and reserved addresses (covering cloud metadata endpoints), and safe_get never auto-follows redirects, revalidating each hop and dropping credentials on cross-origin ones. Resolving before validating also normalizes encoded forms, so http://2130706433/ is rejected as 127.0.0.1 without needing a string blocklist. Adds safe_get_bounded on top of that, which streams the body and abandons it once it crosses max_bytes. The cap counts decoded bytes, which is what a compressed response expands into -- Content-Length describes the wire size and cannot bound that. It also closes the redirect hops, which stream=True would otherwise leave holding their connections. Two risks are documented rather than closed. Validation resolves the hostname and requests resolves it again to connect, so DNS rebinding remains possible; closing it needs the connection pinned to the validated address, which would change behavior for all existing safe_get callers. And the returned text is untrusted remote content entering an agent's context, which input validation cannot address. Also fixes a temp file leak in PDFLoader, which reached the same pymupdf-from-URL path. It wrote downloads to NamedTemporaryFile with delete=False and never unlinked them, so every PDF ingested from a URL left a file behind. It now opens from memory, the way URLReadTool does, which removes the leak by construction instead of relying on cleanup on each error path; its doc.close() also moves into a finally so a failure mid-extraction still releases the handle. PDFLoader had no test file, so this adds one covering both paths plus a regression test asserting no temp file is created. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>