andre/crewAI
1
0
Fork 0
mirror of https://github.com/crewAIInc/crewAI.git synced 2026-09-20 10:03:37 +00:00
Code Issues Actions 4 Packages Projects Releases Wiki Activity

fix(tools): read octet-stream and xlsx urls in urlreadtool (#7261)

Browse Source 
* fix(tools): read octet-stream URLs by sniffing the body

URLReadTool resolved content type from the Content-Type header and then
the URL path extension. Presigned object-store links carry neither: they
pin every object to application/octet-stream and use a content hash for a
path, so a SharePoint download landing in R2 was refused outright.

Sniff the already-fetched body as a third source, consulted only after the
header and both URL extensions come back with nothing. The sniff can turn
a refusal into a read but never a read into a different read, so no URL
that works today changes behavior.

Fails closed: a zip is DOCX only when word/document.xml is in its central
directory, so an .xlsx keeps its honest refusal instead of surfacing a
misleading "failed to read DOCX"; text requires a strict, whole-body UTF-8
decode with no NUL byte; an empty body identifies nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(tools): extract text from XLSX URLs

The reported presigned SharePoint link is a spreadsheet, so sniffing the
body identified it as OOXML but still had nowhere to send it: URLReadTool
had no XLSX extractor, and the file would have been refused even with a
correct spreadsheetml Content-Type.

Read workbooks with openpyxl, already a core crewai dependency, so this
adds no new one. Sheets are emitted as CSV under a "Sheet <name>:" heading,
mirroring the PDF extractor's per-page shape. read_only streams the sheets
instead of building the whole object graph and data_only takes cached
values, both of which matter for a workbook arriving from an untrusted URL.

Cells are written through csv rather than joined, so a comma, quote or
newline inside a cell cannot corrupt the grid, and trailing phantom rows
are trimmed because Excel reports sheet dimensions generously.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(tools): bound xlsx expansion and refuse ambiguous ooxml packages

Bot review found two real defects in the XLSX extractor, both reproduced.

openpyxl pads every row up to a sheet's declared dimension, so a single
stray cell far down the sheet turned a 4.8 KB upload into 100,000 rows and
200,000 cells. Trimming only trailing blanks did not help, because the
stray cell sits at the end and keeps the last row non-empty. Blank rows are
now skipped as they stream, and a cell budget caps what any one workbook
can hand an agent -- announced in the output rather than silently applied.

A zip carrying both word/document.xml and xl/workbook.xml was classified as
DOCX. Two identities is not a positive identification, so it is refused.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(tools): keep whitespace-only xlsx cell values

Bot review, verified: openpyxl's row padding arrives as None, so testing
cells for exactly-empty drops it just as well as .strip() did while leaving
a row whose cells the author really did fill with spaces. And rstrip() on
the rendered grid removed a trailing space from the final cell along with
the line terminator; only the terminator should go.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(tools): bound xlsx scan work, not just emitted cells

The cell budget only counted cells that reached the output, and blank rows
skip before that point. A sheet can declare Excel's maximum dimension while
holding two real cells; openpyxl then pads every row out to 16,384 columns
and yields one row per gap. Measured: a 4,848-byte workbook drove 1.64
billion cell normalizations in 15.2 seconds with the budget never touched.

Charge a separate scan budget per row, before the row is normalized and
before the blank check, so the work a hostile sheet can demand is bounded
whether or not any of it is emitted. The regression test asserts the read
completes in under 5 seconds and is mutation-verified: dropping the per-row
charge takes it back to 26 seconds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(deps): clear the six pip-audit advisories

gitpython 3.1.58 has PYSEC-2026-3785 through -3788, fixed in 3.1.59; the
lock now takes 3.1.61. Its exclude-newer-package cutoff is dropped rather
than bumped -- the global 3-day cutoff has long since passed 2026-08-05, so
that per-package pin was only holding the fix back.

snowflake-sqlalchemy 1.10.0 has GHSA-8g6f-qw9x-4q6q (SQL injection and
local file disclosure), fixed in 1.11.0.

unstructured 0.18.32 has GHSA-4mvj-m6j5-pmf7, a full-read SSRF via the url=
argument of partition(). The patched 0.24.0 requires Python >=3.11 while
crewai-tools supports 3.10, so the floor carries a marker and 3.10 stays on
the old line. 0.24+ also requires beautifulsoup4>=4.14.3, so the bs4 pin
widens from ~=4.13.4 to >=4.13.4,<5 -- a widening, so no existing install
breaks. uv resolves bs4 4.13.5 on 3.10 and 4.15.0 on 3.11+.

pip-audit locally: "No known vulnerabilities found, 5 ignored", with no new
--ignore-vuln entries. Only crewai-tools[xml] grows, gaining spacy and
openai-whisper transitively through unstructured's extras.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(tools): narrow bs4 find_all results without a cast

Widening the beautifulsoup4 pin let uv resolve 4.15.0 on Python 3.11+ while
3.10 stays on 4.13.5, because the old unstructured line holds it back there.
4.15 types find_all precisely, so cast(Tag, link) became redundant and mypy
failed the 3.11-3.13 type-checker jobs while 3.10 passed.

isinstance narrowing is correct under both versions and is what AGENTS.md
asks for anyway. Verified by running mypy against 4.15.0 and again against
4.13.5: browser_toolkit is clean under both, leaving only the pre-existing
errors in crewai/rag/embeddings/providers/ibm.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(deps): declare security floors in crewai-tools, not only as overrides

Bot review caught a regression I introduced. override-dependencies replace
the whole requirement including its marker, so gating the unstructured
override on python_version >= '3.11' dropped the dependency outright on
3.10: the lock held only 0.24.1, never the 0.18 line the comment claimed.
crewai-tools[xml] would have installed no unstructured at all there.

Move the floors into lib/crewai-tools/pyproject.toml, where a marker split
means what it says -- >=0.24.0 on 3.11+, >=0.17.2 below -- and drop the
root override for unstructured entirely. The lock now carries both 0.18.32
and 0.24.1 under complementary markers.

Same reasoning applies to the other two, per the nltk precedent already in
that file: a uv override only shapes this workspace's lock, so consumers
installing crewai-tools[snowflake] or [github] were still getting the
vulnerable floors. Declared there now as well.

Also documents the tool as a fit for presigned and share links from S3, R2,
Google Drive, OneDrive and SharePoint -- the case this PR fixes -- while
saying plainly that it reads a URL and does not authenticate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: update tool specifications

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This commit is contained in:
João Moura
2026-09-04 08:21:47 -03:00
committed by GitHub
parent 92eb5f9183
commit 1e8cbef1b8
7 changed files with 1586 additions and 214 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
lib/crewai-tools/pyproject.toml
View File

@@ -12,7 +12,7 @@ dependencies = [
"requests>=2.33.0,<3",
"crewai==1.15.18",
"tiktoken>=0.8.0,<0.13",
"beautifulsoup4~=4.13.4",
"beautifulsoup4>=4.13.4,<5",
"python-docx~=1.2.0",
"youtube-transcript-api~=1.2.2",
"pymupdf~=1.26.6",
@@ -77,7 +77,10 @@ hyperbrowser = [
snowflake = [
"cryptography>=43.0.3",
"snowflake-connector-python>=3.12.4",
"snowflake-sqlalchemy>=1.7.3",
# <1.11.0 has GHSA-8g6f-qw9x-4q6q (SQL injection, local file disclosure).
# Declared here, not only as a uv override, so consumers installing
# crewai-tools[snowflake] get this floor.
"snowflake-sqlalchemy>=1.11.0",
]
singlestore = [
"singlestoredb>=1.12.4",
@@ -111,8 +114,9 @@ github = [
# GHSA-3f7w-8rr8-f37f (unguarded git option forwarding),
# GHSA-9rj7-rf2p-w77r, GHSA-4gmw-gg2m-w46p, GHSA-hh9p-6wh2-4mfc,
# GHSA-wvpp-8hx9-p66j and GHSA-jm78-9fvv-mhgr (further unguarded git
# option forwarding / arbitrary file read); force 3.1.58+.
"gitpython>=3.1.58,<4",
# option forwarding / arbitrary file read); force 3.1.58+. 3.1.58 then has
# PYSEC-2026-3785 through -3788; force 3.1.59+.
"gitpython>=3.1.59,<4",
"PyGithub==1.59.1",
]
rag = [
@@ -120,7 +124,14 @@ rag = [
"lxml>=6.1.0,<7", # 6.1.0+ required for GHSA-vfmq-68hx-4jfw (XXE in iterparse)
]
xml = [
"unstructured[local-inference, all-docs]>=0.17.2",
# <0.24.0 has GHSA-4mvj-m6j5-pmf7 (full-read SSRF via the url= argument of
# partition()). 0.24.0 requires Python >=3.11, so the floor is split rather
# than applied as a uv override: an override replaces the whole requirement
# including its marker, which would drop unstructured on 3.10 altogether.
# 3.10 therefore stays on the vulnerable line until the Python floor moves.
# TODO: collapse these two back to one entry when 3.10 support is dropped.
"unstructured[local-inference, all-docs]>=0.24.0; python_version >= '3.11'",
"unstructured[local-inference, all-docs]>=0.17.2; python_version < '3.11'",
# unstructured allows nltk>=3.9.2, but <3.10.3 still has PYSEC-2026-3726
# (symlink file read in IPIPANCorpusReader; 3.10.0-3.10.1) plus later
# 3.10.2 findings. 3.10.3 still has unpatched GHSA-8mgp-746c-j5xp

16
lib/crewai-tools/src/crewai_tools/aws/bedrock/browser/browser_toolkit.py
View File

@@ -3,7 +3,7 @@
import asyncio
import json
import logging
from typing import Any, cast
from typing import Any
from urllib.parse import urlparse
from crewai.tools import BaseTool
@@ -341,9 +341,10 @@ class ExtractHyperlinksTool(BrowserBaseTool):
soup = BeautifulSoup(content, "html.parser")
links = []
for link in soup.find_all("a", href=True):
tag = cast(Tag, link)
text = tag.get_text().strip()
href = str(tag.get("href", ""))
if not isinstance(link, Tag):
continue
text = link.get_text().strip()
href = str(link.get("href", ""))
if href.startswith(("http", "https")):
links.append({"text": text, "url": href})
@@ -371,9 +372,10 @@ class ExtractHyperlinksTool(BrowserBaseTool):
soup = BeautifulSoup(content, "html.parser")
links = []
for link in soup.find_all("a", href=True):
tag = cast(Tag, link)
text = tag.get_text().strip()
href = str(tag.get("href", ""))
if not isinstance(link, Tag):
continue
text = link.get_text().strip()
href = str(link.get("href", ""))
if href.startswith(("http", "https")):
links.append({"text": text, "url": href})

199
lib/crewai-tools/src/crewai_tools/tools/url_read_tool/url_read_tool.py
View File

@@ -2,11 +2,13 @@
from __future__ import annotations
from io import BytesIO
import csv
from io import BytesIO, StringIO
from itertools import islice
import re
from typing import Any, Final
from urllib.parse import urlparse
import zipfile
from crewai.tools import BaseTool
from pydantic import BaseModel, Field
@@ -23,6 +25,9 @@ _PDF_TYPE: Final[str] = "application/pdf"
_DOCX_TYPE: Final[str] = (
"application/vnd.openxmlformats-officedocument.wordprocessingml.document"
)
_XLSX_TYPE: Final[str] = (
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
)
_HTML_TYPES: Final[frozenset[str]] = frozenset({"text/html", "application/xhtml+xml"})
_TEXT_TYPES: Final[frozenset[str]] = frozenset(
{
@@ -39,7 +44,8 @@ _TEXT_TYPES: Final[frozenset[str]] = frozenset(
_TEXT_TYPE_SUFFIXES: Final[tuple[str, ...]] = ("+json", "+xml", "+yaml")
# Servers commonly serve static files as octet-stream, or send no type at all,
# so the extension is consulted when the header carries no usable answer.
# so the extension and then the body itself are consulted when the header
# carries no usable answer.
_UNINFORMATIVE_TYPES: Final[frozenset[str]] = frozenset(
{"", "application/octet-stream", "binary/octet-stream"}
)
@@ -52,11 +58,24 @@ _EXTENSION_TYPES: Final[dict[str, str]] = {
".md": "text/markdown",
".pdf": _PDF_TYPE,
".txt": "text/plain",
".xlsx": _XLSX_TYPE,
".xml": "application/xml",
".yaml": "application/yaml",
".yml": "application/yaml",
}
_PDF_MAGIC: Final[bytes] = b"%PDF-"
_ZIP_MAGIC: Final[bytes] = b"PK\x03\x04"
_DOCX_ZIP_ENTRY: Final[str] = "word/document.xml"
_XLSX_ZIP_ENTRY: Final[str] = "xl/workbook.xml"
# A workbook is bounded by max_bytes on the wire but not by what it expands
# into. Two separate ceilings: how much is handed to an agent, and how much
# work a hostile sheet can demand before that decision is even reached.
_XLSX_MAX_CELLS: Final[int] = 200_000
_XLSX_MAX_SCANNED_CELLS: Final[int] = 5_000_000
_HTML_PREFIXES: Final[tuple[str, ...]] = ("<!doctype html", "<html")
_HTML_PREFIX_LEN: Final[int] = max(len(prefix) for prefix in _HTML_PREFIXES)
_SPACES_PATTERN: Final[re.Pattern[str]] = re.compile(r"[ \t]+")
_NEWLINE_PATTERN: Final[re.Pattern[str]] = re.compile(r"\s+\n\s+")
@@ -100,11 +119,22 @@ class URLReadTool(BaseTool):
egress, and that should be a deliberate choice rather than a flag on a
filesystem tool.
Responses are decoded to text according to their content type. PDF and DOCX
bodies have their text extracted, HTML is stripped to visible text, and
text-shaped types (plain text, Markdown, JSON, XML, YAML, CSV) are decoded
as-is. Any other type is refused rather than returned as base64, keeping
this tool's output text-only.
Responses are decoded to text according to their content type. PDF, DOCX
and XLSX bodies have their text extracted, HTML is stripped to visible
text, and text-shaped types (plain text, Markdown, JSON, XML, YAML, CSV)
are decoded as-is. When a server names no usable type -- ``octet-stream``
from a presigned link, say -- the URL extension and then the body's own
leading bytes are consulted before giving up. Any content that none of
those identify is refused rather than returned as base64, keeping this
tool's output text-only.
That last step is what makes this a good fit for cloud storage. Presigned
S3 and Cloudflare R2 URLs, and Google Drive, OneDrive and SharePoint
download links, routinely serve a real document as
``application/octet-stream`` from a path that is a content hash with no
extension, so neither the header nor the URL says what the file is. The
tool reads a URL and does not authenticate, so the link has to already
grant access -- which is exactly what a presigned or shared link is.
Security:
Requests go through :func:`~crewai_tools.security.safe_requests.safe_get_bounded`,
@@ -135,15 +165,24 @@ class URLReadTool(BaseTool):
>>> tool = URLReadTool()
>>> content = tool.run(url="https://example.com/report.pdf")
>>> head = tool.run(url="https://example.com/data.csv", line_count=20)
>>> # A presigned link: no extension, served as octet-stream, read anyway.
>>> sheet = tool.run(
... url="https://bucket.r2.cloudflarestorage.com/a1b2c3?X-Amz-Signature=..."
... )
"""
name: str = "Read content from a URL"
description: str = (
"A tool that reads the content at a URL and returns it as text. To use "
"this tool, provide a 'url' parameter with an http:// or https:// "
"address. PDF, DOCX, HTML, JSON, XML, CSV and plain-text responses are "
"converted to text; other binary types are rejected. URLs that resolve "
"to private or internal network addresses are refused, as are responses "
"address. PDF, DOCX, XLSX, HTML, JSON, XML, CSV and plain-text "
"responses are converted to text; other binary types are rejected. "
"Well suited to presigned and share links from S3, Cloudflare R2, "
"Google Drive, OneDrive and SharePoint, which serve real documents "
"with a generic content type and no file extension; the type is "
"detected from the response bytes. The link must already grant "
"access, as a presigned or shared link does. URLs that resolve to "
"private or internal network addresses are refused, as are responses "
"over the tool's size limit. Optionally provide 'start_line' and "
"'line_count' to read only part of the content."
)
@@ -191,6 +230,8 @@ class URLReadTool(BaseTool):
return "pdf"
if media_type == _DOCX_TYPE:
return "docx"
if media_type == _XLSX_TYPE:
return "xlsx"
if media_type in _HTML_TYPES:
return "html"
if (
@@ -201,10 +242,68 @@ class URLReadTool(BaseTool):
return "text"
return None
def _resolve_kind(self, content_type: str, *urls: str) -> str | None:
"""Decide how to extract text, by content type then by URL extension.
@staticmethod
def _sniff(body: bytes) -> str | None:
"""Identify a supported type from the body's own bytes.
Presigned object-store links routinely serve every file as
octet-stream from an extensionless path, which leaves the bytes as
the only remaining evidence of what was fetched.
Fails closed: anything this cannot positively identify is refused
rather than decoded speculatively, which is what keeps binary
payloads from reaching an agent's context as mojibake.
"""
# An empty body identifies nothing. Without this it would strict-decode
# to "" and read as a successful empty text response.
if not body:
return None
if body.startswith(_PDF_MAGIC):
return "pdf"
if body.startswith(_ZIP_MAGIC):
try:
with zipfile.ZipFile(BytesIO(body)) as archive:
# Central directory only -- nothing is decompressed, so a
# zip bomb costs nothing here. Naming the part that must be
# present is what keeps a .pptx from reaching python-docx
# and surfacing as a misleading "failed to read DOCX"
# instead of an honest refusal.
names = set(archive.namelist())
except zipfile.BadZipFile:
return None
# Exactly one marker, or nothing: a package claiming to be both a
# Word document and a workbook has not been positively identified.
is_docx = _DOCX_ZIP_ENTRY in names
if is_docx == (_XLSX_ZIP_ENTRY in names):
return None
return "docx" if is_docx else "xlsx"
# Strict, whole-body decode: a prefix would split a multi-byte
# character and reject valid text. The body is already resident and
# already capped at max_bytes, so there is nothing to save by slicing.
try:
text = body.decode("utf-8")
except UnicodeDecodeError:
return None
if "\x00" in text:
return None
leading = text.lstrip("\ufeff").lstrip()[:_HTML_PREFIX_LEN].lower()
return "html" if leading.startswith(_HTML_PREFIXES) else "text"
def _resolve_kind(self, body: bytes, content_type: str, *urls: str) -> str | None:
"""Decide how to extract text: content type, URL extension, then bytes.
Each source is consulted only when every earlier one came back with
nothing, and none may override an answer an earlier one gave. That
ordering is what makes the byte sniff a pure widening of what the tool
accepts: it can turn a refusal into a read, never a read into a
different read.
Args:
body: The fetched body, sniffed last when nothing else identifies it.
content_type: The raw Content-Type header value.
*urls: URLs to consult for an extension, most authoritative first.
A ``.pdf`` link that redirects to an extensionless CDN or
@@ -212,7 +311,7 @@ class URLReadTool(BaseTool):
both ends of the chain are worth checking.
Returns:
The extractor name, or None when the content type is unsupported.
The extractor name, or None when nothing identifies the content.
"""
declared = content_type.split(";", 1)[0].strip().lower()
if declared not in _UNINFORMATIVE_TYPES:
@@ -223,7 +322,8 @@ class URLReadTool(BaseTool):
for extension, media_type in _EXTENSION_TYPES.items():
if path.endswith(extension):
return self._classify(media_type)
return None
return self._sniff(body)
def _decode(self, body: bytes, content_type: str) -> str:
"""Decode *body* using the configured, declared, or default encoding.
@@ -281,6 +381,73 @@ class URLReadTool(BaseTool):
if paragraph.text.strip()
)
@staticmethod
def _extract_xlsx(body: bytes) -> str:
"""Extract cell values from XLSX bytes, sheet by sheet, as CSV."""
try:
# openpyxl ships no stubs; same treatment as pymupdf above.
from openpyxl import load_workbook # type: ignore[import-untyped]
except ImportError as e:
raise ImportError(
"Reading XLSX URLs requires openpyxl. Install with: uv add openpyxl"
) from e
# read_only streams the sheets rather than building the whole object
# graph, and data_only takes cached values so formulas do not come
# back as "=SUM(A1:A9)". Both matter for a workbook off an untrusted URL.
workbook = load_workbook(BytesIO(body), read_only=True, data_only=True)
try:
sheets = []
scannable = _XLSX_MAX_SCANNED_CELLS
emittable = _XLSX_MAX_CELLS
truncated = False
for worksheet in workbook.worksheets:
# csv rather than a join: a cell may itself hold a comma, a
# quote or a newline, any of which would corrupt the grid.
buffer = StringIO()
writer = csv.writer(buffer, lineterminator="\n")
for row in worksheet.iter_rows(values_only=True):
# Charged before the row is touched. openpyxl pads every
# row out to the sheet's declared width, and a forged
# dimension makes each *skipped* blank row cost 16k
# normalizations -- 4.8 KB of upload drove 1.6e9 of them.
# Budgeting only what is emitted bounds none of that work.
if len(row) > scannable:
truncated = True
break
scannable -= len(row)
values = ["" if value is None else str(value) for value in row]
# Excel reports a sheet's dimension generously and openpyxl
# pads every row up to it, so one stray far-down cell turns
# a 5 KB upload into 100k blank rows. Padding arrives as
# None, so testing for exactly-empty drops it while leaving
# a cell the author really did fill with spaces alone.
if not any(values):
continue
if len(values) > emittable:
truncated = True
break
emittable -= len(values)
writer.writerow(values)
if buffer.tell():
# Only the line terminator: a bare rstrip() would also eat
# a trailing space the final cell legitimately holds.
grid = buffer.getvalue().rstrip("\n")
sheets.append(f"Sheet {worksheet.title}:\n{grid}")
if truncated:
break
finally:
workbook.close()
if not sheets:
return "[XLSX with no extractable cells]"
text = "\n\n".join(sheets)
if truncated:
text += "\n\n[Truncated: workbook is too large to read in full]"
return text
def _extract_html(self, body: bytes, content_type: str) -> str:
"""Strip HTML bytes down to visible text."""
try:
@@ -304,6 +471,8 @@ class URLReadTool(BaseTool):
return self._extract_pdf(body)
if kind == "docx":
return self._extract_docx(body)
if kind == "xlsx":
return self._extract_xlsx(body)
if kind == "html":
return self._extract_html(body, content_type)
return self._decode(body, content_type)
@@ -355,7 +524,7 @@ class URLReadTool(BaseTool):
except requests.RequestException as e:
return f"Error: Failed to fetch '{url}'. {format_error_for_display(e)}"
kind = self._resolve_kind(content_type, final_url, url)
kind = self._resolve_kind(body, content_type, final_url, url)
if kind is None:
return (
f"Error: Unsupported content type "

500
lib/crewai-tools/tests/url_read_tool_test.py
View File

@@ -1,6 +1,10 @@
import time
from io import BytesIO
from unittest.mock import patch
import zipfile
import pytest
import re
import requests
from crewai_tools import URLReadTool
@@ -56,6 +60,81 @@ def build_pdf(text: str = "Quarterly revenue was 42") -> bytes:
document.close()
PRESIGNED_URL = (
"https://temp.4d4f16c61d89ec64e760039c4ec50717.r2.cloudflarestorage.com/"
"668641/share_point/SHARE_POINT_DOWNLOAD_FILE_BY_SERVER_RELATIVE_URL/"
"response/34e077085d293bdb832a6b7c93b9e222"
"?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Signature=1954b3d4"
)
def build_docx(text: str = "Signed and delivered") -> bytes:
"""Return the bytes of a DOCX holding a single paragraph."""
from docx import Document
document = Document()
document.add_paragraph(text)
buffer = BytesIO()
document.save(buffer)
return buffer.getvalue()
def build_xlsx(rows: list[list[object]], title: str = "Sheet1") -> bytes:
"""Return the bytes of a single-sheet XLSX holding *rows*."""
from openpyxl import Workbook
workbook = Workbook()
worksheet = workbook.active
worksheet.title = title
for row in rows:
worksheet.append(row)
buffer = BytesIO()
workbook.save(buffer)
return buffer.getvalue()
def build_forged_dimension_xlsx() -> bytes:
"""Return a tiny XLSX whose sheet declares Excel's maximum dimension.
openpyxl trusts the declared width and pads every row out to it, so this
4.8 KB file otherwise drives ~1.6e9 cell normalizations.
"""
from openpyxl import Workbook
source = BytesIO()
workbook = Workbook()
worksheet = workbook.active
worksheet["A1"] = "header"
worksheet["B100000"] = "stray"
workbook.save(source)
workbook.close()
rewritten = BytesIO()
with (
zipfile.ZipFile(BytesIO(source.getvalue())) as archive,
zipfile.ZipFile(rewritten, "w", zipfile.ZIP_DEFLATED) as output,
):
for info in archive.infolist():
payload = archive.read(info.filename)
if info.filename == "xl/worksheets/sheet1.xml":
payload = re.sub(
rb'<dimension ref="[^"]*"',
b'<dimension ref="A1:XFD1048576"',
payload,
)
output.writestr(info, payload)
return rewritten.getvalue()
def build_zip(*names: str) -> bytes:
"""Return a zip holding *names*, shaped like an OOXML package."""
buffer = BytesIO()
with zipfile.ZipFile(buffer, "w") as archive:
for name in names:
archive.writestr(name, "<x/>")
return buffer.getvalue()
def fetch_result(
body: bytes,
content_type: str = "text/plain",
@@ -311,6 +390,427 @@ def test_corrupt_pdf_reports_error_without_raising():
assert result.startswith("Error: Failed to read PDF content")
def test_presigned_octet_stream_pdf_is_read_from_its_bytes():
"""The reported failure: octet-stream, no extension, real PDF bytes.
Presigned object-store links from the SharePoint connector use a content
hash for a path and pin every object to octet-stream, which leaves the
body as the only evidence of what was fetched.
"""
tool = URLReadTool()
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(
build_pdf("Signed quarterly report"),
"application/octet-stream",
PRESIGNED_URL,
)
result = tool.run(url=PRESIGNED_URL)
assert "Page 1:" in result
assert "Signed quarterly report" in result
def test_presigned_octet_stream_docx_is_read_from_its_bytes():
"""A DOCX behind the same extensionless presigned link is extracted."""
tool = URLReadTool()
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(
build_docx("Countersigned on Tuesday"),
"application/octet-stream",
PRESIGNED_URL,
)
result = tool.run(url=PRESIGNED_URL)
assert result == "Countersigned on Tuesday"
def test_octet_stream_html_is_sniffed_and_stripped():
"""HTML bytes behind an unhelpful header still lose their markup."""
tool = URLReadTool()
body = b"<!DOCTYPE html><html><body><p>Hi</p><script>x=1</script></body></html>"
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(body, "application/octet-stream", PRESIGNED_URL)
result = tool.run(url=PRESIGNED_URL)
assert "Hi" in result
assert "x=1" not in result
@pytest.mark.parametrize(
"prefix",
[
pytest.param(b"", id="bare"),
pytest.param(b"\xef\xbb\xbf", id="utf8-bom"),
pytest.param(b"\n \t", id="leading-whitespace"),
pytest.param(b"\xef\xbb\xbf\n ", id="bom-then-whitespace"),
],
)
def test_bom_and_whitespace_do_not_hide_the_html_prefix(prefix):
"""A BOM or leading whitespace must not demote HTML to raw text."""
tool = URLReadTool()
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(
prefix + b"<html><body><p>Hi</p></body></html>",
"application/octet-stream",
PRESIGNED_URL,
)
result = tool.run(url=PRESIGNED_URL)
assert "<p>" not in result
assert "Hi" in result
@pytest.mark.parametrize(
"body",
[
pytest.param(b"a,b\n1,2\n", id="csv"),
pytest.param(b'{"a": 1}', id="json"),
pytest.param(b"# Title\n\nBody text.\n", id="markdown"),
pytest.param("plain café text\n".encode(), id="utf8-plain"),
],
)
def test_octet_stream_text_bodies_are_returned_verbatim(body):
"""Decodable, NUL-free bytes are handed back untouched."""
tool = URLReadTool()
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(body, "application/octet-stream", PRESIGNED_URL)
assert tool.run(url=PRESIGNED_URL) == body.decode()
@pytest.mark.parametrize(
"entries",
[
pytest.param(("[Content_Types].xml", "ppt/presentation.xml"), id="pptx"),
pytest.param(("[Content_Types].xml", "visio/document.xml"), id="vsdx"),
pytest.param(("notes.txt",), id="plain-zip"),
],
)
def test_unsupported_zips_are_refused_without_a_misleading_docx_error(entries):
"""A .pptx must get an honest refusal, not a DOCX extraction failure.
Sniffing the zip magic alone would route any OOXML package into
python-docx, which raises and surfaces as "Failed to read DOCX content"
-- a worse answer than the refusal it replaced.
"""
tool = URLReadTool()
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(
build_zip(*entries), "application/octet-stream", PRESIGNED_URL
)
result = tool.run(url=PRESIGNED_URL)
assert "Unsupported content type 'application/octet-stream'" in result
assert "Failed to read DOCX" not in result
@pytest.mark.parametrize(
"body",
[
pytest.param(build_docx()[:120], id="truncated-docx"),
pytest.param(b"PK\x03\x04", id="magic-only"),
pytest.param(b"PK\x03\x04" + b"\xff" * 200, id="garbage-after-magic"),
],
)
def test_malformed_zip_bodies_are_refused_without_raising(body):
"""Zip magic on unreadable bytes fails closed rather than escaping _run."""
tool = URLReadTool()
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(body, "application/octet-stream", PRESIGNED_URL)
result = tool.run(url=PRESIGNED_URL)
assert "Unsupported content type" in result
def test_multibyte_character_past_a_prefix_boundary_still_reads_as_text():
"""The sniff decodes the whole body, so no character is split in half.
Decoding only a leading slice rejects valid UTF-8 whenever a multi-byte
character straddles the cut.
"""
tool = URLReadTool()
body = b"a" * 2047 + "é".encode() + b"b" * 5000
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(body, "application/octet-stream", PRESIGNED_URL)
result = tool.run(url=PRESIGNED_URL)
assert not result.startswith("Error:")
assert result == body.decode()
def test_empty_body_is_refused_rather_than_read_as_empty_text():
"""An empty body identifies nothing; it must not read as a successful "".
Without an explicit guard it strict-decodes to "" with no NUL byte and
would be classified as text.
"""
tool = URLReadTool()
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(b"", "application/octet-stream", PRESIGNED_URL)
result = tool.run(url=PRESIGNED_URL)
assert "Unsupported content type" in result
@pytest.mark.parametrize(
"body",
[
pytest.param("café,x\n".encode("latin-1"), id="latin-1"),
pytest.param("a,b\n".encode("utf-16"), id="utf-16-with-bom"),
pytest.param("a,b\n".encode("utf-16-be"), id="utf-16-be-nul-bytes"),
pytest.param(b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR", id="png"),
],
)
def test_undecodable_or_nul_bearing_bodies_fail_closed(body):
"""Fail-closed is deliberate: only strict UTF-8 without NUL reads as text.
A charset-guessing rescue here would push binary payloads into an agent's
context as mojibake, which is what the tool's text-only contract forbids.
"""
tool = URLReadTool()
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(body, "application/octet-stream", PRESIGNED_URL)
result = tool.run(url=PRESIGNED_URL)
assert "Unsupported content type 'application/octet-stream'" in result
def test_declared_content_type_wins_over_the_body_bytes():
"""A usable header is still authoritative; the sniff never overrides it."""
tool = URLReadTool()
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(
build_pdf("Should not be extracted"), "text/html", PRESIGNED_URL
)
result = tool.run(url=PRESIGNED_URL)
assert "Page 1:" not in result
assert "Should not be extracted" not in result
def test_url_extension_wins_over_the_body_bytes():
"""The extension fallback still runs ahead of the sniff."""
tool = URLReadTool()
url = "https://example.com/export.csv"
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(
build_pdf("Should not be extracted"), "application/octet-stream", url
)
result = tool.run(url=url)
assert result.startswith("%PDF")
assert "Page 1:" not in result
def test_sniffed_content_still_honors_the_line_window():
"""Windowing applies to sniffed bodies like any other."""
tool = URLReadTool()
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(
b"one\ntwo\nthree\nfour\n", "application/octet-stream", PRESIGNED_URL
)
result = tool.run(url=PRESIGNED_URL, start_line=2, line_count=2)
assert result == "two\nthree\n"
def test_presigned_octet_stream_xlsx_is_read_from_its_bytes():
"""The reported file: an XLSX behind an extensionless presigned link."""
tool = URLReadTool()
body = build_xlsx([["RFQ ID", "Title"], ["RFQ-1", "Turbine parts"]], "RFQ Header")
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(body, "application/octet-stream", PRESIGNED_URL)
result = tool.run(url=PRESIGNED_URL)
assert result == "Sheet RFQ Header:\nRFQ ID,Title\nRFQ-1,Turbine parts"
@pytest.mark.parametrize(
("content_type", "url"),
[
pytest.param(
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
PRESIGNED_URL,
id="declared-type",
),
pytest.param(
"application/octet-stream",
"https://example.com/q3.xlsx",
id="url-extension",
),
],
)
def test_xlsx_resolves_from_its_declared_type_and_its_extension(content_type, url):
"""XLSX is reachable by all three routes, not only by sniffing."""
tool = URLReadTool()
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(build_xlsx([["a", "b"]]), content_type, url)
assert tool.run(url=url) == "Sheet Sheet1:\na,b"
def test_xlsx_cells_are_csv_quoted_so_the_grid_survives():
"""A comma, quote or newline inside a cell must not corrupt the row."""
tool = URLReadTool()
body = build_xlsx([["Smith, Jane", 'He said "hi"'], ["line1\nline2", "plain"]])
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(body, "application/octet-stream", PRESIGNED_URL)
result = tool.run(url=PRESIGNED_URL)
assert '"Smith, Jane"' in result
assert '"He said ""hi"""' in result
assert '"line1\nline2"' in result
def test_xlsx_blank_rows_are_dropped():
"""Excel reports generous dimensions; phantom rows must not pad the output."""
tool = URLReadTool()
body = build_xlsx([["a"], [None], ["b"], [None], [None]])
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(body, "application/octet-stream", PRESIGNED_URL)
result = tool.run(url=PRESIGNED_URL)
assert result == "Sheet Sheet1:\na\nb"
def test_one_far_down_cell_does_not_pad_the_output():
"""A stray cell at row 100000 must not expand 5 KB into 100k blank rows.
openpyxl pads every row up to the sheet's declared dimension, so trimming
only trailing blanks left the interior padding in the agent's context.
"""
tool = URLReadTool()
from openpyxl import Workbook
workbook = Workbook()
worksheet = workbook.active
worksheet["A1"] = "header"
worksheet["B100000"] = "stray"
buffer = BytesIO()
workbook.save(buffer)
body = buffer.getvalue()
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(body, "application/octet-stream", PRESIGNED_URL)
result = tool.run(url=PRESIGNED_URL)
assert len(result.splitlines()) == 3
assert "header" in result
assert "stray" in result
def test_oversized_workbook_is_truncated_with_a_visible_notice():
"""A cap that is not announced reads as complete content. Announce it."""
tool = URLReadTool()
body = build_xlsx([[f"r{index}c{column}" for column in range(10)] for index in range(30)])
with (
patch(f"{TOOL_MODULE}._XLSX_MAX_CELLS", 50),
patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch,
):
fetch.return_value = fetch_result(body, "application/octet-stream", PRESIGNED_URL)
result = tool.run(url=PRESIGNED_URL)
assert "[Truncated: workbook is too large to read in full]" in result
assert "r0c0" in result
assert "r29c9" not in result
def test_xlsx_whitespace_only_values_survive():
"""Padding is empty, not blank -- a cell the author filled with spaces stays.
A bare rstrip() on the rendered grid would also eat a trailing space from
the final cell, and dropping rows on .strip() would delete a row whose
cells hold only spaces.
"""
tool = URLReadTool()
body = build_xlsx([["a", "trailing "], [" ", " "], ["b", "c"]])
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(body, "application/octet-stream", PRESIGNED_URL)
result = tool.run(url=PRESIGNED_URL)
assert result == "Sheet Sheet1:\na,trailing \n , \nb,c"
def test_xlsx_trailing_space_in_the_final_cell_survives():
"""The rendered grid loses its line terminator, not the last cell's space."""
tool = URLReadTool()
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(
build_xlsx([["only "]]), "application/octet-stream", PRESIGNED_URL
)
assert tool.run(url=PRESIGNED_URL) == "Sheet Sheet1:\nonly "
def test_forged_sheet_dimension_is_bounded_by_the_scan_budget():
"""A forged dimension must not buy unbounded work off a 5 KB upload.
Blank rows are skipped, so budgeting only emitted cells left the padding
free: 4.8 KB drove 1.6e9 normalizations in 15s. The scan budget is
charged per row before the row is normalized, which is what bounds it.
"""
tool = URLReadTool()
body = build_forged_dimension_xlsx()
assert len(body) < 10_000
started = time.monotonic()
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(body, "application/octet-stream", PRESIGNED_URL)
result = tool.run(url=PRESIGNED_URL)
elapsed = time.monotonic() - started
assert "[Truncated: workbook is too large to read in full]" in result
# Generous vs. the ~15s the unbounded scan took, tight enough to fail if
# the per-row charge is removed.
assert elapsed < 5, f"scan took {elapsed:.1f}s -- the budget is not bounding work"
def test_zip_claiming_to_be_both_docx_and_xlsx_is_refused():
"""A package asserting two identities has not been positively identified."""
tool = URLReadTool()
body = build_zip("[Content_Types].xml", "word/document.xml", "xl/workbook.xml")
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(body, "application/octet-stream", PRESIGNED_URL)
result = tool.run(url=PRESIGNED_URL)
assert "Unsupported content type 'application/octet-stream'" in result
assert "Failed to read" not in result
def test_xlsx_with_no_cells_says_so_instead_of_returning_nothing():
"""An empty workbook reports its emptiness rather than an empty string."""
tool = URLReadTool()
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(
build_xlsx([]), "application/octet-stream", PRESIGNED_URL
)
assert tool.run(url=PRESIGNED_URL) == "[XLSX with no extractable cells]"
def test_xlsx_formula_without_a_cached_value_reads_as_empty():
"""data_only returns cached results, so an uncalculated formula is blank.
Pinning this documents the trade: agents get "42" from a workbook Excel
has saved, never the literal "=SUM(A1:A2)".
"""
tool = URLReadTool()
body = build_xlsx([[1], [2], ["=SUM(A1:A2)"]])
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(body, "application/octet-stream", PRESIGNED_URL)
result = tool.run(url=PRESIGNED_URL)
assert "=SUM" not in result
assert result == "Sheet Sheet1:\n1\n2"
def test_corrupt_xlsx_reports_error_without_raising():
"""A zip that claims to be a workbook but is not becomes an error string."""
tool = URLReadTool()
body = build_zip("[Content_Types].xml", "xl/workbook.xml")
with patch(f"{TOOL_MODULE}.safe_get_bounded") as fetch:
fetch.return_value = fetch_result(body, "application/octet-stream", PRESIGNED_URL)
result = tool.run(url=PRESIGNED_URL)
assert result.startswith("Error: Failed to read XLSX content")
class TestSafeGetBounded:
"""Tests for the bounded-fetch helper itself."""

4
lib/crewai-tools/tool.specs.json
View File

@@ -26886,7 +26886,7 @@
}
},
{
"description": "A tool that reads the content at a URL and returns it as text. To use this tool, provide a 'url' parameter with an http:// or https:// address. PDF, DOCX, HTML, JSON, XML, CSV and plain-text responses are converted to text; other binary types are rejected. URLs that resolve to private or internal network addresses are refused, as are responses over the tool's size limit. Optionally provide 'start_line' and 'line_count' to read only part of the content.",
"description": "A tool that reads the content at a URL and returns it as text. To use this tool, provide a 'url' parameter with an http:// or https:// address. PDF, DOCX, XLSX, HTML, JSON, XML, CSV and plain-text responses are converted to text; other binary types are rejected. Well suited to presigned and share links from S3, Cloudflare R2, Google Drive, OneDrive and SharePoint, which serve real documents with a generic content type and no file extension; the type is detected from the response bytes. The link must already grant access, as a presigned or shared link does. URLs that resolve to private or internal network addresses are refused, as are responses over the tool's size limit. Optionally provide 'start_line' and 'line_count' to read only part of the content.",
"env_vars": [],
"humanized_name": "Read content from a URL",
"init_params_schema": {
@@ -26937,7 +26937,7 @@
"type": "string"
}
},
"description": "Read the content at an arbitrary URL and return it as text.\n\nUnlike :class:`~crewai_tools.tools.file_read_tool.file_read_tool.FileReadTool`,\nwhich is confined to the local filesystem, this tool performs network\nrequests to addresses the caller -- often an LLM -- chooses at runtime. It\nis a separate tool for exactly that reason: granting it is granting network\negress, and that should be a deliberate choice rather than a flag on a\nfilesystem tool.\n\nResponses are decoded to text according to their content type. PDF and DOCX\nbodies have their text extracted, HTML is stripped to visible text, and\ntext-shaped types (plain text, Markdown, JSON, XML, YAML, CSV) are decoded\nas-is. Any other type is refused rather than returned as base64, keeping\nthis tool's output text-only.\n\nSecurity:\n Requests go through :func:`~crewai_tools.security.safe_requests.safe_get_bounded`,\n which resolves each hostname and rejects it when any resolved address is\n private, loopback, link-local, or otherwise reserved -- covering cloud\n metadata endpoints and internal services. Redirects are never followed\n automatically: every hop is revalidated, and credentials are dropped on\n cross-origin hops. Bodies over ``max_bytes`` are abandoned mid-stream.\n\n The fetch pins TCP to the IP that passed validation unless\n ``CREWAI_TOOLS_ALLOW_UNSAFE_PATHS`` is set without\n ``CREWAI_TOOLS_FORCE_SAFE_PATHS``. The returned text is still\n untrusted remote content flowing into an agent's context -- a\n fetched page can attempt to instruct the agent. Network egress\n policy and prompt-level handling cover that.\n\nArgs:\n max_bytes (int): Largest response body to accept, in decoded bytes.\n Defaults to 5 MiB.\n timeout (float): Per-request timeout in seconds. Defaults to 30.\n headers (Optional[dict[str, str]]): Extra request headers. Developer\n supplied, not chosen by the model.\n encoding (Optional[str]): Force a text encoding instead of honoring the\n charset the server declares.\n **kwargs: Additional keyword arguments passed to BaseTool.\n\nExample:\n >>> tool = URLReadTool()\n >>> content = tool.run(url=\"https://example.com/report.pdf\")\n >>> head = tool.run(url=\"https://example.com/data.csv\", line_count=20)",
"description": "Read the content at an arbitrary URL and return it as text.\n\nUnlike :class:`~crewai_tools.tools.file_read_tool.file_read_tool.FileReadTool`,\nwhich is confined to the local filesystem, this tool performs network\nrequests to addresses the caller -- often an LLM -- chooses at runtime. It\nis a separate tool for exactly that reason: granting it is granting network\negress, and that should be a deliberate choice rather than a flag on a\nfilesystem tool.\n\nResponses are decoded to text according to their content type. PDF, DOCX\nand XLSX bodies have their text extracted, HTML is stripped to visible\ntext, and text-shaped types (plain text, Markdown, JSON, XML, YAML, CSV)\nare decoded as-is. When a server names no usable type -- ``octet-stream``\nfrom a presigned link, say -- the URL extension and then the body's own\nleading bytes are consulted before giving up. Any content that none of\nthose identify is refused rather than returned as base64, keeping this\ntool's output text-only.\n\nThat last step is what makes this a good fit for cloud storage. Presigned\nS3 and Cloudflare R2 URLs, and Google Drive, OneDrive and SharePoint\ndownload links, routinely serve a real document as\n``application/octet-stream`` from a path that is a content hash with no\nextension, so neither the header nor the URL says what the file is. The\ntool reads a URL and does not authenticate, so the link has to already\ngrant access -- which is exactly what a presigned or shared link is.\n\nSecurity:\n Requests go through :func:`~crewai_tools.security.safe_requests.safe_get_bounded`,\n which resolves each hostname and rejects it when any resolved address is\n private, loopback, link-local, or otherwise reserved -- covering cloud\n metadata endpoints and internal services. Redirects are never followed\n automatically: every hop is revalidated, and credentials are dropped on\n cross-origin hops. Bodies over ``max_bytes`` are abandoned mid-stream.\n\n The fetch pins TCP to the IP that passed validation unless\n ``CREWAI_TOOLS_ALLOW_UNSAFE_PATHS`` is set without\n ``CREWAI_TOOLS_FORCE_SAFE_PATHS``. The returned text is still\n untrusted remote content flowing into an agent's context -- a\n fetched page can attempt to instruct the agent. Network egress\n policy and prompt-level handling cover that.\n\nArgs:\n max_bytes (int): Largest response body to accept, in decoded bytes.\n Defaults to 5 MiB.\n timeout (float): Per-request timeout in seconds. Defaults to 30.\n headers (Optional[dict[str, str]]): Extra request headers. Developer\n supplied, not chosen by the model.\n encoding (Optional[str]): Force a text encoding instead of honoring the\n charset the server declares.\n **kwargs: Additional keyword arguments passed to BaseTool.\n\nExample:\n >>> tool = URLReadTool()\n >>> content = tool.run(url=\"https://example.com/report.pdf\")\n >>> head = tool.run(url=\"https://example.com/data.csv\", line_count=20)\n >>> # A presigned link: no extension, served as octet-stream, read anyway.\n >>> sheet = tool.run(\n ... url=\"https://bucket.r2.cloudflarestorage.com/a1b2c3?X-Amz-Signature=...\"\n ... )",
"properties": {
"encoding": {
"anyOf": [

19
pyproject.toml
View File

@@ -172,7 +172,7 @@ info = "Commits must follow Conventional Commits 1.0.0."
[tool.uv]
exclude-newer = "3 days"
# These security fixes are newer than the global supply-chain cutoff.
exclude-newer-package = { msgpack = "2026-06-20T00:00:00Z", pydantic-settings = "2026-06-20T00:00:00Z", langsmith = "2026-06-20T00:00:00Z", gitpython = "2026-08-05T00:00:00Z" }
exclude-newer-package = { msgpack = "2026-06-20T00:00:00Z", pydantic-settings = "2026-06-20T00:00:00Z", langsmith = "2026-06-20T00:00:00Z" }
# composio-core pins rich<14 but textual requires rich>=14.
# onnxruntime 1.24+ dropped Python 3.10 wheels; cap it so qdrant[fastembed] resolves on 3.10.
@@ -203,8 +203,10 @@ exclude-newer-package = { msgpack = "2026-06-20T00:00:00Z", pydantic-settings =
# TagReference); force 3.1.57+.
# gitpython <3.1.58 has GHSA-9rj7-rf2p-w77r, GHSA-4gmw-gg2m-w46p, GHSA-hh9p-6wh2-4mfc, GHSA-wvpp-8hx9-p66j and
# GHSA-jm78-9fvv-mhgr (further unguarded git option forwarding in Repo.init, read-tree and git-config, plus
# arbitrary file read via --pathspec-from-file); force 3.1.58+. Its exclude-newer-package cutoff is bumped to
# 2026-08-05 to admit that release.
# arbitrary file read via --pathspec-from-file); force 3.1.58+.
# gitpython 3.1.58 has PYSEC-2026-3785, PYSEC-2026-3786, PYSEC-2026-3787 and PYSEC-2026-3788; all fixed in
# 3.1.59, so force 3.1.59+. Its exclude-newer-package cutoff (2026-08-05) is dropped rather than bumped: the
# global 3-day cutoff is now far later than that date, so the per-package pin only blocked the fix.
# pyasn1 <0.6.4 has GHSA-8ppf-4f7h-5ppj and GHSA-hm4w-wwcw-mr6r; force 0.6.4+.
# urllib3 <2.7.0 has GHSA-qccp-gfcp-xxvc (ProxyManager cross-origin redirect leaks Authorization/Cookie) and GHSA-mf9v-mfxr-j63j (streaming decompression-bomb bypass); force 2.7.0+.
# langsmith <0.8.18 has GHSA-3644-q5cj-c5c7 (public prompt manifest deserialization, SSRF/secret disclosure)
@@ -238,6 +240,14 @@ exclude-newer-package = { msgpack = "2026-06-20T00:00:00Z", pydantic-settings =
# qdrant-client -> httpx[http2].
# torch <=2.12.1 has GHSA-rrmf-rvhw-rf47 (CVE-2025-3000): memory corruption in
# torch.jit.script; fixed in 2.13.0. Transitive via docling/unstructured extras.
# snowflake-sqlalchemy <1.11.0 has GHSA-8g6f-qw9x-4q6q (SQL injection and local file disclosure); fixed in
# 1.11.0. Declared as crewai-tools[snowflake] "snowflake-sqlalchemy>=1.7.3", which the lock resolved to 1.10.0.
# unstructured <0.24.0 has GHSA-4mvj-m6j5-pmf7 (full-read SSRF via partition(url=)); the marker-split floor
# lives in lib/crewai-tools/pyproject.toml rather than here, because an override replaces the whole
# requirement including its marker and would drop the dependency on 3.10. 0.24+ needs beautifulsoup4>=4.14.3,
# which is why the crewai-tools bs4 pin widens from ~=4.13.4 to >=4.13.4,<5 -- a widening, so no existing
# install breaks; uv resolves bs4 4.13.5 on 3.10 and 4.15.0 on 3.11+. Only crewai-tools[xml] grows, gaining
# spacy and openai-whisper transitively on 3.11+.
# snowflake-connector-python >=4.0.0,<4.7.1 has GHSA-5cc2-282f-jjq2 (CVE-2026-15925):
# TLS hostnames are not verified, so a network attacker can impersonate the endpoint;
# fixed in 4.7.1. Declared as crewai-tools[snowflake] "snowflake-connector-python>=3.12.4",
@@ -257,7 +267,7 @@ override-dependencies = [
"pypdf>=6.16.1,<7",
"uv>=0.11.15,<1",
"python-multipart>=0.0.27,<1",
"gitpython>=3.1.58,<4",
"gitpython>=3.1.59,<4",
"pyasn1>=0.6.4",
"langsmith>=0.8.18,<1",
"authlib>=1.6.12",
@@ -275,6 +285,7 @@ override-dependencies = [
"h2>=4.4.1",
"torch>=2.13.0",
"snowflake-connector-python>=4.7.1",
"snowflake-sqlalchemy>=1.11.0",
]
[tool.uv.workspace]

1041
uv.lock generated
View File

File diff suppressed because it is too large Load Diff