Files
crewAI/lib/crewai-tools
João Moura 0c74f23596 feat(tools): add URLReadTool for reading arbitrary URLs (#6834)
* feat(tools): add URLReadTool for reading arbitrary URLs

FileReadTool is confined to the local filesystem, so there was no way for
an agent to read a document that lives behind an http(s) URL. Rather than
adding a flag to FileReadTool, this adds a separate tool: granting it
grants network egress to addresses an LLM picks at runtime, and that
should be a deliberate choice rather than a toggle on a filesystem tool.

URLReadTool fetches a URL and returns its content as text. PDF and DOCX
bodies have their text extracted, HTML is stripped to visible text, and
text-shaped types (plain text, Markdown, JSON, XML, YAML, CSV) are
decoded using the charset the server declares. Any other content type is
refused rather than returned as base64, keeping the output text-only.

Requests reuse the existing SSRF protections in security/safe_requests:
validate_url resolves every hostname and rejects private, loopback,
link-local and reserved addresses (covering cloud metadata endpoints),
and safe_get never auto-follows redirects, revalidating each hop and
dropping credentials on cross-origin ones. Resolving before validating
also normalizes encoded forms, so http://2130706433/ is rejected as
127.0.0.1 without needing a string blocklist.

Adds safe_get_bounded on top of that, which streams the body and
abandons it once it crosses max_bytes. The cap counts decoded bytes,
which is what a compressed response expands into -- Content-Length
describes the wire size and cannot bound that. It also closes the
redirect hops, which stream=True would otherwise leave holding their
connections.

Two risks are documented rather than closed. Validation resolves the
hostname and requests resolves it again to connect, so DNS rebinding
remains possible; closing it needs the connection pinned to the
validated address, which would change behavior for all existing
safe_get callers. And the returned text is untrusted remote content
entering an agent's context, which input validation cannot address.

Also fixes a temp file leak in PDFLoader, which reached the same
pymupdf-from-URL path. It wrote downloads to NamedTemporaryFile with
delete=False and never unlinked them, so every PDF ingested from a URL
left a file behind. It now opens from memory, the way URLReadTool does,
which removes the leak by construction instead of relying on cleanup on
each error path; its doc.close() also moves into a finally so a failure
mid-extraction still releases the handle. PDFLoader had no test file, so
this adds one covering both paths plus a regression test asserting no
temp file is created.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(tools): address review feedback on URLReadTool

Bound start_line and line_count with ge=1 in the args schema. Both were
unbounded, and _window computes stop as start + line_count, so a negative
line_count reached islice, which rejects a negative stop. The windowing
runs outside the tool's error handling, so that escaped _run as a raw
ValueError instead of an error string. BaseTool.run validates kwargs
against args_schema, so the constraint refuses the value before any
request is made. _window also clamps its own bounds now, so it cannot
raise if called directly.

Bound the PDFLoader download with safe_get_bounded. The body is held in
memory for the whole extraction, so it needed a ceiling; it defaults to
50 MiB and takes a max_bytes kwarg to load() for callers ingesting
larger documents.

Patch the loader's own seam in its tests rather than requests.get. Both
safe_get and safe_get_bounded resolve the hostname before requesting, so
the previous mocks made the tests depend on DNS for example.com and fail
in a network-isolated runner for reasons unrelated to the loader.

Exercise the content-type fallback through run() rather than asserting on
_resolve_kind, so the tests survive a refactor of the classification
internals, and cover the octet-stream PDF, missing-type, query-string and
unknown-extension cases as observable behavior.

Add docstrings to the new tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(tools): close streamed redirect hops and widen type fallback

Four findings from the Copilot and Cursor reviews.

safe_get leaked its accumulated hops on every failure path. It closed the
response it was about to abandon but not the ones already in history, and
a caller handed an exception has no handle on them -- under stream=True
each holds its connection until its body is read or closed. The loop now
closes history before re-raising. Hops are still the caller's on success,
where they arrive via response.history.

safe_get_bounded rejected a non-positive max_bytes only after issuing the
request, and then reported it as an oversized body. It now fails before
the request. Its oversized-body error also named the requested URL rather
than the one that served the body, which differ after a redirect.

The content-type fallback consulted only the final URL for an extension,
so a .pdf link redirecting to an extensionless CDN or presigned path was
refused even though the requested URL identified the type. It now checks
the final URL first, then the requested one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 19:38:36 -03:00
..

Logo of crewAI, two people rowing on a boat

CrewAI Tools

Empower your CrewAI agents with powerful, customizable tools to elevate their capabilities and tackle sophisticated, real-world tasks.

CrewAI Tools provide the essential functionality to extend your agents, helping you rapidly enhance your automations with reliable, ready-to-use tools or custom-built solutions tailored precisely to your needs.


Homepage | Documentation | Examples | Community


Available Tools

CrewAI provides an extensive collection of powerful tools ready to enhance your agents:

  • File Management: FileReadTool, FileWriteTool
  • Web Scraping: ScrapeWebsiteTool, SeleniumScrapingTool
  • Database Integrations: MySQLSearchTool
  • Vector Database Integrations: MongoDBVectorSearchTool, QdrantVectorSearchTool, WeaviateVectorSearchTool
  • API Integrations: SerperApiTool, ExaSearchTool
  • AI-powered Tools: DallETool, VisionTool, StagehandTool

And many more robust tools to simplify your agent integrations.


Creating Custom Tools

CrewAI offers two straightforward approaches to creating custom tools:

Subclassing BaseTool

Define your tool by subclassing:

from crewai.tools import BaseTool

class MyCustomTool(BaseTool):
    name: str = "Tool Name"
    description: str = "Detailed description here."

    def _run(self, *args, **kwargs):
        # Your tool logic here

Using the tool Decorator

Quickly create lightweight tools using decorators:

from crewai import tool

@tool("Tool Name")
def my_custom_function(input):
    # Tool logic here
    return output

CrewAI Tools and MCP

CrewAI Tools supports the Model Context Protocol (MCP). It gives you access to thousands of tools from the hundreds of MCP servers out there built by the community.

Before you start using MCP with CrewAI tools, you need to install the mcp extra dependencies:

pip install crewai-tools[mcp]
# or
uv add crewai-tools --extra mcp

To quickly get started with MCP in CrewAI you have 2 options:

Option 1: Fully managed connection

In this scenario we use a contextmanager (with statement) to start and stop the the connection with the MCP server. This is done in the background and you only get to interact with the CrewAI tools corresponding to the MCP server's tools.

For an STDIO based MCP server:

from mcp import StdioServerParameters
from crewai_tools import MCPServerAdapter

serverparams = StdioServerParameters(
    command="uvx",
    args=["--quiet", "pubmedmcp@0.1.3"],
    env={"UV_PYTHON": "3.12", **os.environ},
)

with MCPServerAdapter(serverparams) as tools:
    # tools is now a list of CrewAI Tools matching 1:1 with the MCP server's tools
    agent = Agent(..., tools=tools)
    task = Task(...)
    crew = Crew(..., agents=[agent], tasks=[task])
    crew.kickoff(...)

For an SSE based MCP server:

serverparams = {"url": "http://localhost:8000/sse"}
with MCPServerAdapter(serverparams) as tools:
    # tools is now a list of CrewAI Tools matching 1:1 with the MCP server's tools
    agent = Agent(..., tools=tools)
    task = Task(...)
    crew = Crew(..., agents=[agent], tasks=[task])
    crew.kickoff(...)

Option 2: More control over the MCP connection

If you need more control over the MCP connection, you can instanciate the MCPServerAdapter into an mcp_server_adapter object which can be used to manage the connection with the MCP server and access the available tools.

important: in this case you need to call mcp_server_adapter.stop() to make sure the connection is correctly stopped. We recommend that you use a try ... finally block run to make sure the .stop() is called even in case of errors.

Here is the same example for an STDIO MCP Server:

from mcp import StdioServerParameters
from crewai_tools import MCPServerAdapter

serverparams = StdioServerParameters(
    command="uvx",
    args=["--quiet", "pubmedmcp@0.1.3"],
    env={"UV_PYTHON": "3.12", **os.environ},
)

try:
    mcp_server_adapter = MCPServerAdapter(serverparams)
    tools = mcp_server_adapter.tools
    # tools is now a list of CrewAI Tools matching 1:1 with the MCP server's tools
    agent = Agent(..., tools=tools)
    task = Task(...)
    crew = Crew(..., agents=[agent], tasks=[task])
    crew.kickoff(...)

# ** important ** don't forget to stop the connection
finally: 
    mcp_server_adapter.stop()

And finally the same thing but for an SSE MCP Server:

from mcp import StdioServerParameters
from crewai_tools import MCPServerAdapter

serverparams = {"url": "http://localhost:8000/sse"}

try:
    mcp_server_adapter = MCPServerAdapter(serverparams)
    tools = mcp_server_adapter.tools
    # tools is now a list of CrewAI Tools matching 1:1 with the MCP server's tools
    agent = Agent(..., tools=tools)
    task = Task(...)
    crew = Crew(..., agents=[agent], tasks=[task])
    crew.kickoff(...)

# ** important ** don't forget to stop the connection
finally: 
    mcp_server_adapter.stop()

Considerations & Limitations

Staying Safe with MCP

Always make sure that you trust the MCP Server before using it. Using an STDIO server will execute code on your machine. Using SSE is still not a silver bullet with many injection possible into your application from a malicious MCP server.

Limitations

  • At this time we only support tools from MCP Server not other type of primitives like prompts, resources...
  • We only return the first text output returned by the MCP Server tool using .content[0].text

Why Use CrewAI Tools?

  • Simplicity & Flexibility: Easy-to-use yet powerful enough for complex workflows.
  • Rapid Integration: Seamlessly incorporate external services, APIs, and databases.
  • Enterprise Ready: Built for stability, performance, and consistent results.

Contribution Guidelines

We welcome contributions from the community!

  1. Fork and clone the repository.
  2. Create a new branch (git checkout -b feature/my-feature).
  3. Commit your changes (git commit -m 'Add my feature').
  4. Push your branch (git push origin feature/my-feature).
  5. Open a pull request.

Developer Quickstart

pip install crewai[tools]

Development Setup

  • Install dependencies: uv sync
  • Run tests: uv run pytest
  • Run static type checking: uv run pyright
  • Set up pre-commit hooks: pre-commit install

Support and Community

Join our rapidly growing community and receive real-time support:

Build smarter, faster, and more powerful AI solutions—powered by CrewAI Tools.