mirror of
https://github.com/crewAIInc/crewAI.git
synced 2026-09-21 18:36:47 +00:00
9b1f4938f0
* fix(tools): pin SSRF checks to each redirect hop and peer IP validate_url only inspected the original URL string, so scraping fetches could follow a 302 to an internal address or rebind DNS between check and connect. Route safe_get through an HTTPAdapter that re-validates every hop and connects to the authorised sockaddr, and let FORCE_SAFE_PATHS ignore a tenant-supplied escape hatch on managed workers. Co-authored-by: Rip&Tear <theCyberTech@users.noreply.github.com> * Potential fix for pull request finding 'Except block handles 'BaseException'' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> * test(azure): use a plain stand-in for Responses API delegate mocks MagicMock instances are not reliably stored on Pydantic PrivateAttr via BaseLLM.__setattr__, which left _responses_delegate as None and failed last_response_id / reset_chain assertions on CI. Co-authored-by: Rip&Tear <theCyberTech@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rip&Tear <theCyberTech@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
51 lines
1.9 KiB
Plaintext
51 lines
1.9 KiB
Plaintext
---
|
|
title: Scrape Website
|
|
description: The `ScrapeWebsiteTool` is designed to extract and read the content of a specified website.
|
|
icon: magnifying-glass-location
|
|
mode: "wide"
|
|
---
|
|
|
|
# `ScrapeWebsiteTool`
|
|
|
|
<Note>
|
|
We are still working on improving tools, so there might be unexpected behavior or changes in the future.
|
|
</Note>
|
|
|
|
## Description
|
|
|
|
A tool designed to extract and read the content of a specified website. It is capable of handling various types of web pages by making HTTP requests and parsing the received HTML content.
|
|
This tool can be particularly useful for web scraping tasks, data collection, or extracting specific information from websites.
|
|
|
|
Fetches go through CrewAI's SSRF-safe HTTP helper: the requested URL and every redirect hop are checked against private and reserved ranges (including cloud metadata), and the TCP connection is pinned to an IP that passed that check.
|
|
|
|
## Installation
|
|
|
|
Install the crewai_tools package
|
|
|
|
```shell
|
|
pip install 'crewai[tools]'
|
|
```
|
|
|
|
## Example
|
|
|
|
```python
|
|
from crewai_tools import ScrapeWebsiteTool
|
|
|
|
# To enable scrapping any website it finds during it's execution
|
|
tool = ScrapeWebsiteTool()
|
|
|
|
# Initialize the tool with the website URL,
|
|
# so the agent can only scrap the content of the specified website
|
|
tool = ScrapeWebsiteTool(website_url='https://www.example.com')
|
|
|
|
# Extract the text from the site
|
|
text = tool.run()
|
|
print(text)
|
|
```
|
|
|
|
## Arguments
|
|
|
|
| Argument | Type | Description |
|
|
|:---------------|:---------|:-------------------------------------------------------------------------------------------------------------------------------------|
|
|
| **website_url** | `string` | **Mandatory** website URL to read the file. This is the primary input for the tool, specifying which website's content should be scraped and read. |
|