mirror of
https://github.com/crewAIInc/crewAI.git
synced 2026-05-01 07:13:00 +00:00
Squashed 'packages/tools/' content from commit 78317b9c
git-subtree-dir: packages/tools git-subtree-split: 78317b9c127f18bd040c1d77e3c0840cdc9a5b38
This commit is contained in:
79
crewai_tools/tools/brightdata_tool/README.md
Normal file
79
crewai_tools/tools/brightdata_tool/README.md
Normal file
@@ -0,0 +1,79 @@
|
||||
# BrightData Tools Documentation
|
||||
|
||||
## Description
|
||||
|
||||
A comprehensive suite of CrewAI tools that leverage Bright Data's powerful infrastructure for web scraping, data extraction, and search operations. These tools provide three distinct capabilities:
|
||||
|
||||
- **BrightDataDatasetTool**: Extract structured data from popular data feeds (Amazon, LinkedIn, Instagram, etc.) using pre-built datasets
|
||||
- **BrightDataSearchTool**: Perform web searches across multiple search engines with geo-targeting and device simulation
|
||||
- **BrightDataWebUnlockerTool**: Scrape any website content while bypassing bot protection mechanisms
|
||||
|
||||
## Installation
|
||||
|
||||
To incorporate these tools into your project, follow the installation instructions below:
|
||||
|
||||
```shell
|
||||
pip install crewai[tools] aiohttp requests
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Dataset Tool - Extract Amazon Product Data
|
||||
```python
|
||||
from crewai_tools import BrightDataDatasetTool
|
||||
|
||||
# Initialize with specific dataset and URL
|
||||
tool = BrightDataDatasetTool(
|
||||
dataset_type="amazon_product",
|
||||
url="https://www.amazon.com/dp/B08QB1QMJ5/"
|
||||
)
|
||||
result = tool.run()
|
||||
```
|
||||
|
||||
### Search Tool - Perform Web Search
|
||||
```python
|
||||
from crewai_tools import BrightDataSearchTool
|
||||
|
||||
# Initialize with search query
|
||||
tool = BrightDataSearchTool(
|
||||
query="latest AI trends 2025",
|
||||
search_engine="google",
|
||||
country="us"
|
||||
)
|
||||
result = tool.run()
|
||||
```
|
||||
|
||||
### Web Unlocker Tool - Scrape Website Content
|
||||
```python
|
||||
from crewai_tools import BrightDataWebUnlockerTool
|
||||
|
||||
# Initialize with target URL
|
||||
tool = BrightDataWebUnlockerTool(
|
||||
url="https://example.com",
|
||||
data_format="markdown"
|
||||
)
|
||||
result = tool.run()
|
||||
```
|
||||
|
||||
## Steps to Get Started
|
||||
|
||||
To effectively use the BrightData Tools, follow these steps:
|
||||
|
||||
1. **Package Installation**: Confirm that the `crewai[tools]` package is installed in your Python environment.
|
||||
|
||||
2. **API Key Acquisition**: Register for a Bright Data account at `https://brightdata.com/` and obtain your API credentials from your account settings.
|
||||
|
||||
3. **Environment Configuration**: Set up the required environment variables:
|
||||
```bash
|
||||
export BRIGHT_DATA_API_KEY="your_api_key_here"
|
||||
export BRIGHT_DATA_ZONE="your_zone_here"
|
||||
```
|
||||
|
||||
4. **Tool Selection**: Choose the appropriate tool based on your needs:
|
||||
- Use **DatasetTool** for structured data from supported platforms
|
||||
- Use **SearchTool** for web search operations
|
||||
- Use **WebUnlockerTool** for general website scraping
|
||||
|
||||
## Conclusion
|
||||
|
||||
By integrating BrightData Tools into your CrewAI agents, you gain access to enterprise-grade web scraping and data extraction capabilities. These tools handle complex challenges like bot protection, geo-restrictions, and data parsing, allowing you to focus on building your applications rather than managing scraping infrastructure.
|
||||
Reference in New Issue
Block a user