crewAI/crewai_tools/tools/firecrawl_scrape_website_tool

FirecrawlScrapeWebsiteTool

Description

Firecrawl is a platform for crawling and convert any website into clean markdown or structured data.

Installation

• Get an API key from firecrawl.dev and set it in environment variables (FIRECRAWL_API_KEY).
• Install the Firecrawl SDK along with crewai[tools] package:

pip install firecrawl-py 'crewai[tools]'

Example

Utilize the FirecrawlScrapeWebsiteTool as follows to allow your agent to load websites:

from crewai_tools import FirecrawlScrapeWebsiteTool

tool = FirecrawlScrapeWebsiteTool(config={"formats": ['html']})
tool.run(url="firecrawl.dev")

Arguments

• api_key: Optional. Specifies Firecrawl API key. Defaults is the FIRECRAWL_API_KEY environment variable.
• config: Optional. It contains Firecrawl API parameters.

This is the default configuration

{
    "formats": ["markdown"],
    "only_main_content": True,
    "include_tags": [],
    "exclude_tags": [],
    "headers": {},
    "wait_for": 0,
}