Lorenze Jay 387528c3fb Replay feat using db (#930)
* Cleaned up task execution to now have separate paths for async and sync execution. Updating all kickoff functions to return CrewOutput. WIP. Waiting for Joao feedback on async task execution with task_output

* Consistently storing async and sync output for context

* outline tests I need to create going forward

* Major rehaul of TaskOutput and CrewOutput. Updated all tests to work with new change. Need to add in a few final tricky async tests and add a few more to verify output types on TaskOutput and CrewOutput.

* Encountering issues with callback. Need to test on main. WIP

* working on tests. WIP

* WIP. Figuring out disconnect issue.

* Cleaned up logs now that I've isolated the issue to the LLM

* more wip.

* WIP. It looks like usage metrics has always been broken for async

* Update parent crew who is managing for_each loop

* Merge in main to bugfix/kickoff-for-each-usage-metrics

* Clean up code for review

* Add new tests

* Final cleanup. Ready for review.

* Moving copy functionality from Agent to BaseAgent

* Fix renaming issue

* Fix linting errors

* use BaseAgent instead of Agent where applicable

* Fixing missing function. Working on tests.

* WIP. Needing team to review change

* Fixing issues brought about by merge

* WIP: need to fix json encoder

* WIP need to fix encoder

* WIP

* WIP: replay working with async. need to add tests

* Implement major fixes from yesterdays group conversation. Now working on tests.

* The majority of tasks are working now. Need to fix converter class

* Fix final failing test

* Fix linting and type-checker issues

* Add more tests to fully test CrewOutput and TaskOutput changes

* Add in validation for async cannot depend on other async tasks.

* WIP: working replay feat fixing inputs, need tests

* WIP: core logic of seq and heir for executing tasks added into one

* Update validators and tests

* better logic for seq and hier

* replay working for both seq and hier just need tests

* fixed context

* added cli command + code cleanup TODO: need better refactoring

* refactoring for cleaner code

* added better tests

* removed todo comments and fixed some tests

* fix logging now all tests should pass

* cleaner code

* ensure replay is delcared when replaying specific tasks

* ensure hierarchical works

* better typing for stored_outputs and separated task_output_handler

* added better tests

* added replay feature to crew docs

* easier cli command name

* fixing changes

* using sqllite instead of .json file for logging previous task_outputs

* tools fix

* added to docs and fixed tests

* fixed .db

* fixed docs and removed unneeded comments

* separating ltm and replay db

* fixed printing colors

* added how to doc

---------

Co-authored-by: Brandon Hancock <brandon@brandonhancock.io>
2024-07-15 17:14:10 -03:00
2024-02-05 20:46:47 -08:00
2024-07-05 09:06:56 -03:00
2024-07-15 17:14:10 -03:00
2024-07-15 17:14:10 -03:00
2024-07-15 17:14:10 -03:00
2024-04-01 10:45:17 -03:00
2024-07-15 17:14:10 -03:00
2023-11-14 11:44:12 -03:00
2024-07-15 17:14:10 -03:00

Logo of crewAI, two people rowing on a boat

crewAI

🤖 crewAI: Cutting-edge framework for orchestrating role-playing, autonomous AI agents. By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, tackling complex tasks.

Homepage | Documentation | Chat with Docs | Examples | Discord

GitHub Repo stars License: MIT

Table of contents

Why CrewAI?

The power of AI collaboration has too much to offer. CrewAI is designed to enable AI agents to assume roles, share goals, and operate in a cohesive unit - much like a well-oiled crew. Whether you're building a smart assistant platform, an automated customer service ensemble, or a multi-agent research team, CrewAI provides the backbone for sophisticated multi-agent interactions.

Getting Started

To get started with CrewAI, follow these simple steps:

1. Installation

pip install crewai

If you want to install the 'crewai' package along with its optional features that include additional tools for agents, you can do so by using the following command: pip install 'crewai[tools]'. This command installs the basic package and also adds extra components which require more dependencies to function."

pip install 'crewai[tools]'

2. Setting Up Your Crew

import os
from crewai import Agent, Task, Crew, Process
from crewai_tools import SerperDevTool

os.environ["OPENAI_API_KEY"] = "YOUR_API_KEY"
os.environ["SERPER_API_KEY"] = "Your Key" # serper.dev API key

# You can choose to use a local model through Ollama for example. See https://docs.crewai.com/how-to/LLM-Connections/ for more information.

# os.environ["OPENAI_API_BASE"] = 'http://localhost:11434/v1'
# os.environ["OPENAI_MODEL_NAME"] ='openhermes'  # Adjust based on available model
# os.environ["OPENAI_API_KEY"] ='sk-111111111111111111111111111111111111111111111111'

# You can pass an optional llm attribute specifying what model you wanna use.
# It can be a local model through Ollama / LM Studio or a remote
# model like OpenAI, Mistral, Antrophic or others (https://docs.crewai.com/how-to/LLM-Connections/)
#
# import os
# os.environ['OPENAI_MODEL_NAME'] = 'gpt-3.5-turbo'
#
# OR
#
# from langchain_openai import ChatOpenAI

search_tool = SerperDevTool()

# Define your agents with roles and goals
researcher = Agent(
  role='Senior Research Analyst',
  goal='Uncover cutting-edge developments in AI and data science',
  backstory="""You work at a leading tech think tank.
  Your expertise lies in identifying emerging trends.
  You have a knack for dissecting complex data and presenting actionable insights.""",
  verbose=True,
  allow_delegation=False,
  # You can pass an optional llm attribute specifying what model you wanna use.
  # llm=ChatOpenAI(model_name="gpt-3.5", temperature=0.7),
  tools=[search_tool]
)
writer = Agent(
  role='Tech Content Strategist',
  goal='Craft compelling content on tech advancements',
  backstory="""You are a renowned Content Strategist, known for your insightful and engaging articles.
  You transform complex concepts into compelling narratives.""",
  verbose=True,
  allow_delegation=True
)

# Create tasks for your agents
task1 = Task(
  description="""Conduct a comprehensive analysis of the latest advancements in AI in 2024.
  Identify key trends, breakthrough technologies, and potential industry impacts.""",
  expected_output="Full analysis report in bullet points",
  agent=researcher
)

task2 = Task(
  description="""Using the insights provided, develop an engaging blog
  post that highlights the most significant AI advancements.
  Your post should be informative yet accessible, catering to a tech-savvy audience.
  Make it sound cool, avoid complex words so it doesn't sound like AI.""",
  expected_output="Full blog post of at least 4 paragraphs",
  agent=writer
)

# Instantiate your crew with a sequential process
crew = Crew(
  agents=[researcher, writer],
  tasks=[task1, task2],
  verbose=2, # You can set it to 1 or 2 to different logging levels
  process = Process.sequential
)

# Get your crew to work!
result = crew.kickoff()

print("######################")
print(result)

In addition to the sequential process, you can use the hierarchical process, which automatically assigns a manager to the defined crew to properly coordinate the planning and execution of tasks through delegation and validation of results. See more about the processes here.

Key Features

  • Role-Based Agent Design: Customize agents with specific roles, goals, and tools.
  • Autonomous Inter-Agent Delegation: Agents can autonomously delegate tasks and inquire amongst themselves, enhancing problem-solving efficiency.
  • Flexible Task Management: Define tasks with customizable tools and assign them to agents dynamically.
  • Processes Driven: Currently only supports sequential task execution and hierarchical processes, but more complex processes like consensual and autonomous are being worked on.
  • Save output as file: Save the output of individual tasks as a file, so you can use it later.
  • Parse output as Pydantic or Json: Parse the output of individual tasks as a Pydantic model or as a Json if you want to.
  • Works with Open Source Models: Run your crew using Open AI or open source models refer to the Connect crewAI to LLMs page for details on configuring your agents' connections to models, even ones running locally!

CrewAI Mind Map

Examples

You can test different real life examples of AI crews in the crewAI-examples repo:

Quick Tutorial

CrewAI Tutorial

Write Job Descriptions

Check out code for this example or watch a video below:

Jobs postings

Trip Planner

Check out code for this example or watch a video below:

Trip Planner

Stock Analysis

Check out code for this example or watch a video below:

Stock Analysis

Connecting Your Crew to a Model

crewAI supports using various LLMs through a variety of connection options. By default your agents will use the OpenAI API when querying the model. However, there are several other ways to allow your agents to connect to models. For example, you can configure your agents to use a local model via the Ollama tool.

Please refer to the Connect crewAI to LLMs page for details on configuring you agents' connections to models.

How CrewAI Compares

  • Autogen: While Autogen does good in creating conversational agents capable of working together, it lacks an inherent concept of process. In Autogen, orchestrating agents' interactions requires additional programming, which can become complex and cumbersome as the scale of tasks grows.

  • ChatDev: ChatDev introduced the idea of processes into the realm of AI agents, but its implementation is quite rigid. Customizations in ChatDev are limited and not geared towards production environments, which can hinder scalability and flexibility in real-world applications.

CrewAI's Advantage: CrewAI is built with production in mind. It offers the flexibility of Autogen's conversational agents and the structured process approach of ChatDev, but without the rigidity. CrewAI's processes are designed to be dynamic and adaptable, fitting seamlessly into both development and production workflows.

Contribution

CrewAI is open-source and we welcome contributions. If you're looking to contribute, please:

  • Fork the repository.
  • Create a new branch for your feature.
  • Add your feature or improvement.
  • Send a pull request.
  • We appreciate your input!

Installing Dependencies

poetry lock
poetry install

Virtual Env

poetry shell

Pre-commit hooks

pre-commit install

Running Tests

poetry run pytest

Running static type checks

poetry run mypy

Packaging

poetry build

Installing Locally

pip install dist/*.tar.gz

Telemetry

CrewAI uses anonymous telemetry to collect usage data with the main purpose of helping us improve the library by focusing our efforts on the most used features, integrations and tools.

There is NO data being collected on the prompts, tasks descriptions agents backstories or goals nor tools usage, no API calls, nor responses nor any data that is being processed by the agents, nor any secrets and env vars.

Data collected includes:

  • Version of crewAI
    • So we can understand how many users are using the latest version
  • Version of Python
    • So we can decide on what versions to better support
  • General OS (e.g. number of CPUs, macOS/Windows/Linux)
    • So we know what OS we should focus on and if we could build specific OS related features
  • Number of agents and tasks in a crew
    • So we make sure we are testing internally with similar use cases and educate people on the best practices
  • Crew Process being used
    • Understand where we should focus our efforts
  • If Agents are using memory or allowing delegation
    • Understand if we improved the features or maybe even drop them
  • If Tasks are being executed in parallel or sequentially
    • Understand if we should focus more on parallel execution
  • Language model being used
    • Improved support on most used languages
  • Roles of agents in a crew
    • Understand high level use cases so we can build better tools, integrations and examples about it
  • Tools names available
    • Understand out of the publically available tools, which ones are being used the most so we can improve them

Users can opt-in sharing the complete telemetry data by setting the share_crew attribute to True on their Crews.

License

CrewAI is released under the MIT License.

Description
Framework for orchestrating role-playing, autonomous AI agents. By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, tackling complex tasks.
Readme MIT 344 MiB
Languages
Python 97.9%
JavaScript 1.5%
CSS 0.5%
Jinja 0.1%