clearer usage for crewai create command

* WIP: yaml proper mapping for agents and agent

* WIP: added output_json and output_pydantic setup

* WIP: core logic added, need cleanup

* code cleanup

* updated docs and example template to use yaml to reference agents within tasks

* cleanup type errors

* Update Start-a-New-CrewAI-Project.md

---------

Co-authored-by: João Moura <joaomdmoura@gmail.com>

docs/core-concepts/Crews.md

@@ -220,7 +220,7 @@ These methods provide flexibility in how you manage and execute tasks within you
 ### Replaying from specific task:
 You can now replay from a specific task using our cli command replay.
 
-The replay_from_tasks feature in CrewAI allows you to replay from a specific task using the command-line interface (CLI). By running the command `crewai replay -t <task_id>`, you can specify the `task_id` for the replay process.
+The replay feature in CrewAI allows you to replay from a specific task using the command-line interface (CLI). By running the command `crewai replay -t <task_id>`, you can specify the `task_id` for the replay process.
 
 Kickoffs will now save the latest kickoffs returned task outputs locally for you to be able to replay from.
 

docs/how-to/Replay-tasks-from-latest-Crew-Kickoff.md

@@ -36,14 +36,14 @@ To replay from a task programmatically, use the following steps:
 2. Execute the replay command within a try-except block to handle potential errors.
 
 ```python
-   def replay_from_task():
+   def replay():
     """
     Replay the crew execution from a specific task.
     """
     task_id = '<task_id>'
     inputs = {"topic": "CrewAI Training"} # this is optional, you can pass in the inputs you want to replay otherwise uses the previous kickoffs inputs
     try:
-        YourCrewName_Crew().crew().replay_from_task(task_id=task_id, inputs=inputs)
+        YourCrewName_Crew().crew().replay(task_id=task_id, inputs=inputs)
 
     except Exception as e:
         raise Exception(f"An error occurred while replaying the crew: {e}")

docs/how-to/Start-a-New-CrewAI-Project.md

@@ -16,7 +16,7 @@ We assume you have already installed CrewAI. If not, please refer to the [instal
 To create a new project, run the following CLI command:
 
 ```shell
-$ crewai create my_project
+$ crewai create <project_name>
 ```
 
 This command will create a new project folder with the following structure:
@@ -79,8 +79,77 @@ research_candidates_task:
     {job_requirements}
   expected_output: >
     A list of 10 potential candidates with their contact information and brief profiles highlighting their suitability.
+  agent: researcher # THIS NEEDS TO MATCH THE AGENT NAME IN THE AGENTS.YAML FILE AND THE AGENT DEFINED IN THE Crew.PY FILE
+  context: # THESE NEED TO MATCH THE TASK NAMES DEFINED ABOVE AND THE TASKS.YAML FILE AND THE TASK DEFINED IN THE Crew.PY FILE
+    - researcher
 ```
 
+### Referencing Variables:
+Your defined functions with the same name will be used. For example, you can reference the agent for specific tasks from task.yaml file. Ensure your annotated agent and function name is the same otherwise your task wont recognize the reference properly.
+
+#### Example References
+agent.yaml
+```yaml
+email_summarizer:
+    role: >
+      Email Summarizer
+    goal: >
+      Summarize emails into a concise and clear summary
+    backstory: >
+      You will create a 5 bullet point summary of the report
+    llm: mixtal_llm
+```
+
+task.yaml
+```yaml
+email_summarizer_task:
+    description: >
+      Summarize the email into a 5 bullet point summary
+    expected_output: >
+      A 5 bullet point summary of the email
+    agent: email_summarizer
+    context:
+      - reporting_task
+      - research_task
+```
+
+Use the annotations are used to properly reference the agent and task in the crew.py file.
+
+### Annotations include:
+* @agent
+* @task
+* @crew
+* @llm
+* @tool
+* @callback
+* @output_json
+* @output_pydantic
+* @cache_handler
+
+
+crew.py
+```py
+...
+    @llm
+    def mixtal_llm(self):
+        return ChatGroq(temperature=0, model_name="mixtral-8x7b-32768")
+
+    @agent
+    def email_summarizer(self) -> Agent:
+        return Agent(
+            config=self.agents_config["email_summarizer"],
+        )
+    ## ...other tasks defined
+    @task
+    def email_summarizer_task(self) -> Task:
+        return Task(
+            config=self.tasks_config["email_summarizer_task"],
+        )
+...
+```
+
+
+
 ## Installing Dependencies
 
 To install the dependencies for your project, you can use Poetry. First, navigate to your project directory:

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "crewai"
-version = "0.41.0"
+version = "0.41.1"
 description = "Cutting-edge framework for orchestrating role-playing, autonomous AI agents. By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, tackling complex tasks."
 authors = ["Joao Moura <joao@crewai.com>"]
 readme = "README.md"

src/crewai/cli/cli.py

@@ -5,11 +5,11 @@ from crewai.memory.storage.kickoff_task_outputs_storage import (
     KickoffTaskOutputsSQLiteStorage,
 )
 
-
 from .create_crew import create_crew
-from .train_crew import train_crew
 from .replay_from_task import replay_task_command
 from .reset_memories_command import reset_memories_command
+from .test_crew import test_crew
+from .train_crew import train_crew
 
 
 @click.group()
@@ -126,5 +126,26 @@ def reset_memories(long, short, entities, kickoff_outputs, all):
         click.echo(f"An error occurred while resetting memories: {e}", err=True)
 
 
+@crewai.command()
+@click.option(
+    "-n",
+    "--n_iterations",
+    type=int,
+    default=3,
+    help="Number of iterations to Test the crew",
+)
+@click.option(
+    "-m",
+    "--model",
+    type=str,
+    default="gpt-4o-mini",
+    help="LLM Model to run the tests on the Crew. For now only accepting only OpenAI models.",
+)
+def test(n_iterations: int, model: str):
+    """Test the crew and evaluate the results."""
+    click.echo(f"Testing the crew for {n_iterations} iterations with model {model}")
+    test_crew(n_iterations, model)
+
+
 if __name__ == "__main__":
     crewai()

src/crewai/cli/templates/config/tasks.yaml

@@ -5,6 +5,7 @@ research_task:
     the current year is 2024.
   expected_output: >
     A list with 10 bullet points of the most relevant information about {topic}
+  agent: researcher
 
 reporting_task:
   description: >
@@ -13,3 +14,4 @@ reporting_task:
   expected_output: >
     A fully fledge reports with the mains topics, each with a full section of information.
     Formatted as markdown without '```'
+  agent: reporting_analyst

src/crewai/cli/templates/crew.py

@@ -32,14 +32,12 @@ class {{crew_name}}Crew():
 	def research_task(self) -> Task:
 		return Task(
 			config=self.tasks_config['research_task'],
-			agent=self.researcher()
 		)
 
 	@task
 	def reporting_task(self) -> Task:
 		return Task(
 			config=self.tasks_config['reporting_task'],
-			agent=self.reporting_analyst(),
 			output_file='report.md'
 		)
 

src/crewai/cli/templates/main.py

@@ -39,3 +39,16 @@ def replay():
 
     except Exception as e:
         raise Exception(f"An error occurred while replaying the crew: {e}")
+
+def test():
+    """
+    Test the crew execution and returns the results.
+    """
+    inputs = {
+        "topic": "AI LLMs"
+    }
+    try:
+        {{crew_name}}Crew().crew().test(n_iterations=int(sys.argv[1]), model=sys.argv[2], inputs=inputs)
+
+    except Exception as e:
+        raise Exception(f"An error occurred while replaying the crew: {e}")

src/crewai/cli/templates/pyproject.toml

@@ -6,12 +6,13 @@ authors = ["Your Name <you@example.com>"]
 
 [tool.poetry.dependencies]
 python = ">=3.10,<=3.13"
-crewai = { extras = ["tools"], version = "^0.41.0" }
+crewai = { extras = ["tools"], version = "^0.41.1" }
 
 [tool.poetry.scripts]
 {{folder_name}} = "{{folder_name}}.main:run"
 train = "{{folder_name}}.main:train"
 replay = "{{folder_name}}.main:replay"
+test = "{{folder_name}}.main:test"
 
 [build-system]
 requires = ["poetry-core"]

src/crewai/cli/test_crew.py

@@ -0,0 +1,30 @@
+import subprocess
+
+import click
+
+
+def test_crew(n_iterations: int, model: str) -> None:
+    """
+    Test the crew by running a command in the Poetry environment.
+
+    Args:
+        n_iterations (int): The number of iterations to test the crew.
+        model (str): The model to test the crew with.
+    """
+    command = ["poetry", "run", "test", str(n_iterations), model]
+
+    try:
+        if n_iterations <= 0:
+            raise ValueError("The number of iterations must be a positive integer.")
+
+        result = subprocess.run(command, capture_output=False, text=True, check=True)
+
+        if result.stderr:
+            click.echo(result.stderr, err=True)
+
+    except subprocess.CalledProcessError as e:
+        click.echo(f"An error occurred while testing the crew: {e}", err=True)
+        click.echo(e.output, err=True)
+
+    except Exception as e:
+        click.echo(f"An unexpected error occurred: {e}", err=True)

src/crewai/crew.py

@@ -561,8 +561,13 @@ class Crew(BaseModel):
         self._logger.log("info", "Planning the crew execution")
         result = CrewPlanner(self.tasks)._handle_crew_planning()
 
-        for task, step_plan in zip(self.tasks, result.list_of_plans_per_task):
-            task.description += step_plan
+        if result is not None and hasattr(result, "list_of_plans_per_task"):
+            for task, step_plan in zip(self.tasks, result.list_of_plans_per_task):
+                task.description += step_plan
+        else:
+            self._logger.log(
+                "info", "Something went wrong with the planning process of the Crew"
+            )
 
     def _store_execution_log(
         self,
@@ -961,5 +966,11 @@ class Crew(BaseModel):
 
         return total_usage_metrics
 
+    def test(
+        self, n_iterations: int, model: str, inputs: Optional[Dict[str, Any]] = None
+    ) -> None:
+        """Test the crew with the given inputs."""
+        pass
+
     def __repr__(self):
         return f"Crew(id={self.id}, process={self.process}, number_of_agents={len(self.agents)}, number_of_tasks={len(self.tasks)})"

src/crewai/project/__init__.py

@@ -1,2 +1,25 @@
-from .annotations import agent, crew, task
+from .annotations import (
+    agent,
+    crew,
+    task,
+    output_json,
+    output_pydantic,
+    tool,
+    callback,
+    llm,
+    cache_handler,
+)
 from .crew_base import CrewBase
+
+__all__ = [
+    "agent",
+    "crew",
+    "task",
+    "output_json",
+    "output_pydantic",
+    "tool",
+    "callback",
+    "CrewBase",
+    "llm",
+    "cache_handler",
+]

src/crewai/project/annotations.py

@@ -30,6 +30,37 @@ def agent(func):
     return func
 
 
+def llm(func):
+    func.is_llm = True
+    func = memoize(func)
+    return func
+
+
+def output_json(cls):
+    cls.is_output_json = True
+    return cls
+
+
+def output_pydantic(cls):
+    cls.is_output_pydantic = True
+    return cls
+
+
+def tool(func):
+    func.is_tool = True
+    return memoize(func)
+
+
+def callback(func):
+    func.is_callback = True
+    return memoize(func)
+
+
+def cache_handler(func):
+    func.is_cache_handler = True
+    return memoize(func)
+
+
 def crew(func):
     def wrapper(self, *args, **kwargs):
         instantiated_tasks = []

src/crewai/project/crew_base.py

@@ -1,6 +1,7 @@
 import inspect
 import os
 from pathlib import Path
+from typing import Any, Callable, Dict
 
 import yaml
 from dotenv import load_dotenv
@@ -20,11 +21,6 @@ def CrewBase(cls):
                 base_directory = Path(frame_info.filename).parent.resolve()
                 break
 
-        if base_directory is None:
-            raise Exception(
-                "Unable to dynamically determine the project's base directory, you must run it from the project's root directory."
-            )
-
         original_agents_config_path = getattr(
             cls, "agents_config", "config/agents.yaml"
         )
@@ -32,12 +28,20 @@ def CrewBase(cls):
 
         def __init__(self, *args, **kwargs):
             super().__init__(*args, **kwargs)
+
+            if self.base_directory is None:
+                raise Exception(
+                    "Unable to dynamically determine the project's base directory, you must run it from the project's root directory."
+                )
+
             self.agents_config = self.load_yaml(
                 os.path.join(self.base_directory, self.original_agents_config_path)
             )
             self.tasks_config = self.load_yaml(
                 os.path.join(self.base_directory, self.original_tasks_config_path)
             )
+            self.map_all_agent_variables()
+            self.map_all_task_variables()
 
         @staticmethod
         def load_yaml(config_path: str):
@@ -45,4 +49,138 @@ def CrewBase(cls):
                 # parsedContent = YamlParser.parse(file)  # type: ignore # Argument 1 to "parse" has incompatible type "TextIOWrapper"; expected "YamlParser"
                 return yaml.safe_load(file)
 
+        def _get_all_functions(self):
+            return {
+                name: getattr(self, name)
+                for name in dir(self)
+                if callable(getattr(self, name))
+            }
+
+        def _filter_functions(
+            self, functions: Dict[str, Callable], attribute: str
+        ) -> Dict[str, Callable]:
+            return {
+                name: func
+                for name, func in functions.items()
+                if hasattr(func, attribute)
+            }
+
+        def map_all_agent_variables(self) -> None:
+            all_functions = self._get_all_functions()
+            llms = self._filter_functions(all_functions, "is_llm")
+            tool_functions = self._filter_functions(all_functions, "is_tool")
+            cache_handler_functions = self._filter_functions(
+                all_functions, "is_cache_handler"
+            )
+            callbacks = self._filter_functions(all_functions, "is_callback")
+            agents = self._filter_functions(all_functions, "is_agent")
+
+            for agent_name, agent_info in self.agents_config.items():
+                self._map_agent_variables(
+                    agent_name,
+                    agent_info,
+                    agents,
+                    llms,
+                    tool_functions,
+                    cache_handler_functions,
+                    callbacks,
+                )
+
+        def _map_agent_variables(
+            self,
+            agent_name: str,
+            agent_info: Dict[str, Any],
+            agents: Dict[str, Callable],
+            llms: Dict[str, Callable],
+            tool_functions: Dict[str, Callable],
+            cache_handler_functions: Dict[str, Callable],
+            callbacks: Dict[str, Callable],
+        ) -> None:
+            if llm := agent_info.get("llm"):
+                self.agents_config[agent_name]["llm"] = llms[llm]()
+
+            if tools := agent_info.get("tools"):
+                self.agents_config[agent_name]["tools"] = [
+                    tool_functions[tool]() for tool in tools
+                ]
+
+            if function_calling_llm := agent_info.get("function_calling_llm"):
+                self.agents_config[agent_name]["function_calling_llm"] = agents[
+                    function_calling_llm
+                ]()
+
+            if step_callback := agent_info.get("step_callback"):
+                self.agents_config[agent_name]["step_callback"] = callbacks[
+                    step_callback
+                ]()
+
+            if cache_handler := agent_info.get("cache_handler"):
+                self.agents_config[agent_name]["cache_handler"] = (
+                    cache_handler_functions[cache_handler]()
+                )
+
+        def map_all_task_variables(self) -> None:
+            all_functions = self._get_all_functions()
+            agents = self._filter_functions(all_functions, "is_agent")
+            tasks = self._filter_functions(all_functions, "is_task")
+            output_json_functions = self._filter_functions(
+                all_functions, "is_output_json"
+            )
+            tool_functions = self._filter_functions(all_functions, "is_tool")
+            callback_functions = self._filter_functions(all_functions, "is_callback")
+            output_pydantic_functions = self._filter_functions(
+                all_functions, "is_output_pydantic"
+            )
+
+            for task_name, task_info in self.tasks_config.items():
+                self._map_task_variables(
+                    task_name,
+                    task_info,
+                    agents,
+                    tasks,
+                    output_json_functions,
+                    tool_functions,
+                    callback_functions,
+                    output_pydantic_functions,
+                )
+
+        def _map_task_variables(
+            self,
+            task_name: str,
+            task_info: Dict[str, Any],
+            agents: Dict[str, Callable],
+            tasks: Dict[str, Callable],
+            output_json_functions: Dict[str, Callable],
+            tool_functions: Dict[str, Callable],
+            callback_functions: Dict[str, Callable],
+            output_pydantic_functions: Dict[str, Callable],
+        ) -> None:
+            if context_list := task_info.get("context"):
+                self.tasks_config[task_name]["context"] = [
+                    tasks[context_task_name]() for context_task_name in context_list
+                ]
+
+            if tools := task_info.get("tools"):
+                self.tasks_config[task_name]["tools"] = [
+                    tool_functions[tool]() for tool in tools
+                ]
+
+            if agent_name := task_info.get("agent"):
+                self.tasks_config[task_name]["agent"] = agents[agent_name]()
+
+            if output_json := task_info.get("output_json"):
+                self.tasks_config[task_name]["output_json"] = output_json_functions[
+                    output_json
+                ]
+
+            if output_pydantic := task_info.get("output_pydantic"):
+                self.tasks_config[task_name]["output_pydantic"] = (
+                    output_pydantic_functions[output_pydantic]
+                )
+
+            if callbacks := task_info.get("callbacks"):
+                self.tasks_config[task_name]["callbacks"] = [
+                    callback_functions[callback]() for callback in callbacks
+                ]
+
     return WrappedClass

src/crewai/utilities/planning_handler.py

@@ -1,4 +1,4 @@
-from typing import List
+from typing import List, Optional
 
 from pydantic import BaseModel
 
@@ -14,14 +14,14 @@ class CrewPlanner:
     def __init__(self, tasks: List[Task]):
         self.tasks = tasks
 
-    def _handle_crew_planning(self):
+    def _handle_crew_planning(self) -> Optional[BaseModel]:
         """Handles the Crew planning by creating detailed step-by-step plans for each task."""
         planning_agent = self._create_planning_agent()
         tasks_summary = self._create_tasks_summary()
 
         planner_task = self._create_planner_task(planning_agent, tasks_summary)
 
-        return planner_task.execute_sync()
+        return planner_task.execute_sync().pydantic
 
     def _create_planning_agent(self) -> Agent:
         """Creates the planning agent for the crew planning."""

tests/cassettes/test_replay_setup_context.yaml

@@ -0,0 +1,163 @@
+interactions:
+- request:
+    body: '{"messages": [{"content": "You are test_agent. Test Description\nYour personal
+      goal is: Test GoalTo give my best complete final answer to the task use the
+      exact following format:\n\nThought: I now can give a great answer\nFinal Answer:
+      my best complete final answer to the task.\nYour final answer must be the great
+      and the most complete as possible, it must be outcome described.\n\nI MUST use
+      these formats, my job depends on it!\nCurrent Task: Test Task\n\nThis is the
+      expect criteria for your final answer: Say Hi to John \n you MUST return the
+      actual complete content as the final answer, not a summary.\n\nThis is the context
+      you''re working with:\ncontext raw output\n\nBegin! This is VERY important to
+      you, use the tools available and give your best Final Answer, your job depends
+      on it!\n\nThought:\n", "role": "user"}], "model": "gpt-4o", "logprobs": false,
+      "n": 1, "stop": ["\nObservation"], "stream": true, "temperature": 0.7}'
+    headers:
+      accept:
+      - application/json
+      accept-encoding:
+      - gzip, deflate
+      connection:
+      - keep-alive
+      content-length:
+      - '937'
+      content-type:
+      - application/json
+      host:
+      - api.openai.com
+      user-agent:
+      - OpenAI/Python 1.36.0
+      x-stainless-arch:
+      - arm64
+      x-stainless-async:
+      - 'false'
+      x-stainless-lang:
+      - python
+      x-stainless-os:
+      - MacOS
+      x-stainless-package-version:
+      - 1.36.0
+      x-stainless-runtime:
+      - CPython
+      x-stainless-runtime-version:
+      - 3.11.7
+    method: POST
+    uri: https://api.openai.com/v1/chat/completions
+  response:
+    body:
+      string: 'data: {"id":"chatcmpl-9n6wQ7bzZKcXAmiNgs4nn5Of0EFiM","object":"chat.completion.chunk","created":1721491782,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_400f27fa1f","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wQ7bzZKcXAmiNgs4nn5Of0EFiM","object":"chat.completion.chunk","created":1721491782,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_400f27fa1f","choices":[{"index":0,"delta":{"content":"Thought"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wQ7bzZKcXAmiNgs4nn5Of0EFiM","object":"chat.completion.chunk","created":1721491782,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_400f27fa1f","choices":[{"index":0,"delta":{"content":":"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wQ7bzZKcXAmiNgs4nn5Of0EFiM","object":"chat.completion.chunk","created":1721491782,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_400f27fa1f","choices":[{"index":0,"delta":{"content":"
+        I"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wQ7bzZKcXAmiNgs4nn5Of0EFiM","object":"chat.completion.chunk","created":1721491782,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_400f27fa1f","choices":[{"index":0,"delta":{"content":"
+        now"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wQ7bzZKcXAmiNgs4nn5Of0EFiM","object":"chat.completion.chunk","created":1721491782,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_400f27fa1f","choices":[{"index":0,"delta":{"content":"
+        can"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wQ7bzZKcXAmiNgs4nn5Of0EFiM","object":"chat.completion.chunk","created":1721491782,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_400f27fa1f","choices":[{"index":0,"delta":{"content":"
+        give"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wQ7bzZKcXAmiNgs4nn5Of0EFiM","object":"chat.completion.chunk","created":1721491782,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_400f27fa1f","choices":[{"index":0,"delta":{"content":"
+        a"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wQ7bzZKcXAmiNgs4nn5Of0EFiM","object":"chat.completion.chunk","created":1721491782,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_400f27fa1f","choices":[{"index":0,"delta":{"content":"
+        great"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wQ7bzZKcXAmiNgs4nn5Of0EFiM","object":"chat.completion.chunk","created":1721491782,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_400f27fa1f","choices":[{"index":0,"delta":{"content":"
+        answer"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wQ7bzZKcXAmiNgs4nn5Of0EFiM","object":"chat.completion.chunk","created":1721491782,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_400f27fa1f","choices":[{"index":0,"delta":{"content":"\n"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wQ7bzZKcXAmiNgs4nn5Of0EFiM","object":"chat.completion.chunk","created":1721491782,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_400f27fa1f","choices":[{"index":0,"delta":{"content":"Final"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wQ7bzZKcXAmiNgs4nn5Of0EFiM","object":"chat.completion.chunk","created":1721491782,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_400f27fa1f","choices":[{"index":0,"delta":{"content":"
+        Answer"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wQ7bzZKcXAmiNgs4nn5Of0EFiM","object":"chat.completion.chunk","created":1721491782,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_400f27fa1f","choices":[{"index":0,"delta":{"content":":"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wQ7bzZKcXAmiNgs4nn5Of0EFiM","object":"chat.completion.chunk","created":1721491782,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_400f27fa1f","choices":[{"index":0,"delta":{"content":"
+        Hi"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wQ7bzZKcXAmiNgs4nn5Of0EFiM","object":"chat.completion.chunk","created":1721491782,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_400f27fa1f","choices":[{"index":0,"delta":{"content":"
+        John"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wQ7bzZKcXAmiNgs4nn5Of0EFiM","object":"chat.completion.chunk","created":1721491782,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_400f27fa1f","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}]}
+
+
+        data: [DONE]
+
+
+        '
+    headers:
+      CF-Cache-Status:
+      - DYNAMIC
+      CF-RAY:
+      - 8a643794fe0341e9-EWR
+      Connection:
+      - keep-alive
+      Content-Type:
+      - text/event-stream; charset=utf-8
+      Date:
+      - Sat, 20 Jul 2024 16:09:42 GMT
+      Server:
+      - cloudflare
+      Set-Cookie:
+      - __cf_bm=7kfE3khl2E.6zM44yel5nToHzdtz0QeQ4wkLuGYyqSs-1721491782-1.0.1.1-XUb95eXTriHvSUSCH.TCyAmCGCbPK6L7p_tRTDBon8Fo6ns8TDbDoDGA.wVCFI4MTXSxkqrjD0GpYDj4GBTeSQ;
+        path=/; expires=Sat, 20-Jul-24 16:39:42 GMT; domain=.api.openai.com; HttpOnly;
+        Secure; SameSite=None
+      - _cfuvid=iN41lAEk.DjpRMAtG.K0NEvIN0xB9eS0CUCU2iWmjv4-1721491782137-0.0.1.1-604800000;
+        path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None
+      Transfer-Encoding:
+      - chunked
+      X-Content-Type-Options:
+      - nosniff
+      alt-svc:
+      - h3=":443"; ma=86400
+      openai-organization:
+      - crewai-iuxna1
+      openai-processing-ms:
+      - '104'
+      openai-version:
+      - '2020-10-01'
+      strict-transport-security:
+      - max-age=15552000; includeSubDomains; preload
+      x-ratelimit-limit-requests:
+      - '10000'
+      x-ratelimit-limit-tokens:
+      - '30000000'
+      x-ratelimit-remaining-requests:
+      - '9999'
+      x-ratelimit-remaining-tokens:
+      - '29999791'
+      x-ratelimit-reset-requests:
+      - 6ms
+      x-ratelimit-reset-tokens:
+      - 0s
+      x-request-id:
+      - req_4d90924dd28a0fb48c857f03515f0ca8
+    status:
+      code: 200
+      message: OK
+version: 1

tests/cassettes/test_replay_with_context.yaml

@@ -0,0 +1,159 @@
+interactions:
+- request:
+    body: '{"messages": [{"content": "You are test_agent. Test Description\nYour personal
+      goal is: Test GoalTo give my best complete final answer to the task use the
+      exact following format:\n\nThought: I now can give a great answer\nFinal Answer:
+      my best complete final answer to the task.\nYour final answer must be the great
+      and the most complete as possible, it must be outcome described.\n\nI MUST use
+      these formats, my job depends on it!\nCurrent Task: Test Task\n\nThis is the
+      expect criteria for your final answer: Say Hi \n you MUST return the actual
+      complete content as the final answer, not a summary.\n\nThis is the context
+      you''re working with:\ncontext raw output\n\nBegin! This is VERY important to
+      you, use the tools available and give your best Final Answer, your job depends
+      on it!\n\nThought:\n", "role": "user"}], "model": "gpt-4o", "logprobs": false,
+      "n": 1, "stop": ["\nObservation"], "stream": true, "temperature": 0.7}'
+    headers:
+      accept:
+      - application/json
+      accept-encoding:
+      - gzip, deflate
+      connection:
+      - keep-alive
+      content-length:
+      - '929'
+      content-type:
+      - application/json
+      host:
+      - api.openai.com
+      user-agent:
+      - OpenAI/Python 1.36.0
+      x-stainless-arch:
+      - arm64
+      x-stainless-async:
+      - 'false'
+      x-stainless-lang:
+      - python
+      x-stainless-os:
+      - MacOS
+      x-stainless-package-version:
+      - 1.36.0
+      x-stainless-runtime:
+      - CPython
+      x-stainless-runtime-version:
+      - 3.11.7
+    method: POST
+    uri: https://api.openai.com/v1/chat/completions
+  response:
+    body:
+      string: 'data: {"id":"chatcmpl-9n6wPAClsh4tUGoLYKLh3VoX1vlAx","object":"chat.completion.chunk","created":1721491781,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_c4e5b6fa31","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wPAClsh4tUGoLYKLh3VoX1vlAx","object":"chat.completion.chunk","created":1721491781,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_c4e5b6fa31","choices":[{"index":0,"delta":{"content":"Thought"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wPAClsh4tUGoLYKLh3VoX1vlAx","object":"chat.completion.chunk","created":1721491781,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_c4e5b6fa31","choices":[{"index":0,"delta":{"content":":"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wPAClsh4tUGoLYKLh3VoX1vlAx","object":"chat.completion.chunk","created":1721491781,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_c4e5b6fa31","choices":[{"index":0,"delta":{"content":"
+        I"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wPAClsh4tUGoLYKLh3VoX1vlAx","object":"chat.completion.chunk","created":1721491781,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_c4e5b6fa31","choices":[{"index":0,"delta":{"content":"
+        now"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wPAClsh4tUGoLYKLh3VoX1vlAx","object":"chat.completion.chunk","created":1721491781,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_c4e5b6fa31","choices":[{"index":0,"delta":{"content":"
+        can"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wPAClsh4tUGoLYKLh3VoX1vlAx","object":"chat.completion.chunk","created":1721491781,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_c4e5b6fa31","choices":[{"index":0,"delta":{"content":"
+        give"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wPAClsh4tUGoLYKLh3VoX1vlAx","object":"chat.completion.chunk","created":1721491781,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_c4e5b6fa31","choices":[{"index":0,"delta":{"content":"
+        a"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wPAClsh4tUGoLYKLh3VoX1vlAx","object":"chat.completion.chunk","created":1721491781,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_c4e5b6fa31","choices":[{"index":0,"delta":{"content":"
+        great"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wPAClsh4tUGoLYKLh3VoX1vlAx","object":"chat.completion.chunk","created":1721491781,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_c4e5b6fa31","choices":[{"index":0,"delta":{"content":"
+        answer"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wPAClsh4tUGoLYKLh3VoX1vlAx","object":"chat.completion.chunk","created":1721491781,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_c4e5b6fa31","choices":[{"index":0,"delta":{"content":"\n"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wPAClsh4tUGoLYKLh3VoX1vlAx","object":"chat.completion.chunk","created":1721491781,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_c4e5b6fa31","choices":[{"index":0,"delta":{"content":"Final"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wPAClsh4tUGoLYKLh3VoX1vlAx","object":"chat.completion.chunk","created":1721491781,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_c4e5b6fa31","choices":[{"index":0,"delta":{"content":"
+        Answer"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wPAClsh4tUGoLYKLh3VoX1vlAx","object":"chat.completion.chunk","created":1721491781,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_c4e5b6fa31","choices":[{"index":0,"delta":{"content":":"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wPAClsh4tUGoLYKLh3VoX1vlAx","object":"chat.completion.chunk","created":1721491781,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_c4e5b6fa31","choices":[{"index":0,"delta":{"content":"
+        Hi"},"logprobs":null,"finish_reason":null}]}
+
+
+        data: {"id":"chatcmpl-9n6wPAClsh4tUGoLYKLh3VoX1vlAx","object":"chat.completion.chunk","created":1721491781,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_c4e5b6fa31","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}]}
+
+
+        data: [DONE]
+
+
+        '
+    headers:
+      CF-Cache-Status:
+      - DYNAMIC
+      CF-RAY:
+      - 8a643791a80e8c96-EWR
+      Connection:
+      - keep-alive
+      Content-Type:
+      - text/event-stream; charset=utf-8
+      Date:
+      - Sat, 20 Jul 2024 16:09:41 GMT
+      Server:
+      - cloudflare
+      Set-Cookie:
+      - __cf_bm=cam5sECdaTzbttLIOaiuvh9flDIAXp_FLPODnDEOn6k-1721491781-1.0.1.1-hyFl43P7HIWZsGueyWuDeO579sZ41as2mvrM.cQS1E8KSLG2ZZ0DxDGbVvHYRO0eflTUJohgZu6CGltvjQfMtQ;
+        path=/; expires=Sat, 20-Jul-24 16:39:41 GMT; domain=.api.openai.com; HttpOnly;
+        Secure; SameSite=None
+      - _cfuvid=nmlgS.bqXAu0rZ.OlHPfXrIrdnVgrBSW3e0UuU3N5ng-1721491781661-0.0.1.1-604800000;
+        path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None
+      Transfer-Encoding:
+      - chunked
+      X-Content-Type-Options:
+      - nosniff
+      alt-svc:
+      - h3=":443"; ma=86400
+      openai-organization:
+      - crewai-iuxna1
+      openai-processing-ms:
+      - '126'
+      openai-version:
+      - '2020-10-01'
+      strict-transport-security:
+      - max-age=15552000; includeSubDomains; preload
+      x-ratelimit-limit-requests:
+      - '10000'
+      x-ratelimit-limit-tokens:
+      - '30000000'
+      x-ratelimit-remaining-requests:
+      - '9999'
+      x-ratelimit-remaining-tokens:
+      - '29999794'
+      x-ratelimit-reset-requests:
+      - 6ms
+      x-ratelimit-reset-tokens:
+      - 0s
+      x-request-id:
+      - req_31484eeb0af939af4e0d9c47441ba2db
+    status:
+      code: 200
+      message: OK
+version: 1

tests/cli/cli_test.py

@@ -3,7 +3,7 @@ from unittest import mock
 import pytest
 from click.testing import CliRunner
 
-from crewai.cli.cli import train, version, reset_memories
+from crewai.cli.cli import reset_memories, test, train, version
 
 
 @pytest.fixture
@@ -133,3 +133,33 @@ def test_version_command_with_tools(runner):
         "crewai tools version:" in result.output
         or "crewai tools not installed" in result.output
     )
+
+
+@mock.patch("crewai.cli.cli.test_crew")
+def test_test_default_iterations(test_crew, runner):
+    result = runner.invoke(test)
+
+    test_crew.assert_called_once_with(3, "gpt-4o-mini")
+    assert result.exit_code == 0
+    assert "Testing the crew for 3 iterations with model gpt-4o-mini" in result.output
+
+
+@mock.patch("crewai.cli.cli.test_crew")
+def test_test_custom_iterations(test_crew, runner):
+    result = runner.invoke(test, ["--n_iterations", "5", "--model", "gpt-4o"])
+
+    test_crew.assert_called_once_with(5, "gpt-4o")
+    assert result.exit_code == 0
+    assert "Testing the crew for 5 iterations with model gpt-4o" in result.output
+
+
+@mock.patch("crewai.cli.cli.test_crew")
+def test_test_invalid_string_iterations(test_crew, runner):
+    result = runner.invoke(test, ["--n_iterations", "invalid"])
+
+    test_crew.assert_not_called()
+    assert result.exit_code == 2
+    assert (
+        "Usage: test [OPTIONS]\nTry 'test --help' for help.\n\nError: Invalid value for '-n' / '--n_iterations': 'invalid' is not a valid integer.\n"
+        in result.output
+    )

tests/cli/test_crew_test.py

@@ -0,0 +1,97 @@
+import subprocess
+from unittest import mock
+
+import pytest
+
+from crewai.cli import test_crew
+
+
+@pytest.mark.parametrize(
+    "n_iterations,model",
+    [
+        (1, "gpt-4o"),
+        (5, "gpt-3.5-turbo"),
+        (10, "gpt-4"),
+    ],
+)
+@mock.patch("crewai.cli.test_crew.subprocess.run")
+def test_crew_success(mock_subprocess_run, n_iterations, model):
+    """Test the crew function for successful execution."""
+    mock_subprocess_run.return_value = subprocess.CompletedProcess(
+        args=f"poetry run test {n_iterations} {model}", returncode=0
+    )
+    result = test_crew.test_crew(n_iterations, model)
+
+    mock_subprocess_run.assert_called_once_with(
+        ["poetry", "run", "test", str(n_iterations), model],
+        capture_output=False,
+        text=True,
+        check=True,
+    )
+    assert result is None
+
+
+@mock.patch("crewai.cli.test_crew.click")
+def test_test_crew_zero_iterations(click):
+    test_crew.test_crew(0, "gpt-4o")
+    click.echo.assert_called_once_with(
+        "An unexpected error occurred: The number of iterations must be a positive integer.",
+        err=True,
+    )
+
+
+@mock.patch("crewai.cli.test_crew.click")
+def test_test_crew_negative_iterations(click):
+    test_crew.test_crew(-2, "gpt-4o")
+    click.echo.assert_called_once_with(
+        "An unexpected error occurred: The number of iterations must be a positive integer.",
+        err=True,
+    )
+
+
+@mock.patch("crewai.cli.test_crew.click")
+@mock.patch("crewai.cli.test_crew.subprocess.run")
+def test_test_crew_called_process_error(mock_subprocess_run, click):
+    n_iterations = 5
+    mock_subprocess_run.side_effect = subprocess.CalledProcessError(
+        returncode=1,
+        cmd=["poetry", "run", "test", str(n_iterations), "gpt-4o"],
+        output="Error",
+        stderr="Some error occurred",
+    )
+    test_crew.test_crew(n_iterations, "gpt-4o")
+
+    mock_subprocess_run.assert_called_once_with(
+        ["poetry", "run", "test", "5", "gpt-4o"],
+        capture_output=False,
+        text=True,
+        check=True,
+    )
+    click.echo.assert_has_calls(
+        [
+            mock.call.echo(
+                "An error occurred while testing the crew: Command '['poetry', 'run', 'test', '5', 'gpt-4o']' returned non-zero exit status 1.",
+                err=True,
+            ),
+            mock.call.echo("Error", err=True),
+        ]
+    )
+
+
+@mock.patch("crewai.cli.test_crew.click")
+@mock.patch("crewai.cli.test_crew.subprocess.run")
+def test_test_crew_unexpected_exception(mock_subprocess_run, click):
+    # Arrange
+    n_iterations = 5
+    mock_subprocess_run.side_effect = Exception("Unexpected error")
+    test_crew.test_crew(n_iterations, "gpt-4o")
+
+    mock_subprocess_run.assert_called_once_with(
+        ["poetry", "run", "test", "5", "gpt-4o"],
+        capture_output=False,
+        text=True,
+        check=True,
+    )
+    click.echo.assert_called_once_with(
+        "An unexpected error occurred: Unexpected error", err=True
+    )

tests/crew_test.py

@@ -1917,13 +1917,13 @@ def test_replay_feature():
         )
 
         crew.kickoff()
-        crew.replay_from_task(str(write.id))
+        crew.replay(str(write.id))
         # Ensure context was passed correctly
         assert mock_execute_task.call_count == 3
 
 
 @pytest.mark.vcr(filter_headers=["authorization"])
-def test_crew_replay_from_task_error():
+def test_crew_replay_error():
     task = Task(
         description="Come up with a list of 5 interesting ideas to explore for an article",
         expected_output="5 bullet points with a paragraph for each idea.",
@@ -1936,7 +1936,7 @@ def test_crew_replay_from_task_error():
     )
 
     with pytest.raises(TypeError) as e:
-        crew.replay_from_task()  # type: ignore purposefully throwing err
+        crew.replay()  # type: ignore purposefully throwing err
         assert "task_id is required" in str(e)
 
 
@@ -2071,14 +2071,14 @@ def test_replay_task_with_context():
         with patch.object(Task, "execute_sync") as mock_replay_task:
             mock_replay_task.return_value = mock_task_output4
 
-            replayed_output = crew.replay_from_task(str(task4.id))
+            replayed_output = crew.replay(str(task4.id))
             assert replayed_output.raw == "Presentation on AI advancements..."
 
         db_handler.reset()
 
 
 @pytest.mark.vcr(filter_headers=["authorization"])
-def test_replay_from_task_with_context():
+def test_replay_with_context():
     agent = Agent(role="test_agent", backstory="Test Description", goal="Test Goal")
     task1 = Task(
         description="Context Task", expected_output="Say Task Output", agent=agent
@@ -2130,7 +2130,7 @@ def test_replay_from_task_with_context():
             },
         ],
     ):
-        crew.replay_from_task(str(task2.id))
+        crew.replay(str(task2.id))
 
         assert crew.tasks[1].context[0].output.raw == "context raw output"
 
@@ -2192,7 +2192,7 @@ def test_replay_with_invalid_task_id():
             ValueError,
             match="Task with id bf5b09c9-69bd-4eb8-be12-f9e5bae31c2d not found in the crew's tasks.",
         ):
-            crew.replay_from_task("bf5b09c9-69bd-4eb8-be12-f9e5bae31c2d")
+            crew.replay("bf5b09c9-69bd-4eb8-be12-f9e5bae31c2d")
 
 
 @pytest.mark.vcr(filter_headers=["authorization"])
@@ -2251,13 +2251,13 @@ def test_replay_interpolates_inputs_properly(mock_interpolate_inputs):
             },
         ],
     ):
-        crew.replay_from_task(str(task2.id))
+        crew.replay(str(task2.id))
         assert crew._inputs == {"name": "John"}
         assert mock_interpolate_inputs.call_count == 2
 
 
 @pytest.mark.vcr(filter_headers=["authorization"])
-def test_replay_from_task_setup_context():
+def test_replay_setup_context():
     agent = Agent(role="test_agent", backstory="Test Description", goal="Test Goal")
     task1 = Task(description="Context Task", expected_output="Say {name}", agent=agent)
     task2 = Task(
@@ -2306,7 +2306,7 @@ def test_replay_from_task_setup_context():
             },
         ],
     ):
-        crew.replay_from_task(str(task2.id))
+        crew.replay(str(task2.id))
 
         # Check if the first task's output was set correctly
         assert crew.tasks[0].output is not None

tests/utilities/test_planning_handler.py

@@ -1,4 +1,5 @@
 from unittest.mock import patch
+from crewai.tasks.task_output import TaskOutput
 
 import pytest
 
@@ -31,10 +32,15 @@ class TestCrewPlanner:
 
     def test_handle_crew_planning(self, crew_planner):
         with patch.object(Task, "execute_sync") as execute:
-            execute.return_value = PlannerTaskPydanticOutput(
-                list_of_plans_per_task=["Plan 1", "Plan 2", "Plan 3"]
+            execute.return_value = TaskOutput(
+                description="Description",
+                agent="agent",
+                pydantic=PlannerTaskPydanticOutput(
+                    list_of_plans_per_task=["Plan 1", "Plan 2", "Plan 3"]
+                ),
             )
             result = crew_planner._handle_crew_planning()
+
             assert isinstance(result, PlannerTaskPydanticOutput)
             assert len(result.list_of_plans_per_task) == len(crew_planner.tasks)
             execute.assert_called_once()