chore: address code review feedback

- Fix Crew import warning with TYPE_CHECKING
- Add performance monitoring with @track_agent
- Enhance docstrings and type hints
- Improve test organization

Co-Authored-By: Joe Moura <joao@crewai.com>

README.md

@@ -4,7 +4,7 @@
 
 # **CrewAI**
 
-🤖 **CrewAI**: Production-grade framework for orchestrating sophisticated AI agent systems. From simple automations to complex real-world applications, CrewAI provides precise control and deep customization. By fostering collaborative intelligence through flexible, production-ready architecture, CrewAI empowers agents to work together seamlessly, tackling complex business challenges with predictable, consistent results.
+🤖 **CrewAI**: Cutting-edge framework for orchestrating role-playing, autonomous AI agents. By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, tackling complex tasks.
 
 <h3>
 
@@ -22,17 +22,13 @@
 - [Why CrewAI?](#why-crewai)
 - [Getting Started](#getting-started)
 - [Key Features](#key-features)
-- [Understanding Flows and Crews](#understanding-flows-and-crews)
-- [CrewAI vs LangGraph](#how-crewai-compares)
 - [Examples](#examples)
   - [Quick Tutorial](#quick-tutorial)
   - [Write Job Descriptions](#write-job-descriptions)
   - [Trip Planner](#trip-planner)
   - [Stock Analysis](#stock-analysis)
-  - [Using Crews and Flows Together](#using-crews-and-flows-together)
 - [Connecting Your Crew to a Model](#connecting-your-crew-to-a-model)
 - [How CrewAI Compares](#how-crewai-compares)
-- [Frequently Asked Questions (FAQ)](#frequently-asked-questions-faq)
 - [Contribution](#contribution)
 - [Telemetry](#telemetry)
 - [License](#license)
@@ -40,40 +36,10 @@
 ## Why CrewAI?
 
 The power of AI collaboration has too much to offer.
-CrewAI is a standalone framework, built from the ground up without dependencies on Langchain or other agent frameworks. It's designed to enable AI agents to assume roles, share goals, and operate in a cohesive unit - much like a well-oiled crew. Whether you're building a smart assistant platform, an automated customer service ensemble, or a multi-agent research team, CrewAI provides the backbone for sophisticated multi-agent interactions.
+CrewAI is designed to enable AI agents to assume roles, share goals, and operate in a cohesive unit - much like a well-oiled crew. Whether you're building a smart assistant platform, an automated customer service ensemble, or a multi-agent research team, CrewAI provides the backbone for sophisticated multi-agent interactions.
 
 ## Getting Started
 
-### Learning Resources
-
-Learn CrewAI through our comprehensive courses:
-- [Multi AI Agent Systems with CrewAI](https://www.deeplearning.ai/short-courses/multi-ai-agent-systems-with-crewai/) - Master the fundamentals of multi-agent systems
-- [Practical Multi AI Agents and Advanced Use Cases](https://www.deeplearning.ai/short-courses/practical-multi-ai-agents-and-advanced-use-cases-with-crewai/) - Deep dive into advanced implementations
-
-### Understanding Flows and Crews
-
-CrewAI offers two powerful, complementary approaches that work seamlessly together to build sophisticated AI applications:
-
-1. **Crews**: Teams of AI agents with true autonomy and agency, working together to accomplish complex tasks through role-based collaboration. Crews enable:
-   - Natural, autonomous decision-making between agents
-   - Dynamic task delegation and collaboration
-   - Specialized roles with defined goals and expertise
-   - Flexible problem-solving approaches
-
-2. **Flows**: Production-ready, event-driven workflows that deliver precise control over complex automations. Flows provide:
-   - Fine-grained control over execution paths for real-world scenarios
-   - Secure, consistent state management between tasks
-   - Clean integration of AI agents with production Python code
-   - Conditional branching for complex business logic
-
-The true power of CrewAI emerges when combining Crews and Flows. This synergy allows you to:
-- Build complex, production-grade applications
-- Balance autonomy with precise control
-- Handle sophisticated real-world scenarios
-- Maintain clean, maintainable code structure
-
-### Getting Started with Installation
-
 To get started with CrewAI, follow these simple steps:
 
 ### 1. Installation
@@ -298,16 +264,13 @@ In addition to the sequential process, you can use the hierarchical process, whi
 
 ## Key Features
 
-**Note**: CrewAI is a standalone framework built from the ground up, without dependencies on Langchain or other agent frameworks.
-
-- **Deep Customization**: Build sophisticated agents with full control over the system - from overriding inner prompts to accessing low-level APIs. Customize roles, goals, tools, and behaviors while maintaining clean abstractions.
-- **Autonomous Inter-Agent Delegation**: Agents can autonomously delegate tasks and inquire amongst themselves, enabling complex problem-solving in real-world scenarios.
-- **Flexible Task Management**: Define and customize tasks with granular control, from simple operations to complex multi-step processes.
-- **Production-Grade Architecture**: Support for both high-level abstractions and low-level customization, with robust error handling and state management.
-- **Predictable Results**: Ensure consistent, accurate outputs through programmatic guardrails, agent training capabilities, and flow-based execution control. See our [documentation on guardrails](https://docs.crewai.com/how-to/guardrails/) for implementation details.
-- **Model Flexibility**: Run your crew using OpenAI or open source models with production-ready integrations. See [Connect CrewAI to LLMs](https://docs.crewai.com/how-to/LLM-Connections/) for detailed configuration options.
-- **Event-Driven Flows**: Build complex, real-world workflows with precise control over execution paths, state management, and conditional logic.
-- **Process Orchestration**: Achieve any workflow pattern through flows - from simple sequential and hierarchical processes to complex, custom orchestration patterns with conditional branching and parallel execution.
+- **Role-Based Agent Design**: Customize agents with specific roles, goals, and tools.
+- **Autonomous Inter-Agent Delegation**: Agents can autonomously delegate tasks and inquire amongst themselves, enhancing problem-solving efficiency.
+- **Flexible Task Management**: Define tasks with customizable tools and assign them to agents dynamically.
+- **Processes Driven**: Currently only supports `sequential` task execution and `hierarchical` processes, but more complex processes like consensual and autonomous are being worked on.
+- **Save output as file**: Save the output of individual tasks as a file, so you can use it later.
+- **Parse output as Pydantic or Json**: Parse the output of individual tasks as a Pydantic model or as a Json if you want to.
+- **Works with Open Source Models**: Run your crew using Open AI or open source models refer to the [Connect CrewAI to LLMs](https://docs.crewai.com/how-to/LLM-Connections/) page for details on configuring your agents' connections to models, even ones running locally!
 
 ![CrewAI Mind Map](./docs/crewAI-mindmap.png "CrewAI Mind Map")
 
@@ -342,98 +305,6 @@ You can test different real life examples of AI crews in the [CrewAI-examples re
 
 [![Stock Analysis](https://img.youtube.com/vi/e0Uj4yWdaAg/maxresdefault.jpg)](https://www.youtube.com/watch?v=e0Uj4yWdaAg "Stock Analysis")
 
-### Using Crews and Flows Together
-
-CrewAI's power truly shines when combining Crews with Flows to create sophisticated automation pipelines. Here's how you can orchestrate multiple Crews within a Flow:
-
-```python
-from crewai.flow.flow import Flow, listen, start, router
-from crewai import Crew, Agent, Task
-from pydantic import BaseModel
-
-# Define structured state for precise control
-class MarketState(BaseModel):
-    sentiment: str = "neutral"
-    confidence: float = 0.0
-    recommendations: list = []
-
-class AdvancedAnalysisFlow(Flow[MarketState]):
-    @start()
-    def fetch_market_data(self):
-        # Demonstrate low-level control with structured state
-        self.state.sentiment = "analyzing"
-        return {"sector": "tech", "timeframe": "1W"}  # These parameters match the task description template
-
-    @listen(fetch_market_data)
-    def analyze_with_crew(self, market_data):
-        # Show crew agency through specialized roles
-        analyst = Agent(
-            role="Senior Market Analyst",
-            goal="Conduct deep market analysis with expert insight",
-            backstory="You're a veteran analyst known for identifying subtle market patterns"
-        )
-        researcher = Agent(
-            role="Data Researcher",
-            goal="Gather and validate supporting market data",
-            backstory="You excel at finding and correlating multiple data sources"
-        )
-        
-        analysis_task = Task(
-            description="Analyze {sector} sector data for the past {timeframe}",
-            expected_output="Detailed market analysis with confidence score",
-            agent=analyst
-        )
-        research_task = Task(
-            description="Find supporting data to validate the analysis",
-            expected_output="Corroborating evidence and potential contradictions",
-            agent=researcher
-        )
-        
-        # Demonstrate crew autonomy
-        analysis_crew = Crew(
-            agents=[analyst, researcher],
-            tasks=[analysis_task, research_task],
-            process=Process.sequential,
-            verbose=True
-        )
-        return analysis_crew.kickoff(inputs=market_data)  # Pass market_data as named inputs
-
-    @router(analyze_with_crew)
-    def determine_next_steps(self):
-        # Show flow control with conditional routing
-        if self.state.confidence > 0.8:
-            return "high_confidence"
-        elif self.state.confidence > 0.5:
-            return "medium_confidence"
-        return "low_confidence"
-
-    @listen("high_confidence")
-    def execute_strategy(self):
-        # Demonstrate complex decision making
-        strategy_crew = Crew(
-            agents=[
-                Agent(role="Strategy Expert",
-                      goal="Develop optimal market strategy")
-            ],
-            tasks=[
-                Task(description="Create detailed strategy based on analysis",
-                     expected_output="Step-by-step action plan")
-            ]
-        )
-        return strategy_crew.kickoff()
-
-    @listen("medium_confidence", "low_confidence")
-    def request_additional_analysis(self):
-        self.state.recommendations.append("Gather more data")
-        return "Additional analysis required"
-```
-
-This example demonstrates how to:
-1. Use Python code for basic data operations
-2. Create and execute Crews as steps in your workflow
-3. Use Flow decorators to manage the sequence of operations
-4. Implement conditional branching based on Crew results
-
 ## Connecting Your Crew to a Model
 
 CrewAI supports using various LLMs through a variety of connection options. By default your agents will use the OpenAI API when querying the model. However, there are several other ways to allow your agents to connect to models. For example, you can configure your agents to use a local model via the Ollama tool.
@@ -442,13 +313,9 @@ Please refer to the [Connect CrewAI to LLMs](https://docs.crewai.com/how-to/LLM-
 
 ## How CrewAI Compares
 
-**CrewAI's Advantage**: CrewAI combines autonomous agent intelligence with precise workflow control through its unique Crews and Flows architecture. The framework excels at both high-level orchestration and low-level customization, enabling complex, production-grade systems with granular control.
+**CrewAI's Advantage**: CrewAI is built with production in mind. It offers the flexibility of Autogen's conversational agents and the structured process approach of ChatDev, but without the rigidity. CrewAI's processes are designed to be dynamic and adaptable, fitting seamlessly into both development and production workflows.
 
-- **LangGraph**: While LangGraph provides a foundation for building agent workflows, its approach requires significant boilerplate code and complex state management patterns. The framework's tight coupling with LangChain can limit flexibility when implementing custom agent behaviors or integrating with external systems.
-
-*P.S. CrewAI demonstrates significant performance advantages over LangGraph, executing 5.76x faster in certain cases like this QA task example ([see comparison](https://github.com/crewAIInc/crewAI-examples/tree/main/Notebooks/CrewAI%20Flows%20%26%20Langgraph/QA%20Agent)) while achieving higher evaluation scores with faster completion times in certain coding tasks, like in this example ([detailed analysis](https://github.com/crewAIInc/crewAI-examples/blob/main/Notebooks/CrewAI%20Flows%20%26%20Langgraph/Coding%20Assistant/coding_assistant_eval.ipynb)).*
-
-- **Autogen**: While Autogen excels at creating conversational agents capable of working together, it lacks an inherent concept of process. In Autogen, orchestrating agents' interactions requires additional programming, which can become complex and cumbersome as the scale of tasks grows.
+- **Autogen**: While Autogen does good in creating conversational agents capable of working together, it lacks an inherent concept of process. In Autogen, orchestrating agents' interactions requires additional programming, which can become complex and cumbersome as the scale of tasks grows.
 
 - **ChatDev**: ChatDev introduced the idea of processes into the realm of AI agents, but its implementation is quite rigid. Customizations in ChatDev are limited and not geared towards production environments, which can hinder scalability and flexibility in real-world applications.
 
@@ -573,8 +440,5 @@ A: CrewAI uses anonymous telemetry to collect usage data for improvement purpose
 ### Q: Where can I find examples of CrewAI in action?
 A: You can find various real-life examples in the [CrewAI-examples repository](https://github.com/crewAIInc/crewAI-examples), including trip planners, stock analysis tools, and more.
 
-### Q: What is the difference between Crews and Flows?
-A: Crews and Flows serve different but complementary purposes in CrewAI. Crews are teams of AI agents working together to accomplish specific tasks through role-based collaboration, delivering accurate and predictable results. Flows, on the other hand, are event-driven workflows that can orchestrate both Crews and regular Python code, allowing you to build complex automation pipelines with secure state management and conditional execution paths.
-
 ### Q: How can I contribute to CrewAI?
 A: Contributions are welcome! You can fork the repository, create a new branch for your feature, add your improvement, and send a pull request. Check the Contribution section in the README for more details.

docs/concepts/knowledge.mdx

@@ -171,58 +171,6 @@ crewai reset-memories --knowledge
 
 This is useful when you've updated your knowledge sources and want to ensure that the agents are using the most recent information.
 
-## Agent-Specific Knowledge
-
-While knowledge can be provided at the crew level using `crew.knowledge_sources`, individual agents can also have their own knowledge sources using the `knowledge_sources` parameter:
-
-```python Code
-from crewai import Agent, Task, Crew
-from crewai.knowledge.source.string_knowledge_source import StringKnowledgeSource
-
-# Create agent-specific knowledge about a product
-product_specs = StringKnowledgeSource(
-    content="""The XPS 13 laptop features:
-    - 13.4-inch 4K display
-    - Intel Core i7 processor
-    - 16GB RAM
-    - 512GB SSD storage
-    - 12-hour battery life""",
-    metadata={"category": "product_specs"}
-)
-
-# Create a support agent with product knowledge
-support_agent = Agent(
-    role="Technical Support Specialist",
-    goal="Provide accurate product information and support.",
-    backstory="You are an expert on our laptop products and specifications.",
-    knowledge_sources=[product_specs]  # Agent-specific knowledge
-)
-
-# Create a task that requires product knowledge
-support_task = Task(
-    description="Answer this customer question: {question}",
-    agent=support_agent
-)
-
-# Create and run the crew
-crew = Crew(
-    agents=[support_agent],
-    tasks=[support_task]
-)
-
-# Get answer about the laptop's specifications
-result = crew.kickoff(
-    inputs={"question": "What is the storage capacity of the XPS 13?"}
-)
-```
-
-<Info>
-  Benefits of agent-specific knowledge:
-  - Give agents specialized information for their roles
-  - Maintain separation of concerns between agents
-  - Combine with crew-level knowledge for layered information access
-</Info>
-
 ## Custom Knowledge Sources
 
 CrewAI allows you to create custom knowledge sources for any type of data by extending the `BaseKnowledgeSource` class. Let's create a practical example that fetches and processes space news articles.

src/crewai/agents/agent_builder/base_agent_executor_mixin.py

@@ -26,7 +26,7 @@ class CrewAgentExecutorMixin:
 
     def _should_force_answer(self) -> bool:
         """Determine if a forced answer is required based on iteration count."""
-        return self.iterations >= self.max_iter
+        return (self.iterations >= self.max_iter) and not self.have_forced_answer
 
     def _create_short_term_memory(self, output) -> None:
         """Create and save a short-term memory item if conditions are met."""

src/crewai/crew.py

@@ -4,8 +4,11 @@ import uuid
 import warnings
 from concurrent.futures import Future
 from hashlib import md5
+import warnings
 from typing import Any, Callable, Dict, List, Optional, Tuple, Union
 
+from crewai.llm import LLM
+
 from pydantic import (
     UUID4,
     BaseModel,
@@ -1076,18 +1079,35 @@ class Crew(BaseModel):
         self,
         n_iterations: int,
         openai_model_name: Optional[str] = None,
+        llm: Optional[Union[str, LLM]] = None,
         inputs: Optional[Dict[str, Any]] = None,
     ) -> None:
-        """Test and evaluate the Crew with the given inputs for n iterations concurrently using concurrent.futures."""
+        """Test and evaluate the Crew with the given inputs for n iterations concurrently using concurrent.futures.
+        
+        Args:
+            n_iterations: Number of test iterations to run
+            openai_model_name: (Deprecated) OpenAI model name to use for evaluation. Will be ignored if llm is provided.
+            llm: Language model to use for evaluation, can be a string (model name) or LLM instance.
+                 Takes precedence over openai_model_name if both are provided.
+            inputs: Optional dictionary of inputs to pass to the crew
+
+        Raises:
+            ValueError: If neither openai_model_name nor llm is provided
+        """
+        if openai_model_name and llm:
+            warnings.warn("Both openai_model_name and llm provided. Using llm parameter.")
+        elif not (openai_model_name or llm):
+            raise ValueError("Either openai_model_name or llm must be provided")
+
         test_crew = self.copy()
 
         self._test_execution_span = test_crew._telemetry.test_execution_span(
             test_crew,
             n_iterations,
             inputs,
-            openai_model_name,  # type: ignore[arg-type]
-        )  # type: ignore[arg-type]
-        evaluator = CrewEvaluator(test_crew, openai_model_name)  # type: ignore[arg-type]
+            str(llm) if llm else openai_model_name,
+        )
+        evaluator = CrewEvaluator(test_crew, llm or openai_model_name)
 
         for i in range(1, n_iterations + 1):
             evaluator.set_iteration(i)

src/crewai/knowledge/knowledge.py

@@ -14,13 +14,13 @@ class Knowledge(BaseModel):
     Knowledge is a collection of sources and setup for the vector store to save and query relevant context.
     Args:
         sources: List[BaseKnowledgeSource] = Field(default_factory=list)
-        storage: Optional[KnowledgeStorage] = Field(default=None)
+        storage: KnowledgeStorage = Field(default_factory=KnowledgeStorage)
         embedder_config: Optional[Dict[str, Any]] = None
     """
 
     sources: List[BaseKnowledgeSource] = Field(default_factory=list)
     model_config = ConfigDict(arbitrary_types_allowed=True)
-    storage: Optional[KnowledgeStorage] = Field(default=None)
+    storage: KnowledgeStorage = Field(default_factory=KnowledgeStorage)
     embedder_config: Optional[Dict[str, Any]] = None
     collection_name: Optional[str] = None
 
@@ -49,13 +49,8 @@ class Knowledge(BaseModel):
         """
         Query across all knowledge sources to find the most relevant information.
         Returns the top_k most relevant chunks.
-        
-        Raises:
-            ValueError: If storage is not initialized.
         """
-        if self.storage is None:
-            raise ValueError("Storage is not initialized.")
-            
+
         results = self.storage.search(
             query,
             limit,

src/crewai/knowledge/source/base_file_knowledge_source.py

@@ -22,14 +22,13 @@ class BaseFileKnowledgeSource(BaseKnowledgeSource, ABC):
         default_factory=list, description="The path to the file"
     )
     content: Dict[Path, str] = Field(init=False, default_factory=dict)
-    storage: Optional[KnowledgeStorage] = Field(default=None)
+    storage: KnowledgeStorage = Field(default_factory=KnowledgeStorage)
     safe_file_paths: List[Path] = Field(default_factory=list)
 
     @field_validator("file_path", "file_paths", mode="before")
-    def validate_file_path(cls, v, info):
+    def validate_file_path(cls, v, values):
         """Validate that at least one of file_path or file_paths is provided."""
-        # Single check if both are None, O(1) instead of nested conditions
-        if v is None and info.data.get("file_path" if info.field_name == "file_paths" else "file_paths") is None:
+        if v is None and ("file_path" not in values or values.get("file_path") is None):
             raise ValueError("Either file_path or file_paths must be provided")
         return v
 
@@ -63,10 +62,7 @@ class BaseFileKnowledgeSource(BaseKnowledgeSource, ABC):
 
     def _save_documents(self):
         """Save the documents to the storage."""
-        if self.storage:
-            self.storage.save(self.chunks)
-        else:
-            raise ValueError("No storage found to save documents.")
+        self.storage.save(self.chunks)
 
     def convert_to_path(self, path: Union[Path, str]) -> Path:
         """Convert a path to a Path object."""

src/crewai/knowledge/source/base_knowledge_source.py

@@ -16,7 +16,7 @@ class BaseKnowledgeSource(BaseModel, ABC):
     chunk_embeddings: List[np.ndarray] = Field(default_factory=list)
 
     model_config = ConfigDict(arbitrary_types_allowed=True)
-    storage: Optional[KnowledgeStorage] = Field(default=None)
+    storage: KnowledgeStorage = Field(default_factory=KnowledgeStorage)
     metadata: Dict[str, Any] = Field(default_factory=dict)  # Currently unused
     collection_name: Optional[str] = Field(default=None)
 
@@ -46,7 +46,4 @@ class BaseKnowledgeSource(BaseModel, ABC):
         Save the documents to the storage.
         This method should be called after the chunks and embeddings are generated.
         """
-        if self.storage:
-            self.storage.save(self.chunks)
-        else:
-            raise ValueError("No storage found to save documents.")
+        self.storage.save(self.chunks)

src/crewai/utilities/evaluators/crew_evaluator_handler.py

@@ -1,15 +1,28 @@
 from collections import defaultdict
+from typing import TYPE_CHECKING, Union
 
 from pydantic import BaseModel, Field
 from rich.box import HEAVY_EDGE
 from rich.console import Console
 from rich.table import Table
 
+try:
+    from agentops import track_agent
+except ImportError:
+    def track_agent():
+        def noop(f):
+            return f
+        return noop
+
 from crewai.agent import Agent
+from crewai.llm import LLM
 from crewai.task import Task
 from crewai.tasks.task_output import TaskOutput
 from crewai.telemetry import Telemetry
 
+if TYPE_CHECKING:
+    from crewai.crew import Crew
+
 
 class TaskEvaluationPydanticOutput(BaseModel):
     quality: float = Field(
@@ -17,13 +30,15 @@ class TaskEvaluationPydanticOutput(BaseModel):
     )
 
 
+@track_agent()
 class CrewEvaluator:
-    """
-    A class to evaluate the performance of the agents in the crew based on the tasks they have performed.
+    """Evaluates the performance of a crew's agents on their tasks.
+
+    Handles evaluation of agent performance using specified LLM model.
 
     Attributes:
         crew (Crew): The crew of agents to evaluate.
-        openai_model_name (str): The model to use for evaluating the performance of the agents (for now ONLY OpenAI accepted).
+        llm (Union[str, LLM]): The language model to use for evaluation. Can be a string (model name) or LLM instance.
         tasks_scores (defaultdict): A dictionary to store the scores of the agents for each task.
         iteration (int): The current iteration of the evaluation.
     """
@@ -32,18 +47,29 @@ class CrewEvaluator:
     run_execution_times: defaultdict = defaultdict(list)
     iteration: int = 0
 
-    def __init__(self, crew, openai_model_name: str):
+    def __init__(self, crew: "Crew", llm: Union[str, LLM]) -> None:
+        """Initialize CrewEvaluator with crew and language model.
+        
+        Args:
+            crew: The crew to evaluate
+            llm: Language model to use for evaluation, can be a string (model name) or LLM instance
+        """
         self.crew = crew
-        self.openai_model_name = openai_model_name
+        self.llm = llm if isinstance(llm, LLM) else LLM(model=llm)
         self._telemetry = Telemetry()
         self._setup_for_evaluating()
 
     def _setup_for_evaluating(self) -> None:
-        """Sets up the crew for evaluating."""
+        """Sets up the crew for evaluating by assigning evaluation callbacks to tasks."""
         for task in self.crew.tasks:
             task.callback = self.evaluate
 
-    def _evaluator_agent(self):
+    def _evaluator_agent(self) -> Agent:
+        """Creates an agent specialized in evaluating task performance.
+
+        Returns:
+            Agent: An agent configured to evaluate task execution quality.
+        """
         return Agent(
             role="Task Execution Evaluator",
             goal=(
@@ -51,12 +77,22 @@ class CrewEvaluator:
             ),
             backstory="Evaluator agent for crew evaluation with precise capabilities to evaluate the performance of the agents in the crew based on the tasks they have performed",
             verbose=False,
-            llm=self.openai_model_name,
+            llm=self.llm,
         )
 
     def _evaluation_task(
         self, evaluator_agent: Agent, task_to_evaluate: Task, task_output: str
     ) -> Task:
+        """Creates a task for evaluating another task's execution.
+
+        Args:
+            evaluator_agent: The agent that will perform the evaluation
+            task_to_evaluate: The task whose execution needs to be evaluated
+            task_output: The output produced by the task execution
+
+        Returns:
+            Task: A task configured to evaluate the execution quality
+        """
         return Task(
             description=(
                 "Based on the task description and the expected output, compare and evaluate the performance of the agents in the crew based on the Task Output they have performed using score from 1 to 10 evaluating on completion, quality, and overall performance."
@@ -72,25 +108,28 @@ class CrewEvaluator:
         )
 
     def set_iteration(self, iteration: int) -> None:
+        """Sets the current iteration number for test tracking.
+
+        Args:
+            iteration: The iteration number to set
+        """
         self.iteration = iteration
 
     def print_crew_evaluation_result(self) -> None:
-        """
-        Prints the evaluation result of the crew in a table.
-        A Crew with 2 tasks using the command crewai test -n 3
-        will output the following table:
+        """Prints a formatted table showing evaluation results for all tasks and iterations.
 
+        Displays task scores (1-10), average scores, execution times, and involved agents
+        in a rich-formatted table. Each row represents a task or crew-level metric,
+        with columns for each test iteration and averages.
+
+        Example output:
                         Tasks Scores
                     (1-10 Higher is better)
         ┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
         ┃ Tasks/Crew/Agents  ┃ Run 1 ┃ Run 2 ┃ Run 3 ┃ Avg. Total ┃ Agents                       ┃
         ┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
         │ Task 1             │ 9.0   │ 10.0  │ 9.0   │ 9.3        │ - AI LLMs Senior Researcher  │
-        │                    │       │       │       │            │ - AI LLMs Reporting Analyst  │
-        │                    │       │       │       │            │                              │
         │ Task 2             │ 9.0   │ 9.0   │ 9.0   │ 9.0        │ - AI LLMs Senior Researcher  │
-        │                    │       │       │       │            │ - AI LLMs Reporting Analyst  │
-        │                    │       │       │       │            │                              │
         │ Crew               │ 9.0   │ 9.5   │ 9.0   │ 9.2        │                              │
         │ Execution Time (s) │ 42    │ 79    │ 52    │ 57         │                              │
         └────────────────────┴───────┴───────┴───────┴────────────┴──────────────────────────────┘
@@ -156,8 +195,18 @@ class CrewEvaluator:
         console = Console()
         console.print(table)
 
-    def evaluate(self, task_output: TaskOutput):
-        """Evaluates the performance of the agents in the crew based on the tasks they have performed."""
+    def evaluate(self, task_output: TaskOutput) -> None:
+        """Evaluates the performance of the agents in the crew based on task execution.
+
+        Evaluates task execution quality using a specialized evaluator agent and
+        stores the evaluation results for later analysis.
+
+        Args:
+            task_output: The output from the task execution to evaluate
+
+        Raises:
+            ValueError: If task_output is missing or doesn't match any known task
+        """
         current_task = None
         for task in self.crew.tasks:
             if task.description == task_output.description:
@@ -181,7 +230,7 @@ class CrewEvaluator:
                 self.crew,
                 evaluation_result.pydantic.quality,
                 current_task._execution_time,
-                self.openai_model_name,
+                str(self.llm),
             )
             self.tasks_scores[self.iteration].append(evaluation_result.pydantic.quality)
             self.run_execution_times[self.iteration].append(

tests/crew_test.py

@@ -14,6 +14,7 @@ from crewai.agent import Agent
 from crewai.agents.cache import CacheHandler
 from crewai.crew import Crew
 from crewai.crews.crew_output import CrewOutput
+from crewai.llm import LLM
 from crewai.memory.contextual.contextual_memory import ContextualMemory
 from crewai.process import Process
 from crewai.task import Task
@@ -1123,7 +1124,7 @@ def test_kickoff_for_each_empty_input():
     assert results == []
 
 
-@pytest.mark.vcr(filter_headers=["authorization"])
+@pytest.mark.vcr(filter_headeruvs=["authorization"])
 def test_kickoff_for_each_invalid_input():
     """Tests if kickoff_for_each raises TypeError for invalid input types."""
 
@@ -2814,8 +2815,8 @@ def test_conditional_should_execute():
 @mock.patch("crewai.crew.Crew.kickoff")
 def test_crew_testing_function(kickoff_mock, copy_mock, crew_evaluator):
     task = Task(
-        description="Come up with a list of 5 interesting ideas to explore for an article, then write one amazing paragraph highlight for each idea that showcases how good an article about this topic could be. Return the list of ideas with their paragraph and your notes.",
-        expected_output="5 bullet points with a paragraph for each idea.",
+        description="Test task",
+        expected_output="Expected output",
         agent=researcher,
     )
 
@@ -2844,6 +2845,77 @@ def test_crew_testing_function(kickoff_mock, copy_mock, crew_evaluator):
         ]
     )
 
+class TestCrewTesting:
+    """Tests for Crew.test() functionality."""
+
+    @pytest.fixture
+    def task(self):
+        return Task(
+            description="Test task",
+            expected_output="Expected output",
+            agent=researcher,
+        )
+
+    @pytest.fixture
+    def crew(self, task):
+        return Crew(agents=[researcher], tasks=[task])
+
+    @pytest.mark.parametrize("llm_input", [
+        "gpt-4o-mini",
+        LLM(model="gpt-4o-mini"),
+    ])
+    @mock.patch("crewai.crew.CrewEvaluator")
+    @mock.patch("crewai.crew.Crew.copy")
+    @mock.patch("crewai.crew.Crew.kickoff")
+    def test_crew_test_with_different_llms(self, kickoff_mock, copy_mock, crew_evaluator, crew, llm_input):
+        """Test Crew.test() with different LLM inputs."""
+        copy_mock.return_value = crew
+        crew.test(n_iterations=2, llm=llm_input, inputs={"topic": "AI"})
+
+        kickoff_mock.assert_has_calls([
+            mock.call(inputs={"topic": "AI"}),
+            mock.call(inputs={"topic": "AI"})
+        ])
+
+        crew_evaluator.assert_has_calls([
+            mock.call(crew, llm_input),
+            mock.call().set_iteration(1),
+            mock.call().set_iteration(2),
+            mock.call().print_crew_evaluation_result(),
+        ])
+
+    @mock.patch("crewai.crew.CrewEvaluator")
+    @mock.patch("crewai.crew.Crew.copy")
+    @mock.patch("crewai.crew.Crew.kickoff")
+    def test_crew_test_with_both_llm_and_model_name(self, kickoff_mock, copy_mock, crew_evaluator, crew):
+        """Test that llm parameter takes precedence over openai_model_name."""
+        custom_llm = LLM(model="gpt-4o-mini")
+        copy_mock.return_value = crew
+
+        with pytest.warns(UserWarning, match="Both openai_model_name and llm provided. Using llm parameter."):
+            crew.test(n_iterations=2, llm=custom_llm, openai_model_name="gpt-4", inputs={"topic": "AI"})
+
+        kickoff_mock.assert_has_calls([
+            mock.call(inputs={"topic": "AI"}),
+            mock.call(inputs={"topic": "AI"})
+        ])
+
+        crew_evaluator.assert_has_calls([
+            mock.call(crew, custom_llm),
+            mock.call().set_iteration(1),
+            mock.call().set_iteration(2),
+            mock.call().print_crew_evaluation_result(),
+        ])
+
+    @mock.patch("crewai.crew.CrewEvaluator")
+    @mock.patch("crewai.crew.Crew.copy")
+    @mock.patch("crewai.crew.Crew.kickoff")
+    def test_crew_test_with_no_llm_raises_error(self, kickoff_mock, copy_mock, crew_evaluator, crew):
+        """Test that error is raised when no LLM is provided."""
+        copy_mock.return_value = crew
+        with pytest.raises(ValueError, match="Either openai_model_name or llm must be provided"):
+            crew.test(n_iterations=2, inputs={"topic": "AI"})
+
 
 @pytest.mark.vcr(filter_headers=["authorization"])
 def test_hierarchical_verbose_manager_agent():
@@ -3125,4 +3197,4 @@ def test_multimodal_agent_live_image_analysis():
     # Verify we got a meaningful response
     assert isinstance(result.raw, str)
     assert len(result.raw) > 100  # Expecting a detailed analysis
-    assert "error" not in result.raw.lower()  # No error messages in response
+    assert "error" not in result.raw.lower()  # No error messages in response

tests/knowledge/knowledge_test.py

@@ -584,28 +584,3 @@ def test_docling_source_with_local_file():
     docling_source = CrewDoclingSource(file_paths=[pdf_path])
     assert docling_source.file_paths == [pdf_path]
     assert docling_source.content is not None
-
-
-def test_file_path_validation():
-    """Test file path validation for knowledge sources."""
-    current_dir = Path(__file__).parent
-    pdf_path = current_dir / "crewai_quickstart.pdf"
-
-    # Test valid single file_path
-    source = PDFKnowledgeSource(file_path=pdf_path)
-    assert source.safe_file_paths == [pdf_path]
-
-    # Test valid file_paths list
-    source = PDFKnowledgeSource(file_paths=[pdf_path])
-    assert source.safe_file_paths == [pdf_path]
-
-    # Test both file_path and file_paths provided (should use file_paths)
-    source = PDFKnowledgeSource(file_path=pdf_path, file_paths=[pdf_path])
-    assert source.safe_file_paths == [pdf_path]
-
-    # Test neither file_path nor file_paths provided
-    with pytest.raises(
-        ValueError,
-        match="file_path/file_paths must be a Path, str, or a list of these types"
-    ):
-        PDFKnowledgeSource()

tests/utilities/evaluators/test_crew_evaluator_handler.py

@@ -4,6 +4,7 @@ import pytest
 
 from crewai.agent import Agent
 from crewai.crew import Crew
+from crewai.llm import LLM
 from crewai.task import Task
 from crewai.tasks.task_output import TaskOutput
 from crewai.utilities.evaluators.crew_evaluator_handler import (
@@ -23,7 +24,7 @@ class TestCrewEvaluator:
         )
         crew = Crew(agents=[agent], tasks=[task])
 
-        return CrewEvaluator(crew, openai_model_name="gpt-4o-mini")
+        return CrewEvaluator(crew, "gpt-4o-mini")
 
     def test_setup_for_evaluating(self, crew_planner):
         crew_planner._setup_for_evaluating()
@@ -140,3 +141,30 @@ class TestCrewEvaluator:
             execute().pydantic = TaskEvaluationPydanticOutput(quality=9.5)
             crew_planner.evaluate(task_output)
             assert crew_planner.tasks_scores[0] == [9.5]
+
+    def test_crew_evaluator_with_custom_llm(self):
+        agent = Agent(role="Agent 1", goal="Goal 1", backstory="Backstory 1")
+        task = Task(
+            description="Task 1",
+            expected_output="Output 1",
+            agent=agent,
+        )
+        crew = Crew(agents=[agent], tasks=[task])
+        custom_llm = LLM(model="gpt-4o-mini")
+
+        evaluator = CrewEvaluator(crew, custom_llm)
+        assert evaluator.llm == custom_llm
+
+    def test_crew_evaluator_with_model_name(self):
+        agent = Agent(role="Agent 1", goal="Goal 1", backstory="Backstory 1")
+        task = Task(
+            description="Task 1",
+            expected_output="Output 1",
+            agent=agent,
+        )
+        crew = Crew(agents=[agent], tasks=[task])
+        model_name = "gpt-4o-mini"
+
+        evaluator = CrewEvaluator(crew, model_name)
+        assert isinstance(evaluator.llm, LLM)
+        assert evaluator.llm.model == model_name