Fix lint: Remove unused import os from test_create_directory_true

- Removes F401 lint error: 'os' imported but unused
- All lint checks should now pass

Co-Authored-By: Jo\u00E3o <joao@crewai.com>

docs/en/concepts/crews.mdx

@@ -32,6 +32,7 @@ A crew in crewAI represents a collaborative group of agents working together to
 | **Prompt File** _(optional)_          | `prompt_file`          | Path to the prompt JSON file to be used for the crew.                                                                                                                                                                                                     |
 | **Planning** *(optional)*             | `planning`             | Adds planning ability to the Crew. When activated before each Crew iteration, all Crew data is sent to an AgentPlanner that will plan the tasks and this plan will be added to each task description.                                                     |
 | **Planning LLM** *(optional)*         | `planning_llm`         | The language model used by the AgentPlanner in a planning process.                                                                                                                                                                                        |
+| **Knowledge Sources** _(optional)_    | `knowledge_sources`    | Knowledge sources available at the crew level, accessible to all the agents.                                                                                                                                                                                    |
 
 <Tip>
 **Crew Max RPM**: The `max_rpm` attribute sets the maximum number of requests per minute the crew can perform to avoid rate limits and will override individual agents' `max_rpm` settings if you set it.

docs/en/concepts/tasks.mdx

@@ -57,6 +57,7 @@ crew = Crew(
 | **Output JSON** _(optional)_     | `output_json`     | `Optional[Type[BaseModel]]`   | A Pydantic model to structure the JSON output.                                                                       |
 | **Output Pydantic** _(optional)_ | `output_pydantic` | `Optional[Type[BaseModel]]`   | A Pydantic model for task output.                                                                                    |
 | **Callback** _(optional)_        | `callback`        | `Optional[Any]`               | Function/object to be executed after task completion.                                                                |
+| **Guardrail** _(optional)_       | `guardrail`       | `Optional[Union[Callable, str]]` | Function or string description to validate task output before proceeding to next task.                            |
 
 ## Creating Tasks
 
@@ -86,6 +87,7 @@ research_task:
   expected_output: >
     A list with 10 bullet points of the most relevant information about {topic}
   agent: researcher
+  guardrail: ensure each bullet contains a minimum of 100 words
 
 reporting_task:
   description: >
@@ -332,9 +334,13 @@ Task guardrails provide a way to validate and transform task outputs before they
 are passed to the next task. This feature helps ensure data quality and provides
 feedback to agents when their output doesn't meet specific criteria.
 
-### Using Task Guardrails
+**Guardrails can be defined in two ways:**
+1. **Function-based guardrails**: Python functions that implement custom validation logic
+2. **String-based guardrails**: Natural language descriptions that are automatically converted to LLM-powered validation
 
-To add a guardrail to a task, provide a validation function through the `guardrail` parameter:
+### Function-Based Guardrails
+
+To add a function-based guardrail to a task, provide a validation function through the `guardrail` parameter:
 
 ```python Code
 from typing import Tuple, Union, Dict, Any
@@ -372,9 +378,82 @@ blog_task = Task(
    - On success: it returns a tuple of `(bool, Any)`. For example: `(True, validated_result)`
    - On Failure: it returns a tuple of `(bool, str)`. For example: `(False, "Error message explain the failure")`
 
-### LLMGuardrail
+### String-Based Guardrails
 
-The `LLMGuardrail` class offers a robust mechanism for validating task outputs.
+String-based guardrails allow you to describe validation criteria in natural language. When you provide a string instead of a function, CrewAI automatically converts it to an `LLMGuardrail` that uses an AI agent to validate the task output.
+
+#### Using String Guardrails in Python
+
+```python Code
+from crewai import Task
+
+# Simple string-based guardrail
+blog_task = Task(
+    description="Write a blog post about AI",
+    expected_output="A blog post under 200 words",
+    agent=blog_agent,
+    guardrail="Ensure the blog post is under 200 words and includes practical examples"
+)
+
+# More complex validation criteria
+research_task = Task(
+    description="Research AI trends for 2025",
+    expected_output="A comprehensive research report",
+    agent=research_agent,
+    guardrail="Ensure each finding includes a credible source and is backed by recent data from 2024-2025"
+)
+```
+
+#### Using String Guardrails in YAML
+
+```yaml
+research_task:
+  description: Research the latest AI developments
+  expected_output: A list of 10 bullet points about AI
+  agent: researcher
+  guardrail: ensure each bullet contains a minimum of 100 words
+
+validation_task:
+  description: Validate the research findings
+  expected_output: A validation report
+  agent: validator
+  guardrail: confirm all sources are from reputable publications and published within the last 2 years
+```
+
+#### How String Guardrails Work
+
+When you provide a string guardrail, CrewAI automatically:
+1. Creates an `LLMGuardrail` instance using the string as validation criteria
+2. Uses the task's agent LLM to power the validation
+3. Creates a temporary validation agent that checks the output against your criteria
+4. Returns detailed feedback if validation fails
+
+This approach is ideal when you want to use natural language to describe validation rules without writing custom validation functions.
+
+### LLMGuardrail Class
+
+The `LLMGuardrail` class is the underlying mechanism that powers string-based guardrails. You can also use it directly for more advanced control:
+
+```python Code
+from crewai import Task
+from crewai.tasks.llm_guardrail import LLMGuardrail
+from crewai.llm import LLM
+
+# Create a custom LLMGuardrail with specific LLM
+custom_guardrail = LLMGuardrail(
+    description="Ensure the response contains exactly 5 bullet points with proper citations",
+    llm=LLM(model="gpt-4o-mini")
+)
+
+task = Task(
+    description="Research AI safety measures",
+    expected_output="A detailed analysis with bullet points",
+    agent=research_agent,
+    guardrail=custom_guardrail
+)
+```
+
+**Note**: When you use a string guardrail, CrewAI automatically creates an `LLMGuardrail` instance using your task's agent LLM. Using `LLMGuardrail` directly gives you more control over the validation process and LLM selection.
 
 ### Error Handling Best Practices
 
@@ -798,166 +877,7 @@ While creating and executing tasks, certain validation mechanisms are in place t
 
 These validations help in maintaining the consistency and reliability of task executions within the crewAI framework.
 
-## Task Guardrails
 
-Task guardrails provide a powerful way to validate, transform, or filter task outputs before they are passed to the next task. Guardrails are optional functions that execute before the next task starts, allowing you to ensure that task outputs meet specific requirements or formats.
-
-### Basic Usage
-
-#### Define your own logic to validate
-
-```python Code
-from typing import Tuple, Union
-from crewai import Task
-
-def validate_json_output(result: str) -> Tuple[bool, Union[dict, str]]:
-    """Validate that the output is valid JSON."""
-    try:
-        json_data = json.loads(result)
-        return (True, json_data)
-    except json.JSONDecodeError:
-        return (False, "Output must be valid JSON")
-
-task = Task(
-    description="Generate JSON data",
-    expected_output="Valid JSON object",
-    guardrail=validate_json_output
-)
-```
-
-#### Leverage a no-code approach for validation
-
-```python Code
-from crewai import Task
-
-task = Task(
-    description="Generate JSON data",
-    expected_output="Valid JSON object",
-    guardrail="Ensure the response is a valid JSON object"
-)
-```
-
-#### Using YAML
-
-```yaml
-research_task:
-  ...
-  guardrail: make sure each bullet contains a minimum of 100 words
-  ...
-```
-
-```python Code
-@CrewBase
-class InternalCrew:
-    agents_config = "config/agents.yaml"
-    tasks_config = "config/tasks.yaml"
-
-    ...
-    @task
-    def research_task(self):
-        return Task(config=self.tasks_config["research_task"])  # type: ignore[index]
-    ...
-```
-
-
-#### Use custom models for code generation
-
-```python Code
-from crewai import Task
-from crewai.llm import LLM
-
-task = Task(
-    description="Generate JSON data",
-    expected_output="Valid JSON object",
-    guardrail=LLMGuardrail(
-        description="Ensure the response is a valid JSON object",
-        llm=LLM(model="gpt-4o-mini"),
-    )
-)
-```
-
-### How Guardrails Work
-
-1. **Optional Attribute**: Guardrails are an optional attribute at the task level, allowing you to add validation only where needed.
-2. **Execution Timing**: The guardrail function is executed before the next task starts, ensuring valid data flow between tasks.
-3. **Return Format**: Guardrails must return a tuple of `(success, data)`:
-   - If `success` is `True`, `data` is the validated/transformed result
-   - If `success` is `False`, `data` is the error message
-4. **Result Routing**:
-   - On success (`True`), the result is automatically passed to the next task
-   - On failure (`False`), the error is sent back to the agent to generate a new answer
-
-### Common Use Cases
-
-#### Data Format Validation
-```python Code
-def validate_email_format(result: str) -> Tuple[bool, Union[str, str]]:
-    """Ensure the output contains a valid email address."""
-    import re
-    email_pattern = r'^[\w\.-]+@[\w\.-]+\.\w+$'
-    if re.match(email_pattern, result.strip()):
-        return (True, result.strip())
-    return (False, "Output must be a valid email address")
-```
-
-#### Content Filtering
-```python Code
-def filter_sensitive_info(result: str) -> Tuple[bool, Union[str, str]]:
-    """Remove or validate sensitive information."""
-    sensitive_patterns = ['SSN:', 'password:', 'secret:']
-    for pattern in sensitive_patterns:
-        if pattern.lower() in result.lower():
-            return (False, f"Output contains sensitive information ({pattern})")
-    return (True, result)
-```
-
-#### Data Transformation
-```python Code
-def normalize_phone_number(result: str) -> Tuple[bool, Union[str, str]]:
-    """Ensure phone numbers are in a consistent format."""
-    import re
-    digits = re.sub(r'\D', '', result)
-    if len(digits) == 10:
-        formatted = f"({digits[:3]}) {digits[3:6]}-{digits[6:]}"
-        return (True, formatted)
-    return (False, "Output must be a 10-digit phone number")
-```
-
-### Advanced Features
-
-#### Chaining Multiple Validations
-```python Code
-def chain_validations(*validators):
-    """Chain multiple validators together."""
-    def combined_validator(result):
-        for validator in validators:
-            success, data = validator(result)
-            if not success:
-                return (False, data)
-            result = data
-        return (True, result)
-    return combined_validator
-
-# Usage
-task = Task(
-    description="Get user contact info",
-    expected_output="Email and phone",
-    guardrail=chain_validations(
-        validate_email_format,
-        filter_sensitive_info
-    )
-)
-```
-
-#### Custom Retry Logic
-```python Code
-task = Task(
-    description="Generate data",
-    expected_output="Valid data",
-    guardrail=validate_data,
-    max_retries=5  # Override default retry limit
-)
-```
 
 ## Creating Directories when Saving Files
 

docs/pt-BR/concepts/cli.mdx

@@ -76,6 +76,7 @@ Exemplo:
 crewai train -n 10 -f my_training_data.pkl
 ```
 
+```python
 # Exemplo de uso programático do comando train
 n_iterations = 2
 inputs = {"topic": "Treinamento CrewAI"}
@@ -89,6 +90,7 @@ try:
     )
 except Exception as e:
     raise Exception(f"Ocorreu um erro ao treinar a crew: {e}")
+```
 
 ### 4. Replay
 

docs/pt-BR/concepts/tasks.mdx

@@ -57,6 +57,7 @@ crew = Crew(
 | **Saída JSON** _(opcional)_      | `output_json`     | `Optional[Type[BaseModel]]`  | Um modelo Pydantic para estruturar a saída em JSON.                                                                |
 | **Output Pydantic** _(opcional)_ | `output_pydantic` | `Optional[Type[BaseModel]]`  | Um modelo Pydantic para a saída da tarefa.                                                                         |
 | **Callback** _(opcional)_        | `callback`        | `Optional[Any]`              | Função/objeto a ser executado após a conclusão da tarefa.                                                          |
+| **Guardrail** _(opcional)_       | `guardrail`       | `Optional[Union[Callable, str]]` | Função ou descrição em string para validar a saída da tarefa antes de prosseguir para a próxima tarefa.        |
 
 ## Criando Tarefas
 
@@ -86,6 +87,7 @@ research_task:
   expected_output: >
     Uma lista com 10 tópicos em bullet points das informações mais relevantes sobre {topic}
   agent: researcher
+  guardrail: garanta que cada bullet point contenha no mínimo 100 palavras
 
 reporting_task:
   description: >
@@ -330,9 +332,13 @@ analysis_task = Task(
 
 Guardrails (trilhas de proteção) de tarefas fornecem uma maneira de validar e transformar as saídas das tarefas antes que elas sejam passadas para a próxima tarefa. Esse recurso assegura a qualidade dos dados e oferece feedback aos agentes quando sua saída não atende a critérios específicos.
 
-### Usando Guardrails em Tarefas
+**Guardrails podem ser definidos de duas maneiras:**
+1. **Guardrails baseados em função**: Funções Python que implementam lógica de validação customizada
+2. **Guardrails baseados em string**: Descrições em linguagem natural que são automaticamente convertidas em validação baseada em LLM
 
-Para adicionar um guardrail a uma tarefa, forneça uma função de validação por meio do parâmetro `guardrail`:
+### Guardrails Baseados em Função
+
+Para adicionar um guardrail baseado em função a uma tarefa, forneça uma função de validação por meio do parâmetro `guardrail`:
 
 ```python Code
 from typing import Tuple, Union, Dict, Any
@@ -370,9 +376,82 @@ blog_task = Task(
    - Em caso de sucesso: retorna uma tupla `(True, resultado_validado)`
    - Em caso de falha: retorna uma tupla `(False, "mensagem de erro explicando a falha")`
 
-### LLMGuardrail
+### Guardrails Baseados em String
 
-A classe `LLMGuardrail` oferece um mecanismo robusto para validação das saídas das tarefas.
+Guardrails baseados em string permitem que você descreva critérios de validação em linguagem natural. Quando você fornece uma string em vez de uma função, o CrewAI automaticamente a converte em um `LLMGuardrail` que usa um agente de IA para validar a saída da tarefa.
+
+#### Usando Guardrails de String em Python
+
+```python Code
+from crewai import Task
+
+# Guardrail simples baseado em string
+blog_task = Task(
+    description="Escreva um post de blog sobre IA",
+    expected_output="Um post de blog com menos de 200 palavras",
+    agent=blog_agent,
+    guardrail="Garanta que o post do blog tenha menos de 200 palavras e inclua exemplos práticos"
+)
+
+# Critérios de validação mais complexos
+research_task = Task(
+    description="Pesquise tendências de IA para 2025",
+    expected_output="Um relatório abrangente de pesquisa",
+    agent=research_agent,
+    guardrail="Garanta que cada descoberta inclua uma fonte confiável e seja respaldada por dados recentes de 2024-2025"
+)
+```
+
+#### Usando Guardrails de String em YAML
+
+```yaml
+research_task:
+  description: Pesquise os últimos desenvolvimentos em IA
+  expected_output: Uma lista de 10 bullet points sobre IA
+  agent: researcher
+  guardrail: garanta que cada bullet point contenha no mínimo 100 palavras
+
+validation_task:
+  description: Valide os achados da pesquisa
+  expected_output: Um relatório de validação
+  agent: validator
+  guardrail: confirme que todas as fontes são de publicações respeitáveis e publicadas nos últimos 2 anos
+```
+
+#### Como Funcionam os Guardrails de String
+
+Quando você fornece um guardrail de string, o CrewAI automaticamente:
+1. Cria uma instância `LLMGuardrail` usando a string como critério de validação
+2. Usa o LLM do agente da tarefa para alimentar a validação
+3. Cria um agente temporário de validação que verifica a saída contra seus critérios
+4. Retorna feedback detalhado se a validação falhar
+
+Esta abordagem é ideal quando você quer usar linguagem natural para descrever regras de validação sem escrever funções de validação customizadas.
+
+### Classe LLMGuardrail
+
+A classe `LLMGuardrail` é o mecanismo subjacente que alimenta os guardrails baseados em string. Você também pode usá-la diretamente para maior controle avançado:
+
+```python Code
+from crewai import Task
+from crewai.tasks.llm_guardrail import LLMGuardrail
+from crewai.llm import LLM
+
+# Crie um LLMGuardrail customizado com LLM específico
+custom_guardrail = LLMGuardrail(
+    description="Garanta que a resposta contenha exatamente 5 bullet points com citações adequadas",
+    llm=LLM(model="gpt-4o-mini")
+)
+
+task = Task(
+    description="Pesquise medidas de segurança em IA",
+    expected_output="Uma análise detalhada com bullet points",
+    agent=research_agent,
+    guardrail=custom_guardrail
+)
+```
+
+**Nota**: Quando você usa um guardrail de string, o CrewAI automaticamente cria uma instância `LLMGuardrail` usando o LLM do agente da sua tarefa. Usar `LLMGuardrail` diretamente lhe dá mais controle sobre o processo de validação e seleção de LLM.
 
 ### Melhores Práticas de Tratamento de Erros
 

src/crewai/agent.py

@@ -210,7 +210,6 @@ class Agent(BaseAgent):
                         sources=self.knowledge_sources,
                         embedder=self.embedder,
                         collection_name=self.role,
-                        storage=self.knowledge_storage or None,
                     )
                     self.knowledge.add_sources()
         except (TypeError, ValueError) as e:
@@ -341,7 +340,8 @@ class Agent(BaseAgent):
             self.knowledge_config.model_dump() if self.knowledge_config else {}
         )
 
-        if self.knowledge:
+
+        if self.knowledge or (self.crew and self.crew.knowledge):
             crewai_event_bus.emit(
                 self,
                 event=KnowledgeRetrievalStartedEvent(
@@ -353,25 +353,28 @@ class Agent(BaseAgent):
                     task_prompt
                 )
                 if self.knowledge_search_query:
-                    agent_knowledge_snippets = self.knowledge.query(
-                        [self.knowledge_search_query], **knowledge_config
-                    )
-                    if agent_knowledge_snippets:
-                        self.agent_knowledge_context = extract_knowledge_context(
-                            agent_knowledge_snippets
-                        )
-                        if self.agent_knowledge_context:
-                            task_prompt += self.agent_knowledge_context
-                    if self.crew:
-                        knowledge_snippets = self.crew.query_knowledge(
+                    # Quering agent specific knowledge
+                    if self.knowledge:
+                        agent_knowledge_snippets = self.knowledge.query(
                             [self.knowledge_search_query], **knowledge_config
                         )
-                        if knowledge_snippets:
-                            self.crew_knowledge_context = extract_knowledge_context(
-                                knowledge_snippets
+                        if agent_knowledge_snippets:
+                            self.agent_knowledge_context = extract_knowledge_context(
+                                agent_knowledge_snippets
                             )
-                            if self.crew_knowledge_context:
-                                task_prompt += self.crew_knowledge_context
+                            if self.agent_knowledge_context:
+                                task_prompt += self.agent_knowledge_context
+
+                    # Quering crew specific knowledge
+                    knowledge_snippets = self.crew.query_knowledge(
+                        [self.knowledge_search_query], **knowledge_config
+                    )
+                    if knowledge_snippets:
+                        self.crew_knowledge_context = extract_knowledge_context(
+                            knowledge_snippets
+                        )
+                        if self.crew_knowledge_context:
+                            task_prompt += self.crew_knowledge_context
 
                     crewai_event_bus.emit(
                         self,

src/crewai/crew.py

@@ -1337,7 +1337,7 @@ class Crew(FlowTrackable, BaseModel):
             evaluator = CrewEvaluator(test_crew, llm_instance)
 
             if include_agent_eval:
-                from crewai.experimental.evaluation import create_default_evaluator
+                from crewai.evaluation import create_default_evaluator
                 agent_evaluator = create_default_evaluator(crew=test_crew)
 
             for i in range(1, n_iterations + 1):

src/crewai/evaluation/__init__.py

@@ -1,35 +1,40 @@
-from crewai.experimental.evaluation.base_evaluator import (
+from crewai.evaluation.base_evaluator import (
     BaseEvaluator,
     EvaluationScore,
     MetricCategory,
     AgentEvaluationResult
 )
 
-from crewai.experimental.evaluation.metrics import (
-    SemanticQualityEvaluator,
-    GoalAlignmentEvaluator,
-    ReasoningEfficiencyEvaluator,
+from crewai.evaluation.metrics.semantic_quality_metrics import (
+    SemanticQualityEvaluator
+)
+
+from crewai.evaluation.metrics.goal_metrics import (
+    GoalAlignmentEvaluator
+)
+
+from crewai.evaluation.metrics.reasoning_metrics import (
+    ReasoningEfficiencyEvaluator
+)
+
+
+from crewai.evaluation.metrics.tools_metrics import (
     ToolSelectionEvaluator,
     ParameterExtractionEvaluator,
     ToolInvocationEvaluator
 )
 
-from crewai.experimental.evaluation.evaluation_listener import (
+from crewai.evaluation.evaluation_listener import (
     EvaluationTraceCallback,
     create_evaluation_callbacks
 )
 
-from crewai.experimental.evaluation.agent_evaluator import (
+
+from crewai.evaluation.agent_evaluator import (
     AgentEvaluator,
     create_default_evaluator
 )
 
-from crewai.experimental.evaluation.experiment import (
-    ExperimentRunner,
-    ExperimentResults,
-    ExperimentResult
-)
-
 __all__ = [
     "BaseEvaluator",
     "EvaluationScore",
@@ -44,8 +49,5 @@ __all__ = [
     "EvaluationTraceCallback",
     "create_evaluation_callbacks",
     "AgentEvaluator",
-    "create_default_evaluator",
-    "ExperimentRunner",
-    "ExperimentResults",
-    "ExperimentResult"
-]
+    "create_default_evaluator"
+]

src/crewai/evaluation/agent_evaluator.py

@@ -1,16 +1,15 @@
-from crewai.experimental.evaluation.base_evaluator import AgentEvaluationResult, AggregationStrategy
+from crewai.evaluation.base_evaluator import AgentEvaluationResult, AggregationStrategy
 from crewai.agent import Agent
 from crewai.task import Task
-from crewai.experimental.evaluation.evaluation_display import EvaluationDisplayFormatter
+from crewai.evaluation.evaluation_display import EvaluationDisplayFormatter
 
 from typing import Any, Dict
 from collections import defaultdict
-from crewai.experimental.evaluation import BaseEvaluator, create_evaluation_callbacks
+from crewai.evaluation import BaseEvaluator, create_evaluation_callbacks
 from collections.abc import Sequence
 from crewai.crew import Crew
 from crewai.utilities.events.crewai_event_bus import crewai_event_bus
 from crewai.utilities.events.utils.console_formatter import ConsoleFormatter
-from crewai.experimental.evaluation.evaluation_display import AgentAggregatedEvaluationResult
 
 class AgentEvaluator:
     def __init__(
@@ -37,9 +36,6 @@ class AgentEvaluator:
     def set_iteration(self, iteration: int) -> None:
         self.iteration = iteration
 
-    def reset_iterations_results(self):
-        self.iterations_results = {}
-
     def evaluate_current_iteration(self) -> dict[str, list[AgentEvaluationResult]]:
         if not self.crew:
             raise ValueError("Cannot evaluate: no crew was provided to the evaluator.")
@@ -103,7 +99,7 @@ class AgentEvaluator:
     def display_results_with_iterations(self):
         self.display_formatter.display_summary_results(self.iterations_results)
 
-    def get_agent_evaluation(self, strategy: AggregationStrategy = AggregationStrategy.SIMPLE_AVERAGE, include_evaluation_feedback: bool = False) -> Dict[str, AgentAggregatedEvaluationResult]:
+    def get_agent_evaluation(self, strategy: AggregationStrategy = AggregationStrategy.SIMPLE_AVERAGE, include_evaluation_feedback: bool = False):
         agent_results = {}
         with crewai_event_bus.scoped_handlers():
             task_results = self.get_evaluation_results()
@@ -161,7 +157,7 @@ class AgentEvaluator:
         return result
 
 def create_default_evaluator(crew, llm=None):
-    from crewai.experimental.evaluation import (
+    from crewai.evaluation import (
         GoalAlignmentEvaluator,
         SemanticQualityEvaluator,
         ToolSelectionEvaluator,

src/crewai/evaluation/evaluation_display.py

@@ -3,8 +3,8 @@ from typing import Dict, Any, List
 from rich.table import Table
 from rich.box import HEAVY_EDGE, ROUNDED
 from collections.abc import Sequence
-from crewai.experimental.evaluation.base_evaluator import AgentAggregatedEvaluationResult, AggregationStrategy, AgentEvaluationResult, MetricCategory
-from crewai.experimental.evaluation import EvaluationScore
+from crewai.evaluation.base_evaluator import AgentAggregatedEvaluationResult, AggregationStrategy, AgentEvaluationResult, MetricCategory
+from crewai.evaluation import EvaluationScore
 from crewai.utilities.events.utils.console_formatter import ConsoleFormatter
 from crewai.utilities.llm_utils import create_llm
 

src/crewai/evaluation/metrics/goal_metrics.py

@@ -3,8 +3,8 @@ from typing import Any, Dict
 from crewai.agent import Agent
 from crewai.task import Task
 
-from crewai.experimental.evaluation.base_evaluator import BaseEvaluator, EvaluationScore, MetricCategory
-from crewai.experimental.evaluation.json_parser import extract_json_from_llm_response
+from crewai.evaluation.base_evaluator import BaseEvaluator, EvaluationScore, MetricCategory
+from crewai.evaluation.json_parser import extract_json_from_llm_response
 
 class GoalAlignmentEvaluator(BaseEvaluator):
     @property

src/crewai/evaluation/metrics/reasoning_metrics.py

@@ -16,8 +16,8 @@ from collections.abc import Sequence
 from crewai.agent import Agent
 from crewai.task import Task
 
-from crewai.experimental.evaluation.base_evaluator import BaseEvaluator, EvaluationScore, MetricCategory
-from crewai.experimental.evaluation.json_parser import extract_json_from_llm_response
+from crewai.evaluation.base_evaluator import BaseEvaluator, EvaluationScore, MetricCategory
+from crewai.evaluation.json_parser import extract_json_from_llm_response
 from crewai.tasks.task_output import TaskOutput
 
 class ReasoningPatternType(Enum):

src/crewai/evaluation/metrics/semantic_quality_metrics.py

@@ -3,8 +3,8 @@ from typing import Any, Dict
 from crewai.agent import Agent
 from crewai.task import Task
 
-from crewai.experimental.evaluation.base_evaluator import BaseEvaluator, EvaluationScore, MetricCategory
-from crewai.experimental.evaluation.json_parser import extract_json_from_llm_response
+from crewai.evaluation.base_evaluator import BaseEvaluator, EvaluationScore, MetricCategory
+from crewai.evaluation.json_parser import extract_json_from_llm_response
 
 class SemanticQualityEvaluator(BaseEvaluator):
     @property

src/crewai/evaluation/metrics/tools_metrics.py

@@ -1,8 +1,8 @@
 import json
 from typing import Dict, Any
 
-from crewai.experimental.evaluation.base_evaluator import BaseEvaluator, EvaluationScore, MetricCategory
-from crewai.experimental.evaluation.json_parser import extract_json_from_llm_response
+from crewai.evaluation.base_evaluator import BaseEvaluator, EvaluationScore, MetricCategory
+from crewai.evaluation.json_parser import extract_json_from_llm_response
 from crewai.agent import Agent
 from crewai.task import Task
 

src/crewai/experimental/__init__.py

@@ -1,40 +0,0 @@
-from crewai.experimental.evaluation import (
-    BaseEvaluator,
-    EvaluationScore,
-    MetricCategory,
-    AgentEvaluationResult,
-    SemanticQualityEvaluator,
-    GoalAlignmentEvaluator,
-    ReasoningEfficiencyEvaluator,
-    ToolSelectionEvaluator,
-    ParameterExtractionEvaluator,
-    ToolInvocationEvaluator,
-    EvaluationTraceCallback,
-    create_evaluation_callbacks,
-    AgentEvaluator,
-    create_default_evaluator,
-    ExperimentRunner,
-    ExperimentResults,
-    ExperimentResult,
-)
-
-
-__all__ = [
-    "BaseEvaluator",
-    "EvaluationScore",
-    "MetricCategory",
-    "AgentEvaluationResult",
-    "SemanticQualityEvaluator",
-    "GoalAlignmentEvaluator",
-    "ReasoningEfficiencyEvaluator",
-    "ToolSelectionEvaluator",
-    "ParameterExtractionEvaluator",
-    "ToolInvocationEvaluator",
-    "EvaluationTraceCallback",
-    "create_evaluation_callbacks",
-    "AgentEvaluator",
-    "create_default_evaluator",
-    "ExperimentRunner",
-    "ExperimentResults",
-    "ExperimentResult"
-]

src/crewai/experimental/evaluation/experiment/__init__.py

@@ -1,8 +0,0 @@
-from crewai.experimental.evaluation.experiment.runner import ExperimentRunner
-from crewai.experimental.evaluation.experiment.result import ExperimentResults, ExperimentResult
-
-__all__ = [
-    "ExperimentRunner",
-    "ExperimentResults",
-    "ExperimentResult"
-]

src/crewai/experimental/evaluation/experiment/result.py

@@ -1,122 +0,0 @@
-import json
-import os
-from datetime import datetime, timezone
-from typing import Any
-from pydantic import BaseModel
-
-class ExperimentResult(BaseModel):
-    identifier: str
-    inputs: dict[str, Any]
-    score: int | dict[str, int | float]
-    expected_score: int | dict[str, int | float]
-    passed: bool
-    agent_evaluations: dict[str, Any] | None = None
-
-class ExperimentResults:
-    def __init__(self, results: list[ExperimentResult], metadata: dict[str, Any] | None = None):
-        self.results = results
-        self.metadata = metadata or {}
-        self.timestamp = datetime.now(timezone.utc)
-
-        from crewai.experimental.evaluation.experiment.result_display import ExperimentResultsDisplay
-        self.display = ExperimentResultsDisplay()
-
-    def to_json(self, filepath: str | None = None) -> dict[str, Any]:
-        data = {
-            "timestamp": self.timestamp.isoformat(),
-            "metadata": self.metadata,
-            "results": [r.model_dump(exclude={"agent_evaluations"}) for r in self.results]
-        }
-
-        if filepath:
-            with open(filepath, 'w') as f:
-                json.dump(data, f, indent=2)
-            self.display.console.print(f"[green]Results saved to {filepath}[/green]")
-
-        return data
-
-    def compare_with_baseline(self, baseline_filepath: str, save_current: bool = True, print_summary: bool = False) -> dict[str, Any]:
-        baseline_runs = []
-
-        if os.path.exists(baseline_filepath) and os.path.getsize(baseline_filepath) > 0:
-            try:
-                with open(baseline_filepath, 'r') as f:
-                    baseline_data = json.load(f)
-
-                if isinstance(baseline_data, dict) and "timestamp" in baseline_data:
-                    baseline_runs = [baseline_data]
-                elif isinstance(baseline_data, list):
-                    baseline_runs = baseline_data
-            except (json.JSONDecodeError, FileNotFoundError) as e:
-                self.display.console.print(f"[yellow]Warning: Could not load baseline file: {str(e)}[/yellow]")
-
-        if not baseline_runs:
-            if save_current:
-                current_data = self.to_json()
-                with open(baseline_filepath, 'w') as f:
-                    json.dump([current_data], f, indent=2)
-                self.display.console.print(f"[green]Saved current results as new baseline to {baseline_filepath}[/green]")
-            return {"is_baseline": True, "changes": {}}
-
-        baseline_runs.sort(key=lambda x: x.get("timestamp", ""), reverse=True)
-        latest_run = baseline_runs[0]
-
-        comparison = self._compare_with_run(latest_run)
-
-        if print_summary:
-            self.display.comparison_summary(comparison, latest_run["timestamp"])
-
-        if save_current:
-            current_data = self.to_json()
-            baseline_runs.append(current_data)
-            with open(baseline_filepath, 'w') as f:
-                json.dump(baseline_runs, f, indent=2)
-            self.display.console.print(f"[green]Added current results to baseline file {baseline_filepath}[/green]")
-
-        return comparison
-
-    def _compare_with_run(self, baseline_run: dict[str, Any]) -> dict[str, Any]:
-        baseline_results = baseline_run.get("results", [])
-
-        baseline_lookup = {}
-        for result in baseline_results:
-            test_identifier = result.get("identifier")
-            if test_identifier:
-                baseline_lookup[test_identifier] = result
-
-        improved = []
-        regressed = []
-        unchanged = []
-        new_tests = []
-
-        for result in self.results:
-            test_identifier = result.identifier
-            if not test_identifier or test_identifier not in baseline_lookup:
-                new_tests.append(test_identifier)
-                continue
-
-            baseline_result = baseline_lookup[test_identifier]
-            baseline_passed = baseline_result.get("passed", False)
-            if result.passed and not baseline_passed:
-                improved.append(test_identifier)
-            elif not result.passed and baseline_passed:
-                regressed.append(test_identifier)
-            else:
-                unchanged.append(test_identifier)
-
-        missing_tests = []
-        current_test_identifiers = {result.identifier for result in self.results}
-        for result in baseline_results:
-            test_identifier = result.get("identifier")
-            if test_identifier and test_identifier not in current_test_identifiers:
-                missing_tests.append(test_identifier)
-
-        return {
-            "improved": improved,
-            "regressed": regressed,
-            "unchanged": unchanged,
-            "new_tests": new_tests,
-            "missing_tests": missing_tests,
-            "total_compared": len(improved) + len(regressed) + len(unchanged),
-            "baseline_timestamp": baseline_run.get("timestamp", "unknown")
-        }

src/crewai/experimental/evaluation/experiment/result_display.py

@@ -1,70 +0,0 @@
-from typing import Dict, Any
-from rich.console import Console
-from rich.table import Table
-from rich.panel import Panel
-from crewai.experimental.evaluation.experiment.result import ExperimentResults
-
-class ExperimentResultsDisplay:
-    def __init__(self):
-        self.console = Console()
-
-    def summary(self, experiment_results: ExperimentResults):
-        total = len(experiment_results.results)
-        passed = sum(1 for r in experiment_results.results if r.passed)
-
-        table = Table(title="Experiment Summary")
-        table.add_column("Metric", style="cyan")
-        table.add_column("Value", style="green")
-
-        table.add_row("Total Test Cases", str(total))
-        table.add_row("Passed", str(passed))
-        table.add_row("Failed", str(total - passed))
-        table.add_row("Success Rate", f"{(passed / total * 100):.1f}%" if total > 0 else "N/A")
-
-        self.console.print(table)
-
-    def comparison_summary(self, comparison: Dict[str, Any], baseline_timestamp: str):
-        self.console.print(Panel(f"[bold]Comparison with baseline run from {baseline_timestamp}[/bold]",
-                                 expand=False))
-
-        table = Table(title="Results Comparison")
-        table.add_column("Metric", style="cyan")
-        table.add_column("Count", style="white")
-        table.add_column("Details", style="dim")
-
-        improved = comparison.get("improved", [])
-        if improved:
-            details = ", ".join([f"{test_identifier}" for test_identifier in improved[:3]])
-            if len(improved) > 3:
-                details += f" and {len(improved) - 3} more"
-            table.add_row("✅ Improved", str(len(improved)), details)
-        else:
-            table.add_row("✅ Improved", "0", "")
-
-        regressed = comparison.get("regressed", [])
-        if regressed:
-            details = ", ".join([f"{test_identifier}" for test_identifier in regressed[:3]])
-            if len(regressed) > 3:
-                details += f" and {len(regressed) - 3} more"
-            table.add_row("❌ Regressed", str(len(regressed)), details, style="red")
-        else:
-            table.add_row("❌ Regressed", "0", "")
-
-        unchanged = comparison.get("unchanged", [])
-        table.add_row("⏺ Unchanged", str(len(unchanged)), "")
-
-        new_tests = comparison.get("new_tests", [])
-        if new_tests:
-            details = ", ".join(new_tests[:3])
-            if len(new_tests) > 3:
-                details += f" and {len(new_tests) - 3} more"
-            table.add_row("➕ New Tests", str(len(new_tests)), details)
-
-        missing_tests = comparison.get("missing_tests", [])
-        if missing_tests:
-            details = ", ".join(missing_tests[:3])
-            if len(missing_tests) > 3:
-                details += f" and {len(missing_tests) - 3} more"
-            table.add_row("➖ Missing Tests", str(len(missing_tests)), details)
-
-        self.console.print(table)

src/crewai/experimental/evaluation/experiment/runner.py

@@ -1,117 +0,0 @@
-from collections import defaultdict
-from hashlib import md5
-from typing import Any
-
-from crewai import Crew
-from crewai.experimental.evaluation import AgentEvaluator, create_default_evaluator
-from crewai.experimental.evaluation.experiment.result_display import ExperimentResultsDisplay
-from crewai.experimental.evaluation.experiment.result import ExperimentResults, ExperimentResult
-from crewai.experimental.evaluation.evaluation_display import AgentAggregatedEvaluationResult
-
-class ExperimentRunner:
-    def __init__(self, dataset: list[dict[str, Any]]):
-        self.dataset = dataset or []
-        self.evaluator: AgentEvaluator | None = None
-        self.display = ExperimentResultsDisplay()
-
-    def run(self, crew: Crew, print_summary: bool = False) -> ExperimentResults:
-        self.evaluator = create_default_evaluator(crew=crew)
-
-        results = []
-
-        for test_case in self.dataset:
-            self.evaluator.reset_iterations_results()
-            result = self._run_test_case(test_case, crew)
-            results.append(result)
-
-        experiment_results = ExperimentResults(results)
-
-        if print_summary:
-            self.display.summary(experiment_results)
-
-        return experiment_results
-
-    def _run_test_case(self, test_case: dict[str, Any], crew: Crew) -> ExperimentResult:
-        inputs = test_case["inputs"]
-        expected_score = test_case["expected_score"]
-        identifier = test_case.get("identifier") or md5(str(test_case).encode(), usedforsecurity=False).hexdigest()
-
-        try:
-            self.display.console.print(f"[dim]Running crew with input: {str(inputs)[:50]}...[/dim]")
-            self.display.console.print("\n")
-            crew.kickoff(inputs=inputs)
-
-            assert self.evaluator is not None
-            agent_evaluations = self.evaluator.get_agent_evaluation()
-
-            actual_score = self._extract_scores(agent_evaluations)
-
-            passed = self._assert_scores(expected_score, actual_score)
-            return ExperimentResult(
-                identifier=identifier,
-                inputs=inputs,
-                score=actual_score,
-                expected_score=expected_score,
-                passed=passed,
-                agent_evaluations=agent_evaluations
-            )
-
-        except Exception as e:
-            self.display.console.print(f"[red]Error running test case: {str(e)}[/red]")
-            return ExperimentResult(
-                identifier=identifier,
-                inputs=inputs,
-                score=0,
-                expected_score=expected_score,
-                passed=False
-            )
-
-    def _extract_scores(self, agent_evaluations: dict[str, AgentAggregatedEvaluationResult]) -> float | dict[str,  float]:
-        all_scores: dict[str, list[float]] = defaultdict(list)
-        for evaluation in agent_evaluations.values():
-            for metric_name, score in evaluation.metrics.items():
-                if score.score is not None:
-                    all_scores[metric_name.value].append(score.score)
-
-        avg_scores = {m: sum(s)/len(s) for m, s in all_scores.items()}
-
-        if len(avg_scores) == 1:
-            return list(avg_scores.values())[0]
-
-        return avg_scores
-
-    def _assert_scores(self, expected: float | dict[str, float],
-                        actual: float | dict[str, float]) -> bool:
-        """
-        Compare expected and actual scores, and return whether the test case passed.
-
-        The rules for comparison are as follows:
-        - If both expected and actual scores are single numbers, the actual score must be >= expected.
-        - If expected is a single number and actual is a dict, compare against the average of actual values.
-        - If expected is a dict and actual is a single number, actual must be >= all expected values.
-        - If both are dicts, actual must have matching keys with values >= expected values.
-        """
-
-        if isinstance(expected, (int, float)) and isinstance(actual, (int, float)):
-            return actual >= expected
-
-        if isinstance(expected, dict) and isinstance(actual, (int, float)):
-            return all(actual >= exp_score for exp_score in expected.values())
-
-        if isinstance(expected, (int, float)) and isinstance(actual, dict):
-            if not actual:
-                return False
-            avg_score = sum(actual.values()) / len(actual)
-            return avg_score >= expected
-
-        if isinstance(expected, dict) and isinstance(actual, dict):
-            if not expected:
-                return True
-            matching_keys = set(expected.keys()) & set(actual.keys())
-            if not matching_keys:
-                return False
-
-            # All matching keys must have actual >= expected
-            return all(actual[key] >= expected[key] for key in matching_keys)
-
-        return False

src/crewai/experimental/evaluation/metrics/__init__.py

@@ -1,26 +0,0 @@
-from crewai.experimental.evaluation.metrics.reasoning_metrics import (
-    ReasoningEfficiencyEvaluator
-)
-
-from crewai.experimental.evaluation.metrics.tools_metrics import (
-    ToolSelectionEvaluator,
-    ParameterExtractionEvaluator,
-    ToolInvocationEvaluator
-)
-
-from crewai.experimental.evaluation.metrics.goal_metrics import (
-    GoalAlignmentEvaluator
-)
-
-from crewai.experimental.evaluation.metrics.semantic_quality_metrics import (
-    SemanticQualityEvaluator
-)
-
-__all__ = [
-    "ReasoningEfficiencyEvaluator",
-    "ToolSelectionEvaluator",
-    "ParameterExtractionEvaluator",
-    "ToolInvocationEvaluator",
-    "GoalAlignmentEvaluator",
-    "SemanticQualityEvaluator"
-]

src/crewai/lite_agent.py

@@ -28,7 +28,7 @@ from pydantic import (
     InstanceOf,
     PrivateAttr,
     model_validator,
-    field_validator,
+    field_validator
 )
 
 from crewai.agents.agent_builder.base_agent import BaseAgent
@@ -40,7 +40,7 @@ from crewai.agents.parser import (
     OutputParserException,
 )
 from crewai.flow.flow_trackable import FlowTrackable
-from crewai.llm import LLM
+from crewai.llm import LLM, BaseLLM
 from crewai.tools.base_tool import BaseTool
 from crewai.tools.structured_tool import CrewStructuredTool
 from crewai.utilities import I18N
@@ -135,7 +135,7 @@ class LiteAgent(FlowTrackable, BaseModel):
     role: str = Field(description="Role of the agent")
     goal: str = Field(description="Goal of the agent")
     backstory: str = Field(description="Backstory of the agent")
-    llm: Optional[Union[str, InstanceOf[LLM], Any]] = Field(
+    llm: Optional[Union[str, InstanceOf[BaseLLM], Any]] = Field(
         default=None, description="Language model that will run the agent"
     )
     tools: List[BaseTool] = Field(
@@ -209,8 +209,8 @@ class LiteAgent(FlowTrackable, BaseModel):
     def setup_llm(self):
         """Set up the LLM and other components after initialization."""
         self.llm = create_llm(self.llm)
-        if not isinstance(self.llm, LLM):
-            raise ValueError("Unable to create LLM instance")
+        if not isinstance(self.llm, BaseLLM):
+            raise ValueError(f"Expected LLM instance of type BaseLLM, got {type(self.llm).__name__}")
 
         # Initialize callbacks
         token_callback = TokenCalcHandler(token_cost_process=self._token_process)
@@ -232,7 +232,8 @@ class LiteAgent(FlowTrackable, BaseModel):
         elif isinstance(self.guardrail, str):
             from crewai.tasks.llm_guardrail import LLMGuardrail
 
-            assert isinstance(self.llm, LLM)
+            if not isinstance(self.llm, BaseLLM):
+                raise TypeError(f"Guardrail requires LLM instance of type BaseLLM, got {type(self.llm).__name__}")
 
             self._guardrail = LLMGuardrail(description=self.guardrail, llm=self.llm)
 
@@ -620,4 +621,4 @@ class LiteAgent(FlowTrackable, BaseModel):
 
     def _append_message(self, text: str, role: str = "assistant") -> None:
         """Append a message to the message list with the given role."""
-        self._messages.append(format_message_for_llm(text, role=role))
+        self._messages.append(format_message_for_llm(text, role=role))

src/crewai/task.py

@@ -67,6 +67,7 @@ class Task(BaseModel):
         description: Descriptive text detailing task's purpose and execution.
         expected_output: Clear definition of expected task outcome.
         output_file: File path for storing task output.
+        create_directory: Whether to create the directory for output_file if it doesn't exist.
         output_json: Pydantic model for structuring JSON output.
         output_pydantic: Pydantic model for task output.
         security_config: Security configuration including fingerprinting.
@@ -115,6 +116,10 @@ class Task(BaseModel):
         description="A file path to be used to create a file output.",
         default=None,
     )
+    create_directory: Optional[bool] = Field(
+        description="Whether to create the directory for output_file if it doesn't exist.",
+        default=True,
+    )
     output: Optional[TaskOutput] = Field(
         description="Task output, it's final result after being executed", default=None
     )
@@ -753,8 +758,10 @@ Follow these guidelines:
             resolved_path = Path(self.output_file).expanduser().resolve()
             directory = resolved_path.parent
 
-            if not directory.exists():
+            if self.create_directory and not directory.exists():
                 directory.mkdir(parents=True, exist_ok=True)
+            elif not self.create_directory and not directory.exists():
+                raise RuntimeError(f"Directory {directory} does not exist and create_directory is False")
 
             with resolved_path.open("w", encoding="utf-8") as file:
                 if isinstance(result, dict):

src/crewai/tasks/llm_guardrail.py

@@ -1,10 +1,9 @@
-from typing import Any, Optional, Tuple
+from typing import Any, Tuple
 
 from pydantic import BaseModel, Field
 
 from crewai.agent import Agent, LiteAgentOutput
-from crewai.llm import LLM
-from crewai.task import Task
+from crewai.llm import BaseLLM
 from crewai.tasks.task_output import TaskOutput
 
 
@@ -32,11 +31,11 @@ class LLMGuardrail:
     def __init__(
         self,
         description: str,
-        llm: LLM,
+        llm: BaseLLM,
     ):
         self.description = description
 
-        self.llm: LLM = llm
+        self.llm: BaseLLM = llm
 
     def _validate_output(self, task_output: TaskOutput) -> LiteAgentOutput:
         agent = Agent(

tests/agent_test.py

@@ -1896,6 +1896,80 @@ def test_agent_with_knowledge_sources_generate_search_query():
         assert "red" in result.raw.lower()
 
 
+@pytest.mark.vcr(record_mode='none', filter_headers=["authorization"])
+def test_agent_with_knowledge_with_no_crewai_knowledge():
+    mock_knowledge = MagicMock(spec=Knowledge)
+
+    agent = Agent(
+        role="Information Agent",
+        goal="Provide information based on knowledge sources",
+        backstory="You have access to specific knowledge sources.",
+        llm=LLM(model="openrouter/openai/gpt-4o-mini",api_key=os.getenv('OPENROUTER_API_KEY')),
+        knowledge=mock_knowledge
+    )
+
+    # Create a task that requires the agent to use the knowledge
+    task = Task(
+        description="What is Vidit's favorite color?",
+        expected_output="Vidit's favorclearite color.",
+        agent=agent,
+    )
+
+    crew = Crew(agents=[agent], tasks=[task])
+    crew.kickoff()
+    mock_knowledge.query.assert_called_once()
+
+
+@pytest.mark.vcr(record_mode='none', filter_headers=["authorization"])
+def test_agent_with_only_crewai_knowledge():
+    mock_knowledge = MagicMock(spec=Knowledge)
+
+    agent = Agent(
+        role="Information Agent",
+        goal="Provide information based on knowledge sources",
+        backstory="You have access to specific knowledge sources.",
+        llm=LLM(model="openrouter/openai/gpt-4o-mini",api_key=os.getenv('OPENROUTER_API_KEY'))
+    )
+
+    # Create a task that requires the agent to use the knowledge
+    task = Task(
+        description="What is Vidit's favorite color?",
+        expected_output="Vidit's favorclearite color.",
+        agent=agent
+    )
+
+    crew = Crew(agents=[agent], tasks=[task],knowledge=mock_knowledge)
+    crew.kickoff()
+    mock_knowledge.query.assert_called_once()
+
+
+@pytest.mark.vcr(record_mode='none', filter_headers=["authorization"])
+def test_agent_knowledege_with_crewai_knowledge():
+    crew_knowledge = MagicMock(spec=Knowledge)
+    agent_knowledge = MagicMock(spec=Knowledge)
+
+
+    agent = Agent(
+        role="Information Agent",
+        goal="Provide information based on knowledge sources",
+        backstory="You have access to specific knowledge sources.",
+        llm=LLM(model="openrouter/openai/gpt-4o-mini",api_key=os.getenv('OPENROUTER_API_KEY')),
+        knowledge=agent_knowledge
+    )
+
+    # Create a task that requires the agent to use the knowledge
+    task = Task(
+        description="What is Vidit's favorite color?",
+        expected_output="Vidit's favorclearite color.",
+        agent=agent,
+    )
+
+    crew = Crew(agents=[agent],tasks=[task],knowledge=crew_knowledge)
+    crew.kickoff()
+    agent_knowledge.query.assert_called_once()
+    crew_knowledge.query.assert_called_once()
+
+
 @pytest.mark.vcr(filter_headers=["authorization"])
 def test_litellm_auth_error_handling():
     """Test that LiteLLM authentication errors are handled correctly and not retried."""

tests/cassettes/test_agent_knowledege_with_crewai_knowledge.yaml

@@ -0,0 +1,150 @@
+interactions:
+- request:
+    body: '{"model": "openai/gpt-4o-mini", "messages": [{"role": "system", "content":
+      "Your goal is to rewrite the user query so that it is optimized for retrieval
+      from a vector database. Consider how the query will be used to find relevant
+      documents, and aim to make it more specific and context-aware. \n\n Do not include
+      any other text than the rewritten query, especially any preamble or postamble
+      and only add expected output format if its relevant to the rewritten query.
+      \n\n Focus on the key words of the intended task and to retrieve the most relevant
+      information. \n\n There will be some extra context provided that might need
+      to be removed such as expected_output formats structured_outputs and other instructions."},
+      {"role": "user", "content": "The original query is: What is Vidit''s favorite
+      color?\n\nThis is the expected criteria for your final answer: Vidit''s favorclearite
+      color.\nyou MUST return the actual complete content as the final answer, not
+      a summary.."}], "stream": false, "stop": ["\nObservation:"]}'
+    headers:
+      accept:
+      - '*/*'
+      accept-encoding:
+      - gzip, deflate
+      connection:
+      - keep-alive
+      content-length:
+      - '1017'
+      content-type:
+      - application/json
+      host:
+      - openrouter.ai
+      http-referer:
+      - https://litellm.ai
+      user-agent:
+      - litellm/1.68.0
+      x-title:
+      - liteLLM
+    method: POST
+    uri: https://openrouter.ai/api/v1/chat/completions
+  response:
+    body:
+      string: !!binary |
+        H4sIAAAAAAAAAwAAAP//4lKAAS4AAAAA//90kE1vE0EMhv9K9V64TMrmgyadG8ceECAhhIrQarrj
+        3bidHY/GTgSK9r+jpUpaJLja78djn8ARHgPlxXK72a6X6+12szhq7Id72d2V8b58/nbzQb98gkOp
+        cuRIFR4fC+X3d3AYJVKChxTKgd8OxRYbWYycGQ7y8EidwaPbB7vuZCyJjCXDoasUjCL8S61Dtxfu
+        SOG/n5BkKFUeFD4fUnLoObPu20pBJcNDTQoccjA+UvufLedIP+Ebh5FUw0DwJ1RJBI+gymoh20wj
+        2SjPpF85sr3Rqz4cpbLRVSdJ6jUcKvUHDenM81zFeXgeTNMPB/2lRuMMM1Atlf8k9qVt1rer3WrV
+        3DZwOJw5SpWxWGvyRFnnR7ybQc4/usxvHEwspBfhbun+NreRLHDSObUL3Z7iRdxM/wh9rb/c8coy
+        Tb8BAAD//wMAqVt3JyMCAAA=
+    headers:
+      Access-Control-Allow-Origin:
+      - '*'
+      CF-RAY:
+      - 9402cb503aec46c0-BOM
+      Connection:
+      - keep-alive
+      Content-Encoding:
+      - gzip
+      Content-Type:
+      - application/json
+      Date:
+      - Thu, 15 May 2025 12:56:14 GMT
+      Server:
+      - cloudflare
+      Transfer-Encoding:
+      - chunked
+      Vary:
+      - Accept-Encoding
+      x-clerk-auth-message:
+      - Invalid JWT form. A JWT consists of three parts separated by dots. (reason=token-invalid,
+        token-carrier=header)
+      x-clerk-auth-reason:
+      - token-invalid
+      x-clerk-auth-status:
+      - signed-out
+    status:
+      code: 200
+      message: OK
+- request:
+    body: '{"model": "openai/gpt-4o-mini", "messages": [{"role": "system", "content":
+      "You are Information Agent. You have access to specific knowledge sources.\nYour
+      personal goal is: Provide information based on knowledge sources\nTo give my
+      best complete final answer to the task respond using the exact following format:\n\nThought:
+      I now can give a great answer\nFinal Answer: Your final answer must be the great
+      and the most complete as possible, it must be outcome described.\n\nI MUST use
+      these formats, my job depends on it!"}, {"role": "user", "content": "\nCurrent
+      Task: What is Vidit''s favorite color?\n\nThis is the expected criteria for
+      your final answer: Vidit''s favorclearite color.\nyou MUST return the actual
+      complete content as the final answer, not a summary.\n\nBegin! This is VERY
+      important to you, use the tools available and give your best Final Answer, your
+      job depends on it!\n\nThought:"}], "stream": false, "stop": ["\nObservation:"]}'
+    headers:
+      accept:
+      - '*/*'
+      accept-encoding:
+      - gzip, deflate
+      connection:
+      - keep-alive
+      content-length:
+      - '951'
+      content-type:
+      - application/json
+      host:
+      - openrouter.ai
+      http-referer:
+      - https://litellm.ai
+      user-agent:
+      - litellm/1.68.0
+      x-title:
+      - liteLLM
+    method: POST
+    uri: https://openrouter.ai/api/v1/chat/completions
+  response:
+    body:
+      string: !!binary |
+        H4sIAAAAAAAAAwAAAP//4lKAAS4AAAAA///iQjABAAAA//90kE9rG0EMxb/K8C69jNON7WJ7boFS
+        CD2ENm2g/1jGs/Ja7aw0zIydBuPvXjbBcQrtUU9P0u/pAO7g0JNMLhfzxexytli8mdy8r7c6/3Lb
+        v13eff00088fPj7AImXdc0cZDjeJ5OoaFoN2FOGgicTz6z7VyVwnAwvDQtc/KVQ4hK2vF0GHFKmy
+        CixCJl+pgzuftQhb5UAF7tsBUfuUdV3gZBejxYaFy7bN5IsKHErVBAvxlffU/qfL0tFvuMZioFJ8
+        T3AHZI0EB18Kl+qljjQqlWQkvTai9yZ4MT3vyXjTj6DGS7mnbMx3ecfio7l6rJ25447rq2I2fq+Z
+        K5mgUbPhYtZxRxewyLTZFR9PMZ4IWfon4Xj8YVEeSqVhzNBTTpkfQTapbWar6XI6bVYNLHYn/JR1
+        SLWt+oukjP9rRv7Ta8/6yqJq9fGsLFf27+m2o+o5lnFt8GFL3bO5Of5j60v/c5AXI8fjHwAAAP//
+        AwDEkP8dZgIAAA==
+    headers:
+      Access-Control-Allow-Origin:
+      - '*'
+      CF-RAY:
+      - 9402cb55c9fe46c0-BOM
+      Connection:
+      - keep-alive
+      Content-Encoding:
+      - gzip
+      Content-Type:
+      - application/json
+      Date:
+      - Thu, 15 May 2025 12:56:15 GMT
+      Server:
+      - cloudflare
+      Transfer-Encoding:
+      - chunked
+      Vary:
+      - Accept-Encoding
+      x-clerk-auth-message:
+      - Invalid JWT form. A JWT consists of three parts separated by dots. (reason=token-invalid,
+        token-carrier=header)
+      x-clerk-auth-reason:
+      - token-invalid
+      x-clerk-auth-status:
+      - signed-out
+    status:
+      code: 200
+      message: OK
+version: 1

tests/cassettes/test_agent_with_knowledge_with_no_crewai_knowledge.yaml

@@ -0,0 +1,151 @@
+interactions:
+- request:
+    body: '{"model": "openai/gpt-4o-mini", "messages": [{"role": "system", "content":
+      "Your goal is to rewrite the user query so that it is optimized for retrieval
+      from a vector database. Consider how the query will be used to find relevant
+      documents, and aim to make it more specific and context-aware. \n\n Do not include
+      any other text than the rewritten query, especially any preamble or postamble
+      and only add expected output format if its relevant to the rewritten query.
+      \n\n Focus on the key words of the intended task and to retrieve the most relevant
+      information. \n\n There will be some extra context provided that might need
+      to be removed such as expected_output formats structured_outputs and other instructions."},
+      {"role": "user", "content": "The original query is: What is Vidit''s favorite
+      color?\n\nThis is the expected criteria for your final answer: Vidit''s favorclearite
+      color.\nyou MUST return the actual complete content as the final answer, not
+      a summary.."}], "stream": false, "stop": ["\nObservation:"]}'
+    headers:
+      accept:
+      - '*/*'
+      accept-encoding:
+      - gzip, deflate
+      connection:
+      - keep-alive
+      content-length:
+      - '1017'
+      content-type:
+      - application/json
+      host:
+      - openrouter.ai
+      http-referer:
+      - https://litellm.ai
+      user-agent:
+      - litellm/1.68.0
+      x-title:
+      - liteLLM
+    method: POST
+    uri: https://openrouter.ai/api/v1/chat/completions
+  response:
+    body:
+      string: !!binary |
+        H4sIAAAAAAAAAwAAAP//4lKAAS4AAAAA//90kE1vE0EMhv9K9V64TGCbNGQ7N46gIg6IXhBaTWed
+        Xbez49HYiaii/e9oqRKKBFf7/XjsE7iHx0B5db272W2uN++b3ep585k+jcmo/XqnYXvX5m/3cChV
+        jtxThceXQvnDRzhM0lOChxTKgd8NxVY3spo4Mxzk4ZGiwSOOwd5GmUoiY8lwiJWCUQ9/qW0d4igc
+        SeG/n5BkKFUeFD4fUnLYc2Ydu0pBJcNDTQoccjA+UvefLeeefsI3DhOphoHgT6iSCB5BldVCtoVG
+        slFeSO+5Z3ujV/twlMpGV1GSVDhU2h80pDPOSxPn4WUwzz8c9FmNpoVloFoq/w7cl67Z3K7b9bq5
+        beBwOGOUKlOxzuSJsi5/2C4c5xdd5lsHEwvpj7Bt3N/mricLnHRJjSGO1F/EzfyP0Nf6yx2vLPP8
+        CwAA//8DAOHu/cIiAgAA
+    headers:
+      Access-Control-Allow-Origin:
+      - '*'
+      CF-RAY:
+      - 9402c73df9d8859c-BOM
+      Connection:
+      - keep-alive
+      Content-Encoding:
+      - gzip
+      Content-Type:
+      - application/json
+      Date:
+      - Thu, 15 May 2025 12:53:27 GMT
+      Server:
+      - cloudflare
+      Transfer-Encoding:
+      - chunked
+      Vary:
+      - Accept-Encoding
+      x-clerk-auth-message:
+      - Invalid JWT form. A JWT consists of three parts separated by dots. (reason=token-invalid,
+        token-carrier=header)
+      x-clerk-auth-reason:
+      - token-invalid
+      x-clerk-auth-status:
+      - signed-out
+    status:
+      code: 200
+      message: OK
+- request:
+    body: '{"model": "openai/gpt-4o-mini", "messages": [{"role": "system", "content":
+      "You are Information Agent. You have access to specific knowledge sources.\nYour
+      personal goal is: Provide information based on knowledge sources\nTo give my
+      best complete final answer to the task respond using the exact following format:\n\nThought:
+      I now can give a great answer\nFinal Answer: Your final answer must be the great
+      and the most complete as possible, it must be outcome described.\n\nI MUST use
+      these formats, my job depends on it!"}, {"role": "user", "content": "\nCurrent
+      Task: What is Vidit''s favorite color?\n\nThis is the expected criteria for
+      your final answer: Vidit''s favorclearite color.\nyou MUST return the actual
+      complete content as the final answer, not a summary.\n\nBegin! This is VERY
+      important to you, use the tools available and give your best Final Answer, your
+      job depends on it!\n\nThought:"}], "stream": false, "stop": ["\nObservation:"]}'
+    headers:
+      accept:
+      - '*/*'
+      accept-encoding:
+      - gzip, deflate
+      connection:
+      - keep-alive
+      content-length:
+      - '951'
+      content-type:
+      - application/json
+      host:
+      - openrouter.ai
+      http-referer:
+      - https://litellm.ai
+      user-agent:
+      - litellm/1.68.0
+      x-title:
+      - liteLLM
+    method: POST
+    uri: https://openrouter.ai/api/v1/chat/completions
+  response:
+    body:
+      string: !!binary |
+        H4sIAAAAAAAAAwAAAP//4lKAAS4AAAAA///iQjABAAAA//90kUGPEzEMhf+K5QuXdJmlpbvkthIg
+        emFXQoIDoMpNPFNDJo6STLul6n9H09KyIDjmxc9+/rxH8Wix4zi5vpndTK+n8+Z2wo9vXj28fHff
+        vW4+PNT5j1l6/wkNpqwb8ZzR4n3ieLdAg716DmhRE0eS512qk5lOeomCBnX1jV1Fi25N9cppnwJX
+        0YgGXWaq7NH+HmvQrVUcF7Sf9xi0S1lXBW0cQjDYSpSyXmamohEtlqoJDUaqsuHlf34len5E2xjs
+        uRTqGO0eswZGi1SKlEqxjmk0Vo5j0gVE3YKjCJ1sGAi6MShQLFvOAF/iW4kU4O74tvBRvNRnBVra
+        aJbK4DRoBikQtcJWPIcdeHVDz7GyB4mQhlUQF3ZAG5JAq8BQdMiOi4GisBiHj+ZftIHA87hePeY5
+        5cjcUfYSO1hLgZLYSSvurxRXaDBzOxQKZ4gnPhK7k3A4fDVYdqVyPxLsOKcsRwxtWvoVOZo3vm3Q
+        4HCGl7L2qS6rfudYxus1I73zYS/69NZg1UrhorwYD/yHe+m5koQytnXk1uwvxc3hH12f1l8WeWI5
+        HH4CAAD//wMAhZKqO+QCAAA=
+    headers:
+      Access-Control-Allow-Origin:
+      - '*'
+      CF-RAY:
+      - 9402c7459f3f859c-BOM
+      Connection:
+      - keep-alive
+      Content-Encoding:
+      - gzip
+      Content-Type:
+      - application/json
+      Date:
+      - Thu, 15 May 2025 12:53:28 GMT
+      Server:
+      - cloudflare
+      Transfer-Encoding:
+      - chunked
+      Vary:
+      - Accept-Encoding
+      x-clerk-auth-message:
+      - Invalid JWT form. A JWT consists of three parts separated by dots. (reason=token-invalid,
+        token-carrier=header)
+      x-clerk-auth-reason:
+      - token-invalid
+      x-clerk-auth-status:
+      - signed-out
+    status:
+      code: 200
+      message: OK
+version: 1

tests/cassettes/test_agent_with_only_crewai_knowledge.yaml

@@ -0,0 +1,150 @@
+interactions:
+- request:
+    body: '{"model": "openai/gpt-4o-mini", "messages": [{"role": "system", "content":
+      "Your goal is to rewrite the user query so that it is optimized for retrieval
+      from a vector database. Consider how the query will be used to find relevant
+      documents, and aim to make it more specific and context-aware. \n\n Do not include
+      any other text than the rewritten query, especially any preamble or postamble
+      and only add expected output format if its relevant to the rewritten query.
+      \n\n Focus on the key words of the intended task and to retrieve the most relevant
+      information. \n\n There will be some extra context provided that might need
+      to be removed such as expected_output formats structured_outputs and other instructions."},
+      {"role": "user", "content": "The original query is: What is Vidit''s favorite
+      color?\n\nThis is the expected criteria for your final answer: Vidit''s favorclearite
+      color.\nyou MUST return the actual complete content as the final answer, not
+      a summary.."}], "stream": false, "stop": ["\nObservation:"]}'
+    headers:
+      accept:
+      - '*/*'
+      accept-encoding:
+      - gzip, deflate
+      connection:
+      - keep-alive
+      content-length:
+      - '1017'
+      content-type:
+      - application/json
+      host:
+      - openrouter.ai
+      http-referer:
+      - https://litellm.ai
+      user-agent:
+      - litellm/1.68.0
+      x-title:
+      - liteLLM
+    method: POST
+    uri: https://openrouter.ai/api/v1/chat/completions
+  response:
+    body:
+      string: !!binary |
+        H4sIAAAAAAAAAwAAAP//4lKAAS4AAAAA//90kE1PIzEMhv8Kei97Sdnplwq5gTgAF8ShcFitRmnG
+        nTFk4ihxq11V899Xs6gFJLja78djH8ANLFqKk+lqsZpP56vpYqJhublfP1eP65v1i79Lt9fdMwxS
+        lj03lGHxkChe3cGgl4YCLCRRdPyzTTpZyKTnyDCQzQt5hYXvnJ576VMgZYkw8JmcUgP7XmvgO2FP
+        BfbXAUHalGVTYOMuBIMtRy5dnckVibAoKgkG0Snvqf5my7GhP7CVQU+luJZgD8gSCBauFC7qoo40
+        EpXiSPrEDeuPcrZ1e8msdOYlSIZBpu2uuHDEeWvi2L4NhuG3QflblPqRpaWcMv8P3Ka6ml/OLmaz
+        6rKCwe6IkbL0SWuVV4pl/MNy5Di+6DRfGqioC+/Ci8p8NtcNqeNQxlTvfEfNSVwNX4R+1J/u+GAZ
+        hn8AAAD//wMAIwJ79CICAAA=
+    headers:
+      Access-Control-Allow-Origin:
+      - '*'
+      CF-RAY:
+      - 9402c9db99ec4722-BOM
+      Connection:
+      - keep-alive
+      Content-Encoding:
+      - gzip
+      Content-Type:
+      - application/json
+      Date:
+      - Thu, 15 May 2025 12:55:14 GMT
+      Server:
+      - cloudflare
+      Transfer-Encoding:
+      - chunked
+      Vary:
+      - Accept-Encoding
+      x-clerk-auth-message:
+      - Invalid JWT form. A JWT consists of three parts separated by dots. (reason=token-invalid,
+        token-carrier=header)
+      x-clerk-auth-reason:
+      - token-invalid
+      x-clerk-auth-status:
+      - signed-out
+    status:
+      code: 200
+      message: OK
+- request:
+    body: '{"model": "openai/gpt-4o-mini", "messages": [{"role": "system", "content":
+      "You are Information Agent. You have access to specific knowledge sources.\nYour
+      personal goal is: Provide information based on knowledge sources\nTo give my
+      best complete final answer to the task respond using the exact following format:\n\nThought:
+      I now can give a great answer\nFinal Answer: Your final answer must be the great
+      and the most complete as possible, it must be outcome described.\n\nI MUST use
+      these formats, my job depends on it!"}, {"role": "user", "content": "\nCurrent
+      Task: What is Vidit''s favorite color?\n\nThis is the expected criteria for
+      your final answer: Vidit''s favorclearite color.\nyou MUST return the actual
+      complete content as the final answer, not a summary.\n\nBegin! This is VERY
+      important to you, use the tools available and give your best Final Answer, your
+      job depends on it!\n\nThought:"}], "stream": false, "stop": ["\nObservation:"]}'
+    headers:
+      accept:
+      - '*/*'
+      accept-encoding:
+      - gzip, deflate
+      connection:
+      - keep-alive
+      content-length:
+      - '951'
+      content-type:
+      - application/json
+      host:
+      - openrouter.ai
+      http-referer:
+      - https://litellm.ai
+      user-agent:
+      - litellm/1.68.0
+      x-title:
+      - liteLLM
+    method: POST
+    uri: https://openrouter.ai/api/v1/chat/completions
+  response:
+    body:
+      string: !!binary |
+        H4sIAAAAAAAAAwAAAP//4lKAAS4AAAAA///iQjABAAAA//90kN1qGzEQRl9FfNdyul4nday73ARy
+        VUpLE2jLIu+O15NoZ4QkOy1moa/R1+uTlE1wnEB7qU/zc84cwB0cepLZfHm+XMwXy/nF7II/3d7V
+        H+tOPvsS3le3d+keFjHpnjtKcPgQSa5uYDFoRwEOGkk8v+tjmZ3rbGBhWOj6ntoCh3bry1mrQwxU
+        WAUWbSJfqIM7rbVot8otZbivBwTtY9J1hpNdCBYbFs7bJpHPKnDIRSMsxBfeU/OfX5aOfsBVFgPl
+        7HuCOyBpIDj4nDkXL2WiUSkkE+mNEX00rRfT856MN/0EarzkR0rGfJNrFh/M1dPbmS/ccfnz63c2
+        G7/XxIVMq0GT4WzWYUdnsEi02WUfjiLPjCz9czCO3y3yz1xomCx6SjHxE8omNtViVV/WdbWqYLE7
+        CsSkQyxN0QeSPF2wmgyOxz3lK4uixYdTcrmyb7ubjornkKexrW+31L0UV+M/pr6ufxF51TKOfwEA
+        AP//AwBybekMaAIAAA==
+    headers:
+      Access-Control-Allow-Origin:
+      - '*'
+      CF-RAY:
+      - 9402c9e1b94a4722-BOM
+      Connection:
+      - keep-alive
+      Content-Encoding:
+      - gzip
+      Content-Type:
+      - application/json
+      Date:
+      - Thu, 15 May 2025 12:55:15 GMT
+      Server:
+      - cloudflare
+      Transfer-Encoding:
+      - chunked
+      Vary:
+      - Accept-Encoding
+      x-clerk-auth-message:
+      - Invalid JWT form. A JWT consists of three parts separated by dots. (reason=token-invalid,
+        token-carrier=header)
+      x-clerk-auth-reason:
+      - token-invalid
+      x-clerk-auth-status:
+      - signed-out
+    status:
+      code: 200
+      message: OK
+version: 1

tests/evaluation/metrics/test_goal_metrics.py

@@ -1,8 +1,8 @@
 from unittest.mock import patch, MagicMock
 from tests.evaluation.metrics.base_evaluation_metrics_test import BaseEvaluationMetricsTest
 
-from crewai.experimental.evaluation.base_evaluator import EvaluationScore
-from crewai.experimental.evaluation.metrics.goal_metrics import GoalAlignmentEvaluator
+from crewai.evaluation.base_evaluator import EvaluationScore
+from crewai.evaluation.metrics.goal_metrics import GoalAlignmentEvaluator
 from crewai.utilities.llm_utils import LLM
 
 

tests/evaluation/metrics/test_reasoning_metrics.py

@@ -3,12 +3,12 @@ from unittest.mock import patch, MagicMock
 from typing import List, Dict, Any
 
 from crewai.tasks.task_output import TaskOutput
-from crewai.experimental.evaluation.metrics.reasoning_metrics import (
+from crewai.evaluation.metrics.reasoning_metrics import (
     ReasoningEfficiencyEvaluator,
 )
 from tests.evaluation.metrics.base_evaluation_metrics_test import BaseEvaluationMetricsTest
 from crewai.utilities.llm_utils import LLM
-from crewai.experimental.evaluation.base_evaluator import EvaluationScore
+from crewai.evaluation.base_evaluator import EvaluationScore
 
 class TestReasoningEfficiencyEvaluator(BaseEvaluationMetricsTest):
     @pytest.fixture

tests/evaluation/metrics/test_semantic_quality_metrics.py

@@ -1,7 +1,7 @@
 from unittest.mock import patch, MagicMock
 
-from crewai.experimental.evaluation.base_evaluator import EvaluationScore
-from crewai.experimental.evaluation.metrics.semantic_quality_metrics import SemanticQualityEvaluator
+from crewai.evaluation.base_evaluator import EvaluationScore
+from crewai.evaluation.metrics.semantic_quality_metrics import SemanticQualityEvaluator
 from tests.evaluation.metrics.base_evaluation_metrics_test import BaseEvaluationMetricsTest
 from crewai.utilities.llm_utils import LLM
 

tests/evaluation/metrics/test_tools_metrics.py

@@ -1,6 +1,6 @@
 from unittest.mock import patch, MagicMock
 
-from crewai.experimental.evaluation.metrics.tools_metrics import (
+from crewai.evaluation.metrics.tools_metrics import (
     ToolSelectionEvaluator,
     ParameterExtractionEvaluator,
     ToolInvocationEvaluator

tests/evaluation/test_agent_evaluator.py

@@ -3,9 +3,9 @@ import pytest
 from crewai.agent import Agent
 from crewai.task import Task
 from crewai.crew import Crew
-from crewai.experimental.evaluation.agent_evaluator import AgentEvaluator
-from crewai.experimental.evaluation.base_evaluator import AgentEvaluationResult
-from crewai.experimental.evaluation import (
+from crewai.evaluation.agent_evaluator import AgentEvaluator
+from crewai.evaluation.base_evaluator import AgentEvaluationResult
+from crewai.evaluation import (
     GoalAlignmentEvaluator,
     SemanticQualityEvaluator,
     ToolSelectionEvaluator,
@@ -14,7 +14,7 @@ from crewai.experimental.evaluation import (
     ReasoningEfficiencyEvaluator
 )
 
-from crewai.experimental.evaluation import create_default_evaluator
+from crewai.evaluation import create_default_evaluator
 class TestAgentEvaluator:
     @pytest.fixture
     def mock_crew(self):

tests/evaluation/test_experiment_result.py

@@ -1,111 +0,0 @@
-import pytest
-from unittest.mock import MagicMock, patch
-
-from crewai.experimental.evaluation.experiment.result import ExperimentResult, ExperimentResults
-
-
-class TestExperimentResult:
-    @pytest.fixture
-    def mock_results(self):
-        return [
-            ExperimentResult(
-                identifier="test-1",
-                inputs={"query": "What is the capital of France?"},
-                score=10,
-                expected_score=7,
-                passed=True
-            ),
-            ExperimentResult(
-                identifier="test-2",
-                inputs={"query": "Who wrote Hamlet?"},
-                score={"relevance": 9, "factuality": 8},
-                expected_score={"relevance": 7, "factuality": 7},
-                passed=True,
-                agent_evaluations={"agent1": {"metrics": {"goal_alignment": {"score": 9}}}}
-            ),
-            ExperimentResult(
-                identifier="test-3",
-                inputs={"query": "Any query"},
-                score={"relevance": 9, "factuality": 8},
-                expected_score={"relevance": 7, "factuality": 7},
-                passed=False,
-                agent_evaluations={"agent1": {"metrics": {"goal_alignment": {"score": 9}}}}
-            ),
-            ExperimentResult(
-                identifier="test-4",
-                inputs={"query": "Another query"},
-                score={"relevance": 9, "factuality": 8},
-                expected_score={"relevance": 7, "factuality": 7},
-                passed=True,
-                agent_evaluations={"agent1": {"metrics": {"goal_alignment": {"score": 9}}}}
-            ),
-            ExperimentResult(
-                identifier="test-6",
-                inputs={"query": "Yet another query"},
-                score={"relevance": 9, "factuality": 8},
-                expected_score={"relevance": 7, "factuality": 7},
-                passed=True,
-                agent_evaluations={"agent1": {"metrics": {"goal_alignment": {"score": 9}}}}
-            )
-        ]
-
-    @patch('os.path.exists', return_value=True)
-    @patch('os.path.getsize', return_value=1)
-    @patch('json.load')
-    @patch('builtins.open', new_callable=MagicMock)
-    def test_experiment_results_compare_with_baseline(self, mock_open, mock_json_load, mock_path_getsize, mock_path_exists, mock_results):
-        baseline_data = {
-            "timestamp": "2023-01-01T00:00:00+00:00",
-            "results": [
-                {
-                    "identifier": "test-1",
-                    "inputs": {"query": "What is the capital of France?"},
-                    "score": 7,
-                    "expected_score": 7,
-                    "passed": False
-                },
-                {
-                    "identifier": "test-2",
-                    "inputs": {"query": "Who wrote Hamlet?"},
-                    "score": {"relevance": 8, "factuality": 7},
-                    "expected_score": {"relevance": 7, "factuality": 7},
-                    "passed": True
-                },
-                {
-                    "identifier": "test-3",
-                    "inputs": {"query": "Any query"},
-                    "score": {"relevance": 8, "factuality": 7},
-                    "expected_score": {"relevance": 7, "factuality": 7},
-                    "passed": True
-                },
-                {
-                    "identifier": "test-4",
-                    "inputs": {"query": "Another query"},
-                    "score": {"relevance": 8, "factuality": 7},
-                    "expected_score": {"relevance": 7, "factuality": 7},
-                    "passed": True
-                },
-                {
-                    "identifier": "test-5",
-                    "inputs": {"query": "Another query"},
-                    "score": {"relevance": 8, "factuality": 7},
-                    "expected_score": {"relevance": 7, "factuality": 7},
-                    "passed": True
-                }
-            ]
-        }
-
-        mock_json_load.return_value = baseline_data
-
-        results = ExperimentResults(results=mock_results)
-        results.display = MagicMock()
-
-        comparison = results.compare_with_baseline(baseline_filepath="baseline.json")
-
-        assert "baseline_timestamp" in comparison
-        assert comparison["baseline_timestamp"] == "2023-01-01T00:00:00+00:00"
-        assert comparison["improved"] == ["test-1"]
-        assert comparison["regressed"] == ["test-3"]
-        assert comparison["unchanged"] == ["test-2", "test-4"]
-        assert comparison["new_tests"] == ["test-6"]
-        assert comparison["missing_tests"] == ["test-5"]

tests/evaluation/test_experiment_runner.py

@@ -1,197 +0,0 @@
-import pytest
-from unittest.mock import MagicMock, patch
-
-from crewai.crew import Crew
-from crewai.experimental.evaluation.experiment.runner import ExperimentRunner
-from crewai.experimental.evaluation.experiment.result import ExperimentResults
-from crewai.experimental.evaluation.evaluation_display import AgentAggregatedEvaluationResult
-from crewai.experimental.evaluation.base_evaluator import MetricCategory, EvaluationScore
-
-
-class TestExperimentRunner:
-    @pytest.fixture
-    def mock_crew(self):
-        return MagicMock(llm=Crew)
-
-    @pytest.fixture
-    def mock_evaluator_results(self):
-        agent_evaluation = AgentAggregatedEvaluationResult(
-            agent_id="Test Agent",
-            agent_role="Test Agent Role",
-            metrics={
-                MetricCategory.GOAL_ALIGNMENT: EvaluationScore(
-                    score=9,
-                    feedback="Test feedback for goal alignment",
-                    raw_response="Test raw response for goal alignment"
-                ),
-                MetricCategory.REASONING_EFFICIENCY: EvaluationScore(
-                    score=None,
-                    feedback="Reasoning efficiency not applicable",
-                    raw_response="Reasoning efficiency not applicable"
-                ),
-                MetricCategory.PARAMETER_EXTRACTION: EvaluationScore(
-                    score=7,
-                    feedback="Test parameter extraction explanation",
-                    raw_response="Test raw output"
-                ),
-                MetricCategory.TOOL_SELECTION: EvaluationScore(
-                    score=8,
-                    feedback="Test tool selection explanation",
-                    raw_response="Test raw output"
-                )
-            }
-        )
-
-        return {"Test Agent": agent_evaluation}
-
-    @patch('crewai.experimental.evaluation.experiment.runner.create_default_evaluator')
-    def test_run_success(self, mock_create_evaluator, mock_crew, mock_evaluator_results):
-        dataset = [
-            {
-                "identifier": "test-case-1",
-                "inputs": {"query": "Test query 1"},
-                "expected_score": 8
-            },
-            {
-                "identifier": "test-case-2",
-                "inputs": {"query": "Test query 2"},
-                "expected_score": {"goal_alignment": 7}
-            },
-            {
-                "inputs": {"query": "Test query 3"},
-                "expected_score": {"tool_selection": 9}
-            }
-        ]
-
-        mock_evaluator = MagicMock()
-        mock_evaluator.get_agent_evaluation.return_value = mock_evaluator_results
-        mock_evaluator.reset_iterations_results = MagicMock()
-        mock_create_evaluator.return_value = mock_evaluator
-
-        runner = ExperimentRunner(dataset=dataset)
-
-        results = runner.run(crew=mock_crew)
-
-        assert isinstance(results, ExperimentResults)
-        result_1, result_2, result_3 = results.results
-        assert len(results.results) == 3
-
-        assert result_1.identifier == "test-case-1"
-        assert result_1.inputs == {"query": "Test query 1"}
-        assert result_1.expected_score == 8
-        assert result_1.passed is True
-
-        assert result_2.identifier == "test-case-2"
-        assert result_2.inputs == {"query": "Test query 2"}
-        assert isinstance(result_2.expected_score, dict)
-        assert "goal_alignment" in result_2.expected_score
-        assert result_2.passed is True
-
-        assert result_3.identifier == "c2ed49e63aa9a83af3ca382794134fd5"
-        assert result_3.inputs == {"query": "Test query 3"}
-        assert isinstance(result_3.expected_score, dict)
-        assert "tool_selection" in result_3.expected_score
-        assert result_3.passed is False
-
-        assert mock_crew.kickoff.call_count == 3
-        mock_crew.kickoff.assert_any_call(inputs={"query": "Test query 1"})
-        mock_crew.kickoff.assert_any_call(inputs={"query": "Test query 2"})
-        mock_crew.kickoff.assert_any_call(inputs={"query": "Test query 3"})
-
-        assert mock_evaluator.reset_iterations_results.call_count == 3
-        assert mock_evaluator.get_agent_evaluation.call_count == 3
-
-
-    @patch('crewai.experimental.evaluation.experiment.runner.create_default_evaluator')
-    def test_run_success_with_unknown_metric(self, mock_create_evaluator, mock_crew, mock_evaluator_results):
-        dataset = [
-            {
-                "identifier": "test-case-2",
-                "inputs": {"query": "Test query 2"},
-                "expected_score": {"goal_alignment": 7, "unknown_metric": 8}
-            }
-        ]
-
-        mock_evaluator = MagicMock()
-        mock_evaluator.get_agent_evaluation.return_value = mock_evaluator_results
-        mock_evaluator.reset_iterations_results = MagicMock()
-        mock_create_evaluator.return_value = mock_evaluator
-
-        runner = ExperimentRunner(dataset=dataset)
-
-        results = runner.run(crew=mock_crew)
-
-        result, = results.results
-
-        assert result.identifier == "test-case-2"
-        assert result.inputs == {"query": "Test query 2"}
-        assert isinstance(result.expected_score, dict)
-        assert "goal_alignment" in result.expected_score.keys()
-        assert "unknown_metric" in result.expected_score.keys()
-        assert result.passed is True
-
-    @patch('crewai.experimental.evaluation.experiment.runner.create_default_evaluator')
-    def test_run_success_with_single_metric_evaluator_and_expected_specific_metric(self, mock_create_evaluator, mock_crew, mock_evaluator_results):
-        dataset = [
-            {
-                "identifier": "test-case-2",
-                "inputs": {"query": "Test query 2"},
-                "expected_score": {"goal_alignment": 7}
-            }
-        ]
-
-        mock_evaluator = MagicMock()
-        mock_create_evaluator["Test Agent"].metrics = {
-            MetricCategory.GOAL_ALIGNMENT: EvaluationScore(
-                    score=9,
-                    feedback="Test feedback for goal alignment",
-                    raw_response="Test raw response for goal alignment"
-                )
-        }
-        mock_evaluator.get_agent_evaluation.return_value = mock_evaluator_results
-        mock_evaluator.reset_iterations_results = MagicMock()
-        mock_create_evaluator.return_value = mock_evaluator
-
-        runner = ExperimentRunner(dataset=dataset)
-
-        results = runner.run(crew=mock_crew)
-        result, = results.results
-
-        assert result.identifier == "test-case-2"
-        assert result.inputs == {"query": "Test query 2"}
-        assert isinstance(result.expected_score, dict)
-        assert "goal_alignment" in result.expected_score.keys()
-        assert result.passed is True
-
-    @patch('crewai.experimental.evaluation.experiment.runner.create_default_evaluator')
-    def test_run_success_when_expected_metric_is_not_available(self, mock_create_evaluator, mock_crew, mock_evaluator_results):
-        dataset = [
-            {
-                "identifier": "test-case-2",
-                "inputs": {"query": "Test query 2"},
-                "expected_score": {"unknown_metric": 7}
-            }
-        ]
-
-        mock_evaluator = MagicMock()
-        mock_create_evaluator["Test Agent"].metrics = {
-            MetricCategory.GOAL_ALIGNMENT: EvaluationScore(
-                score=5,
-                feedback="Test feedback for goal alignment",
-                raw_response="Test raw response for goal alignment"
-            )
-        }
-        mock_evaluator.get_agent_evaluation.return_value = mock_evaluator_results
-        mock_evaluator.reset_iterations_results = MagicMock()
-        mock_create_evaluator.return_value = mock_evaluator
-
-        runner = ExperimentRunner(dataset=dataset)
-
-        results = runner.run(crew=mock_crew)
-        result, = results.results
-
-        assert result.identifier == "test-case-2"
-        assert result.inputs == {"query": "Test query 2"}
-        assert isinstance(result.expected_score, dict)
-        assert "unknown_metric" in result.expected_score.keys()
-        assert result.passed is False

tests/task_test.py

@@ -1133,6 +1133,119 @@ def test_output_file_validation():
         )
 
 
+def test_create_directory_true():
+    """Test that directories are created when create_directory=True."""
+    from pathlib import Path
+    
+    output_path = "test_create_dir/output.txt"
+    
+    task = Task(
+        description="Test task",
+        expected_output="Test output",
+        output_file=output_path,
+        create_directory=True,
+    )
+    
+    resolved_path = Path(output_path).expanduser().resolve()
+    resolved_dir = resolved_path.parent
+    
+    if resolved_path.exists():
+        resolved_path.unlink()
+    if resolved_dir.exists():
+        import shutil
+        shutil.rmtree(resolved_dir)
+    
+    assert not resolved_dir.exists()
+    
+    task._save_file("test content")
+    
+    assert resolved_dir.exists()
+    assert resolved_path.exists()
+    
+    if resolved_path.exists():
+        resolved_path.unlink()
+    if resolved_dir.exists():
+        import shutil
+        shutil.rmtree(resolved_dir)
+
+
+def test_create_directory_false():
+    """Test that directories are not created when create_directory=False."""
+    from pathlib import Path
+    
+    output_path = "nonexistent_test_dir/output.txt"
+    
+    task = Task(
+        description="Test task",
+        expected_output="Test output",
+        output_file=output_path,
+        create_directory=False,
+    )
+    
+    resolved_path = Path(output_path).expanduser().resolve()
+    resolved_dir = resolved_path.parent
+    
+    if resolved_dir.exists():
+        import shutil
+        shutil.rmtree(resolved_dir)
+    
+    assert not resolved_dir.exists()
+    
+    with pytest.raises(RuntimeError, match="Directory .* does not exist and create_directory is False"):
+        task._save_file("test content")
+
+
+def test_create_directory_default():
+    """Test that create_directory defaults to True for backward compatibility."""
+    task = Task(
+        description="Test task",
+        expected_output="Test output",
+        output_file="output.txt",
+    )
+    
+    assert task.create_directory is True
+
+
+def test_create_directory_with_existing_directory():
+    """Test that create_directory=False works when directory already exists."""
+    from pathlib import Path
+    
+    output_path = "existing_test_dir/output.txt"
+    
+    resolved_path = Path(output_path).expanduser().resolve()
+    resolved_dir = resolved_path.parent
+    resolved_dir.mkdir(parents=True, exist_ok=True)
+    
+    task = Task(
+        description="Test task",
+        expected_output="Test output",
+        output_file=output_path,
+        create_directory=False,
+    )
+    
+    task._save_file("test content")
+    assert resolved_path.exists()
+    
+    if resolved_path.exists():
+        resolved_path.unlink()
+    if resolved_dir.exists():
+        import shutil
+        shutil.rmtree(resolved_dir)
+
+
+def test_github_issue_3149_reproduction():
+    """Test that reproduces the exact issue from GitHub issue #3149."""
+    task = Task(
+        description="Test task for issue reproduction",
+        expected_output="Test output",
+        output_file="test_output.txt",
+        create_directory=True,
+    )
+    
+    assert task.create_directory is True
+    assert task.output_file == "test_output.txt"
+
+
 @pytest.mark.vcr(filter_headers=["authorization"])
 def test_task_execution_times():
     researcher = Agent(

tests/test_lite_agent.py

@@ -12,6 +12,8 @@ from crewai.tools import BaseTool
 from crewai.utilities.events import crewai_event_bus
 from crewai.utilities.events.agent_events import LiteAgentExecutionStartedEvent
 from crewai.utilities.events.tool_usage_events import ToolUsageStartedEvent
+from crewai.llms.base_llm import BaseLLM
+from unittest.mock import patch
 
 
 # A simple test tool
@@ -418,3 +420,76 @@ def test_agent_output_when_guardrail_returns_base_model():
     result = agent.kickoff(messages="Top 10 best players in the world?")
 
     assert result.pydantic == Player(name="Lionel Messi", country="Argentina")
+
+def test_lite_agent_with_custom_llm_and_guardrails():
+    """Test that CustomLLM (inheriting from BaseLLM) works with guardrails."""
+    class CustomLLM(BaseLLM):
+        def __init__(self, response: str = "Custom response"):
+            super().__init__(model="custom-model")
+            self.response = response
+            self.call_count = 0
+
+        def call(self, messages, tools=None, callbacks=None, available_functions=None, from_task=None, from_agent=None) -> str:
+            self.call_count += 1
+
+            if "valid" in str(messages) and "feedback" in str(messages):
+                return '{"valid": true, "feedback": null}'
+
+            if "Thought:" in str(messages):
+                return f"Thought: I will analyze soccer players\nFinal Answer: {self.response}"
+
+            return self.response
+
+        def supports_function_calling(self) -> bool:
+            return False
+
+        def supports_stop_words(self) -> bool:
+            return False
+
+        def get_context_window_size(self) -> int:
+            return 4096
+
+    custom_llm = CustomLLM(response="Brazilian soccer players are the best!")
+
+    agent = LiteAgent(
+        role="Sports Analyst",
+        goal="Analyze soccer players",
+        backstory="You analyze soccer players and their performance.",
+        llm=custom_llm,
+        guardrail="Only include Brazilian players"
+    )
+
+    result = agent.kickoff("Tell me about the best soccer players")
+
+    assert custom_llm.call_count > 0
+    assert "Brazilian" in result.raw
+
+    custom_llm2 = CustomLLM(response="Original response")
+
+    def test_guardrail(output):
+        return (True, "Modified by guardrail")
+
+    agent2 = LiteAgent(
+        role="Test Agent",
+        goal="Test goal",
+        backstory="Test backstory",
+        llm=custom_llm2,
+        guardrail=test_guardrail
+    )
+
+    result2 = agent2.kickoff("Test message")
+    assert result2.raw == "Modified by guardrail"
+
+
+@pytest.mark.vcr(filter_headers=["authorization"])
+def test_lite_agent_with_invalid_llm():
+    """Test that LiteAgent raises proper error when create_llm returns None."""
+    with patch('crewai.lite_agent.create_llm', return_value=None):
+        with pytest.raises(ValueError) as exc_info:
+            LiteAgent(
+                role="Test Agent",
+                goal="Test goal", 
+                backstory="Test backstory",
+                llm="invalid-model"
+            )
+        assert "Expected LLM instance of type BaseLLM" in str(exc_info.value)