mirror of
https://github.com/crewAIInc/crewAI.git
synced 2026-04-30 14:52:36 +00:00
Compare commits
7 Commits
gl/docs/ha
...
devin/1747
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
10b5082c0a | ||
|
|
bddfe1c780 | ||
|
|
c3dc839b12 | ||
|
|
270a473d5d | ||
|
|
98df434eb9 | ||
|
|
9973011be5 | ||
|
|
547e46b8cf |
@@ -21,7 +21,7 @@ Think of an agent as a specialized team member with specific skills, expertise,
|
||||
<Note type="info" title="Enterprise Enhancement: Visual Agent Builder">
|
||||
CrewAI Enterprise includes a Visual Agent Builder that simplifies agent creation and configuration without writing code. Design your agents visually and test them in real-time.
|
||||
|
||||

|
||||

|
||||
|
||||
The Visual Agent Builder enables:
|
||||
- Intuitive agent configuration with form-based interfaces
|
||||
|
||||
@@ -4,7 +4,7 @@ description: Learn how to use the CrewAI CLI to interact with CrewAI.
|
||||
icon: terminal
|
||||
---
|
||||
|
||||
## Overview
|
||||
# CrewAI CLI Documentation
|
||||
|
||||
The CrewAI CLI provides a set of commands to interact with CrewAI, allowing you to create, train, run, and manage crews & flows.
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: Exploring the dynamics of agent collaboration within the CrewAI fra
|
||||
icon: screen-users
|
||||
---
|
||||
|
||||
## Overview
|
||||
## Collaboration Fundamentals
|
||||
|
||||
Collaboration in CrewAI is fundamental, enabling agents to combine their skills, share information, and assist each other in task execution, embodying a truly cooperative ecosystem.
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: Understanding and utilizing crews in the crewAI framework with comp
|
||||
icon: people-group
|
||||
---
|
||||
|
||||
## Overview
|
||||
## What is a Crew?
|
||||
|
||||
A crew in crewAI represents a collaborative group of agents working together to achieve a set of tasks. Each crew defines the strategy for task execution, agent collaboration, and the overall workflow.
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: 'Tap into CrewAI events to build custom integrations and monitoring
|
||||
icon: spinner
|
||||
---
|
||||
|
||||
## Overview
|
||||
# Event Listeners
|
||||
|
||||
CrewAI provides a powerful event system that allows you to listen for and react to various events that occur during the execution of your Crew. This feature enables you to build custom integrations, monitoring solutions, logging systems, or any other functionality that needs to be triggered based on CrewAI's internal events.
|
||||
|
||||
@@ -21,7 +21,7 @@ When specific actions occur in CrewAI (like a Crew starting execution, an Agent
|
||||
<Note type="info" title="Enterprise Enhancement: Prompt Tracing">
|
||||
CrewAI Enterprise provides a built-in Prompt Tracing feature that leverages the event system to track, store, and visualize all prompts, completions, and associated metadata. This provides powerful debugging capabilities and transparency into your agent operations.
|
||||
|
||||

|
||||

|
||||
|
||||
With Prompt Tracing you can:
|
||||
- View the complete history of all prompts sent to your LLM
|
||||
@@ -224,15 +224,6 @@ CrewAI provides a wide range of events that you can listen for:
|
||||
- **ToolExecutionErrorEvent**: Emitted when a tool execution encounters an error
|
||||
- **ToolSelectionErrorEvent**: Emitted when there's an error selecting a tool
|
||||
|
||||
### Knowledge Events
|
||||
|
||||
- **KnowledgeRetrievalStartedEvent**: Emitted when a knowledge retrieval is started
|
||||
- **KnowledgeRetrievalCompletedEvent**: Emitted when a knowledge retrieval is completed
|
||||
- **KnowledgeQueryStartedEvent**: Emitted when a knowledge query is started
|
||||
- **KnowledgeQueryCompletedEvent**: Emitted when a knowledge query is completed
|
||||
- **KnowledgeQueryFailedEvent**: Emitted when a knowledge query fails
|
||||
- **KnowledgeSearchQueryFailedEvent**: Emitted when a knowledge search query fails
|
||||
|
||||
### Flow Events
|
||||
|
||||
- **FlowCreatedEvent**: Emitted when a Flow is created
|
||||
|
||||
@@ -4,7 +4,7 @@ description: Learn how to create and manage AI workflows using CrewAI Flows.
|
||||
icon: arrow-progress
|
||||
---
|
||||
|
||||
## Overview
|
||||
## Introduction
|
||||
|
||||
CrewAI Flows is a powerful feature designed to streamline the creation and management of AI workflows. Flows allow developers to combine and coordinate coding tasks and Crews efficiently, providing a robust framework for building sophisticated AI automations.
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: What is knowledge in CrewAI and how to use it.
|
||||
icon: book
|
||||
---
|
||||
|
||||
## Overview
|
||||
## What is Knowledge?
|
||||
|
||||
Knowledge in CrewAI is a powerful system that allows AI agents to access and utilize external information sources during their tasks.
|
||||
Think of it as giving your agents a reference library they can consult while working.
|
||||
@@ -36,15 +36,12 @@ CrewAI supports various types of knowledge sources out of the box:
|
||||
|
||||
## Supported Knowledge Parameters
|
||||
|
||||
<ParamField body="sources" type="List[BaseKnowledgeSource]" required="Yes">
|
||||
List of knowledge sources that provide content to be stored and queried. Can include PDF, CSV, Excel, JSON, text files, or string content.
|
||||
</ParamField>
|
||||
<ParamField body="collection_name" type="str">
|
||||
Name of the collection where the knowledge will be stored. Used to identify different sets of knowledge. Defaults to \"knowledge\" if not provided.
|
||||
</ParamField>
|
||||
<ParamField body="storage" type="Optional[KnowledgeStorage]">
|
||||
Custom storage configuration for managing how the knowledge is stored and retrieved. If not provided, a default storage will be created.
|
||||
</ParamField>
|
||||
| Parameter | Type | Required | Description |
|
||||
| :--------------------------- | :---------------------------------- | :------- | :---------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `sources` | **List[BaseKnowledgeSource]** | Yes | List of knowledge sources that provide content to be stored and queried. Can include PDF, CSV, Excel, JSON, text files, or string content. |
|
||||
| `collection_name` | **str** | No | Name of the collection where the knowledge will be stored. Used to identify different sets of knowledge. Defaults to "knowledge" if not provided. |
|
||||
| `storage` | **Optional[KnowledgeStorage]** | No | Custom storage configuration for managing how the knowledge is stored and retrieved. If not provided, a default storage will be created. |
|
||||
|
||||
|
||||
<Tip>
|
||||
Unlike retrieval from a vector database using a tool, agents preloaded with knowledge will not need a retrieval persona or task.
|
||||
@@ -435,46 +432,6 @@ Query rewriting happens transparently using a system prompt that instructs the L
|
||||
This mechanism is fully automatic and requires no configuration from users. The agent's LLM is used to perform the query rewriting, so using a more capable LLM can improve the quality of rewritten queries.
|
||||
</Tip>
|
||||
|
||||
## Knowledge Events
|
||||
|
||||
CrewAI emits events during the knowledge retrieval process that you can listen for using the event system. These events allow you to monitor, debug, and analyze how knowledge is being retrieved and used by your agents.
|
||||
|
||||
### Available Knowledge Events
|
||||
|
||||
- **KnowledgeRetrievalStartedEvent**: Emitted when an agent starts retrieving knowledge from sources
|
||||
- **KnowledgeRetrievalCompletedEvent**: Emitted when knowledge retrieval is completed, including the query used and the retrieved content
|
||||
- **KnowledgeQueryStartedEvent**: Emitted when a query to knowledge sources begins
|
||||
- **KnowledgeQueryCompletedEvent**: Emitted when a query completes successfully
|
||||
- **KnowledgeQueryFailedEvent**: Emitted when a query to knowledge sources fails
|
||||
- **KnowledgeSearchQueryFailedEvent**: Emitted when a search query fails
|
||||
|
||||
### Example: Monitoring Knowledge Retrieval
|
||||
|
||||
```python
|
||||
from crewai.utilities.events import (
|
||||
KnowledgeRetrievalStartedEvent,
|
||||
KnowledgeRetrievalCompletedEvent,
|
||||
)
|
||||
from crewai.utilities.events.base_event_listener import BaseEventListener
|
||||
|
||||
class KnowledgeMonitorListener(BaseEventListener):
|
||||
def setup_listeners(self, crewai_event_bus):
|
||||
@crewai_event_bus.on(KnowledgeRetrievalStartedEvent)
|
||||
def on_knowledge_retrieval_started(source, event):
|
||||
print(f"Agent '{event.agent.role}' started retrieving knowledge")
|
||||
|
||||
@crewai_event_bus.on(KnowledgeRetrievalCompletedEvent)
|
||||
def on_knowledge_retrieval_completed(source, event):
|
||||
print(f"Agent '{event.agent.role}' completed knowledge retrieval")
|
||||
print(f"Query: {event.query}")
|
||||
print(f"Retrieved {len(event.retrieved_knowledge)} knowledge chunks")
|
||||
|
||||
# Create an instance of your listener
|
||||
knowledge_monitor = KnowledgeMonitorListener()
|
||||
```
|
||||
|
||||
For more information on using events, see the [Event Listeners](https://docs.crewai.com/concepts/event-listener) documentation.
|
||||
|
||||
### Example
|
||||
|
||||
```python
|
||||
|
||||
@@ -4,10 +4,9 @@ description: 'A comprehensive guide to configuring and using Large Language Mode
|
||||
icon: 'microchip-ai'
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
CrewAI integrates with multiple LLM providers through LiteLLM, giving you the flexibility to choose the right model for your specific use case. This guide will help you understand how to configure and use different LLM providers in your CrewAI projects.
|
||||
|
||||
<Note>
|
||||
CrewAI integrates with multiple LLM providers through LiteLLM, giving you the flexibility to choose the right model for your specific use case. This guide will help you understand how to configure and use different LLM providers in your CrewAI projects.
|
||||
</Note>
|
||||
|
||||
## What are LLMs?
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: Leveraging memory systems in the CrewAI framework to enhance agent
|
||||
icon: database
|
||||
---
|
||||
|
||||
## Overview
|
||||
## Introduction to Memory Systems in CrewAI
|
||||
|
||||
The crewAI framework introduces a sophisticated memory system designed to significantly enhance the capabilities of AI agents.
|
||||
This system comprises `short-term memory`, `long-term memory`, `entity memory`, and `contextual memory`, each serving a unique purpose in aiding agents to remember,
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
---
|
||||
title: Planning
|
||||
description: Learn how to add planning to your CrewAI Crew and improve their performance.
|
||||
icon: ruler-combined
|
||||
icon: brain
|
||||
---
|
||||
|
||||
## Overview
|
||||
## Introduction
|
||||
|
||||
The planning feature in CrewAI allows you to add planning capability to your crew. When enabled, before each Crew iteration,
|
||||
all Crew information is sent to an AgentPlanner that will plan the tasks step by step, and this plan will be added to each task description.
|
||||
|
||||
@@ -4,8 +4,7 @@ description: Detailed guide on workflow management through processes in CrewAI,
|
||||
icon: bars-staggered
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
## Understanding Processes
|
||||
<Tip>
|
||||
Processes orchestrate the execution of tasks by agents, akin to project management in human teams.
|
||||
These processes ensure tasks are distributed and executed efficiently, in alignment with a predefined strategy.
|
||||
|
||||
@@ -1,14 +1,12 @@
|
||||
---
|
||||
title: Reasoning
|
||||
description: "Learn how to enable and use agent reasoning to improve task execution."
|
||||
icon: brain
|
||||
title: "Agent Reasoning"
|
||||
---
|
||||
|
||||
## Overview
|
||||
# Agent Reasoning
|
||||
|
||||
Agent reasoning is a feature that allows agents to reflect on a task and create a plan before execution. This helps agents approach tasks more methodically and ensures they're ready to perform the assigned work.
|
||||
|
||||
## Usage
|
||||
## How to Use Agent Reasoning
|
||||
|
||||
To enable reasoning for an agent, simply set `reasoning=True` when creating the agent:
|
||||
|
||||
@@ -37,13 +35,8 @@ This process helps the agent break down complex tasks into manageable steps and
|
||||
|
||||
## Configuration Options
|
||||
|
||||
<ParamField body="reasoning" type="bool" default="False">
|
||||
Enable or disable reasoning
|
||||
</ParamField>
|
||||
|
||||
<ParamField body="max_reasoning_attempts" type="int" default="None">
|
||||
Maximum number of attempts to refine the plan before proceeding with execution. If None (default), the agent will continue refining until it's ready.
|
||||
</ParamField>
|
||||
- `reasoning` (bool): Enable or disable reasoning (default: False)
|
||||
- `max_reasoning_attempts` (int, optional): Maximum number of attempts to refine the plan before proceeding with execution. If None (default), the agent will continue refining until it's ready.
|
||||
|
||||
## Example
|
||||
|
||||
@@ -116,7 +109,7 @@ Here's an example of what a reasoning plan might look like for a data analysis t
|
||||
Task: Analyze the provided sales data and identify key trends.
|
||||
|
||||
Reasoning Plan:
|
||||
I'll analyze the sales data to identify the top 3 trends.
|
||||
I'll analyze the sales data to identify the top 3 trends.
|
||||
|
||||
1. Understanding of the task:
|
||||
I need to analyze sales data to identify key trends that would be valuable for business decision-making.
|
||||
|
||||
@@ -4,7 +4,7 @@ description: Detailed guide on managing and creating tasks within the CrewAI fra
|
||||
icon: list-check
|
||||
---
|
||||
|
||||
## Overview
|
||||
## Overview of a Task
|
||||
|
||||
In the CrewAI framework, a `Task` is a specific assignment completed by an `Agent`.
|
||||
|
||||
@@ -15,7 +15,7 @@ Tasks within CrewAI can be collaborative, requiring multiple agents to work toge
|
||||
<Note type="info" title="Enterprise Enhancement: Visual Task Builder">
|
||||
CrewAI Enterprise includes a Visual Task Builder in Crew Studio that simplifies complex task creation and chaining. Design your task flows visually and test them in real-time without writing code.
|
||||
|
||||

|
||||

|
||||
|
||||
The Visual Task Builder enables:
|
||||
- Drag-and-drop task creation
|
||||
|
||||
@@ -4,7 +4,7 @@ description: Learn how to test your CrewAI Crew and evaluate their performance.
|
||||
icon: vial
|
||||
---
|
||||
|
||||
## Overview
|
||||
## Introduction
|
||||
|
||||
Testing is a crucial part of the development process, and it is essential to ensure that your crew is performing as expected. With crewAI, you can easily test your crew and evaluate its performance using the built-in testing capabilities.
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: Understanding and leveraging tools within the CrewAI framework for
|
||||
icon: screwdriver-wrench
|
||||
---
|
||||
|
||||
## Overview
|
||||
## Introduction
|
||||
|
||||
CrewAI tools empower agents with capabilities ranging from web searching and data analysis to collaboration and delegating tasks among coworkers.
|
||||
This documentation outlines how to create, integrate, and leverage these tools within the CrewAI framework, including a new focus on collaboration tools.
|
||||
@@ -18,6 +18,8 @@ enabling everything from simple searches to complex interactions and effective t
|
||||
<Note type="info" title="Enterprise Enhancement: Tools Repository">
|
||||
CrewAI Enterprise provides a comprehensive Tools Repository with pre-built integrations for common business systems and APIs. Deploy agents with enterprise tools in minutes instead of days.
|
||||
|
||||

|
||||
|
||||
The Enterprise Tools Repository includes:
|
||||
- Pre-built connectors for popular enterprise systems
|
||||
- Custom tool creation interface
|
||||
|
||||
@@ -4,7 +4,7 @@ description: Learn how to train your CrewAI agents by giving them feedback early
|
||||
icon: dumbbell
|
||||
---
|
||||
|
||||
## Overview
|
||||
## Introduction
|
||||
|
||||
The training feature in CrewAI allows you to train your AI agents using the command-line interface (CLI).
|
||||
By running the command `crewai train -n <n_iterations>`, you can specify the number of iterations for the training process.
|
||||
|
||||
@@ -74,7 +74,6 @@
|
||||
"concepts/collaboration",
|
||||
"concepts/training",
|
||||
"concepts/memory",
|
||||
"concepts/reasoning",
|
||||
"concepts/planning",
|
||||
"concepts/testing",
|
||||
"concepts/cli",
|
||||
@@ -144,7 +143,7 @@
|
||||
"group": "MCP Integration",
|
||||
"pages": [
|
||||
"mcp/crewai-mcp-integration"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"group": "Agent Monitoring & Observability",
|
||||
@@ -213,8 +212,7 @@
|
||||
"pages": [
|
||||
"enterprise/features/tool-repository",
|
||||
"enterprise/features/webhook-streaming",
|
||||
"enterprise/features/traces",
|
||||
"enterprise/features/hallucination-guardrail"
|
||||
"enterprise/features/traces"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -309,4 +307,4 @@
|
||||
"reddit": "https://www.reddit.com/r/crewAIInc/"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,245 +0,0 @@
|
||||
---
|
||||
title: Hallucination Guardrail
|
||||
description: "Prevent and detect AI hallucinations in your CrewAI tasks"
|
||||
icon: "shield-check"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
The Hallucination Guardrail is an enterprise feature that validates AI-generated content to ensure it's grounded in facts and doesn't contain hallucinations. It analyzes task outputs against reference context and provides detailed feedback when potentially hallucinated content is detected.
|
||||
|
||||
## What are Hallucinations?
|
||||
|
||||
AI hallucinations occur when language models generate content that appears plausible but is factually incorrect or not supported by the provided context. The Hallucination Guardrail helps prevent these issues by:
|
||||
|
||||
- Comparing outputs against reference context
|
||||
- Evaluating faithfulness to source material
|
||||
- Providing detailed feedback on problematic content
|
||||
- Supporting custom thresholds for validation strictness
|
||||
|
||||
## Basic Usage
|
||||
|
||||
### Setting Up the Guardrail
|
||||
|
||||
```python
|
||||
from crewai.tasks.hallucination_guardrail import HallucinationGuardrail
|
||||
from crewai import LLM
|
||||
|
||||
# Initialize the guardrail with reference context
|
||||
guardrail = HallucinationGuardrail(
|
||||
context="AI helps with various tasks including analysis and generation.",
|
||||
llm=LLM(model="gpt-4o-mini")
|
||||
)
|
||||
```
|
||||
|
||||
### Adding to Tasks
|
||||
|
||||
```python
|
||||
from crewai import Task
|
||||
|
||||
# Create your task with the guardrail
|
||||
task = Task(
|
||||
description="Write a summary about AI capabilities",
|
||||
expected_output="A factual summary based on the provided context",
|
||||
agent=my_agent,
|
||||
guardrail=guardrail # Add the guardrail to validate output
|
||||
)
|
||||
```
|
||||
|
||||
## Advanced Configuration
|
||||
|
||||
### Custom Threshold Validation
|
||||
|
||||
For stricter validation, you can set a custom faithfulness threshold (0-10 scale):
|
||||
|
||||
```python
|
||||
# Strict guardrail requiring high faithfulness score
|
||||
strict_guardrail = HallucinationGuardrail(
|
||||
context="Quantum computing uses qubits that exist in superposition states.",
|
||||
llm=LLM(model="gpt-4o-mini"),
|
||||
threshold=8.0 # Requires score >= 8 to pass validation
|
||||
)
|
||||
```
|
||||
|
||||
### Including Tool Response Context
|
||||
|
||||
When your task uses tools, you can include tool responses for more accurate validation:
|
||||
|
||||
```python
|
||||
# Guardrail with tool response context
|
||||
weather_guardrail = HallucinationGuardrail(
|
||||
context="Current weather information for the requested location",
|
||||
llm=LLM(model="gpt-4o-mini"),
|
||||
tool_response="Weather API returned: Temperature 22°C, Humidity 65%, Clear skies"
|
||||
)
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
### Validation Process
|
||||
|
||||
1. **Context Analysis**: The guardrail compares task output against the provided reference context
|
||||
2. **Faithfulness Scoring**: Uses an internal evaluator to assign a faithfulness score (0-10)
|
||||
3. **Verdict Determination**: Determines if content is faithful or contains hallucinations
|
||||
4. **Threshold Checking**: If a custom threshold is set, validates against that score
|
||||
5. **Feedback Generation**: Provides detailed reasons when validation fails
|
||||
|
||||
### Validation Logic
|
||||
|
||||
- **Default Mode**: Uses verdict-based validation (FAITHFUL vs HALLUCINATED)
|
||||
- **Threshold Mode**: Requires faithfulness score to meet or exceed the specified threshold
|
||||
- **Error Handling**: Gracefully handles evaluation errors and provides informative feedback
|
||||
|
||||
## Guardrail Results
|
||||
|
||||
The guardrail returns structured results indicating validation status:
|
||||
|
||||
```python
|
||||
# Example of guardrail result structure
|
||||
{
|
||||
"valid": False,
|
||||
"feedback": "Content appears to be hallucinated (score: 4.2/10, verdict: HALLUCINATED). The output contains information not supported by the provided context."
|
||||
}
|
||||
```
|
||||
|
||||
### Result Properties
|
||||
|
||||
- **valid**: Boolean indicating whether the output passed validation
|
||||
- **feedback**: Detailed explanation when validation fails, including:
|
||||
- Faithfulness score
|
||||
- Verdict classification
|
||||
- Specific reasons for failure
|
||||
|
||||
## Integration with Task System
|
||||
|
||||
### Automatic Validation
|
||||
|
||||
When a guardrail is added to a task, it automatically validates the output before the task is marked as complete:
|
||||
|
||||
```python
|
||||
# Task output validation flow
|
||||
task_output = agent.execute_task(task)
|
||||
validation_result = guardrail(task_output)
|
||||
|
||||
if validation_result.valid:
|
||||
# Task completes successfully
|
||||
return task_output
|
||||
else:
|
||||
# Task fails with validation feedback
|
||||
raise ValidationError(validation_result.feedback)
|
||||
```
|
||||
|
||||
### Event Tracking
|
||||
|
||||
The guardrail integrates with CrewAI's event system to provide observability:
|
||||
|
||||
- **Validation Started**: When guardrail evaluation begins
|
||||
- **Validation Completed**: When evaluation finishes with results
|
||||
- **Validation Failed**: When technical errors occur during evaluation
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Context Guidelines
|
||||
|
||||
<Steps>
|
||||
<Step title="Provide Comprehensive Context">
|
||||
Include all relevant factual information that the AI should base its output on:
|
||||
|
||||
```python
|
||||
context = """
|
||||
Company XYZ was founded in 2020 and specializes in renewable energy solutions.
|
||||
They have 150 employees and generated $50M revenue in 2023.
|
||||
Their main products include solar panels and wind turbines.
|
||||
"""
|
||||
```
|
||||
</Step>
|
||||
|
||||
<Step title="Keep Context Relevant">
|
||||
Only include information directly related to the task to avoid confusion:
|
||||
|
||||
```python
|
||||
# Good: Focused context
|
||||
context = "The current weather in New York is 18°C with light rain."
|
||||
|
||||
# Avoid: Unrelated information
|
||||
context = "The weather is 18°C. The city has 8 million people. Traffic is heavy."
|
||||
```
|
||||
</Step>
|
||||
|
||||
<Step title="Update Context Regularly">
|
||||
Ensure your reference context reflects current, accurate information.
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
### Threshold Selection
|
||||
|
||||
<Steps>
|
||||
<Step title="Start with Default Validation">
|
||||
Begin without custom thresholds to understand baseline performance.
|
||||
</Step>
|
||||
|
||||
<Step title="Adjust Based on Requirements">
|
||||
- **High-stakes content**: Use threshold 8-10 for maximum accuracy
|
||||
- **General content**: Use threshold 6-7 for balanced validation
|
||||
- **Creative content**: Use threshold 4-5 or default verdict-based validation
|
||||
</Step>
|
||||
|
||||
<Step title="Monitor and Iterate">
|
||||
Track validation results and adjust thresholds based on false positives/negatives.
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Impact on Execution Time
|
||||
|
||||
- **Validation Overhead**: Each guardrail adds ~1-3 seconds per task
|
||||
- **LLM Efficiency**: Choose efficient models for evaluation (e.g., gpt-4o-mini)
|
||||
|
||||
### Cost Optimization
|
||||
|
||||
- **Model Selection**: Use smaller, efficient models for guardrail evaluation
|
||||
- **Context Size**: Keep reference context concise but comprehensive
|
||||
- **Caching**: Consider caching validation results for repeated content
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
<Accordion title="Validation Always Fails">
|
||||
**Possible Causes:**
|
||||
- Context is too restrictive or unrelated to task output
|
||||
- Threshold is set too high for the content type
|
||||
- Reference context contains outdated information
|
||||
|
||||
**Solutions:**
|
||||
- Review and update context to match task requirements
|
||||
- Lower threshold or use default verdict-based validation
|
||||
- Ensure context is current and accurate
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="False Positives (Valid Content Marked Invalid)">
|
||||
**Possible Causes:**
|
||||
- Threshold too high for creative or interpretive tasks
|
||||
- Context doesn't cover all valid aspects of the output
|
||||
- Evaluation model being overly conservative
|
||||
|
||||
**Solutions:**
|
||||
- Lower threshold or use default validation
|
||||
- Expand context to include broader acceptable content
|
||||
- Test with different evaluation models
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Evaluation Errors">
|
||||
**Possible Causes:**
|
||||
- Network connectivity issues
|
||||
- LLM model unavailable or rate limited
|
||||
- Malformed task output or context
|
||||
|
||||
**Solutions:**
|
||||
- Check network connectivity and LLM service status
|
||||
- Implement retry logic for transient failures
|
||||
- Validate task output format before guardrail evaluation
|
||||
</Accordion>
|
||||
|
||||
<Card title="Need Help?" icon="headset" href="mailto:support@crewai.com">
|
||||
Contact our support team for assistance with hallucination guardrail configuration or troubleshooting.
|
||||
</Card>
|
||||
@@ -12,8 +12,7 @@ The before kickoff hook is executed before the crew starts its tasks. It receive
|
||||
Here's an example of defining a before kickoff function in your `crew.py`:
|
||||
|
||||
```python
|
||||
from crewai import CrewBase
|
||||
from crewai.project import before_kickoff
|
||||
from crewai import CrewBase, before_kickoff
|
||||
|
||||
@CrewBase
|
||||
class MyCrew:
|
||||
@@ -35,8 +34,7 @@ The after kickoff hook is executed after the crew has completed its tasks. It re
|
||||
Here's how you can define an after kickoff function in your `crew.py`:
|
||||
|
||||
```python
|
||||
from crewai import CrewBase
|
||||
from crewai.project import after_kickoff
|
||||
from crewai import CrewBase, after_kickoff
|
||||
|
||||
@CrewBase
|
||||
class MyCrew:
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "crewai"
|
||||
version = "0.121.0"
|
||||
version = "0.120.1"
|
||||
description = "Cutting-edge framework for orchestrating role-playing, autonomous AI agents. By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, tackling complex tasks."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10,<3.13"
|
||||
|
||||
@@ -17,7 +17,7 @@ warnings.filterwarnings(
|
||||
category=UserWarning,
|
||||
module="pydantic.main",
|
||||
)
|
||||
__version__ = "0.121.0"
|
||||
__version__ = "0.120.1"
|
||||
__all__ = [
|
||||
"Agent",
|
||||
"Crew",
|
||||
|
||||
@@ -5,7 +5,7 @@ description = "{{name}} using crewAI"
|
||||
authors = [{ name = "Your Name", email = "you@example.com" }]
|
||||
requires-python = ">=3.10,<3.13"
|
||||
dependencies = [
|
||||
"crewai[tools]>=0.121.0,<1.0.0"
|
||||
"crewai[tools]>=0.120.1,<1.0.0"
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
|
||||
@@ -5,7 +5,7 @@ description = "{{name}} using crewAI"
|
||||
authors = [{ name = "Your Name", email = "you@example.com" }]
|
||||
requires-python = ">=3.10,<3.13"
|
||||
dependencies = [
|
||||
"crewai[tools]>=0.121.0,<1.0.0",
|
||||
"crewai[tools]>=0.120.1,<1.0.0",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
|
||||
@@ -5,7 +5,7 @@ description = "Power up your crews with {{folder_name}}"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10,<3.13"
|
||||
dependencies = [
|
||||
"crewai[tools]>=0.121.0"
|
||||
"crewai[tools]>=0.120.1"
|
||||
]
|
||||
|
||||
[tool.crewai]
|
||||
|
||||
@@ -161,7 +161,7 @@ def tree_find_and_replace(directory, find, replace):
|
||||
for filename in files:
|
||||
filepath = os.path.join(path, filename)
|
||||
|
||||
with open(filepath, "r", encoding="utf-8", errors="ignore") as file:
|
||||
with open(filepath, "r") as file:
|
||||
contents = file.read()
|
||||
with open(filepath, "w") as file:
|
||||
file.write(contents.replace(find, replace))
|
||||
|
||||
@@ -315,7 +315,9 @@ class Crew(FlowTrackable, BaseModel):
|
||||
"""Initialize private memory attributes."""
|
||||
self._external_memory = (
|
||||
# External memory doesn’t support a default value since it was designed to be managed entirely externally
|
||||
self.external_memory.set_crew(self) if self.external_memory else None
|
||||
self.external_memory.set_crew(self)
|
||||
if self.external_memory
|
||||
else None
|
||||
)
|
||||
|
||||
self._long_term_memory = self.long_term_memory
|
||||
@@ -1202,6 +1204,7 @@ class Crew(FlowTrackable, BaseModel):
|
||||
"_long_term_memory",
|
||||
"_entity_memory",
|
||||
"_external_memory",
|
||||
"_telemetry",
|
||||
"agents",
|
||||
"tasks",
|
||||
"knowledge_sources",
|
||||
@@ -1394,10 +1397,10 @@ class Crew(FlowTrackable, BaseModel):
|
||||
memory_systems = self._get_memory_systems()
|
||||
|
||||
for memory_type, config in memory_systems.items():
|
||||
if (system := config.get("system")) is not None:
|
||||
name = config.get("name")
|
||||
if (system := config.get('system')) is not None:
|
||||
name = config.get('name')
|
||||
try:
|
||||
reset_fn: Callable = cast(Callable, config.get("reset"))
|
||||
reset_fn: Callable = cast(Callable, config.get('reset'))
|
||||
reset_fn(system)
|
||||
self._logger.log(
|
||||
"info",
|
||||
@@ -1419,14 +1422,14 @@ class Crew(FlowTrackable, BaseModel):
|
||||
"""
|
||||
memory_systems = self._get_memory_systems()
|
||||
config = memory_systems[memory_type]
|
||||
system = config.get("system")
|
||||
name = config.get("name")
|
||||
system = config.get('system')
|
||||
name = config.get('name')
|
||||
|
||||
if system is None:
|
||||
raise RuntimeError(f"{name} memory system is not initialized")
|
||||
|
||||
|
||||
try:
|
||||
reset_fn: Callable = cast(Callable, config.get("reset"))
|
||||
reset_fn: Callable = cast(Callable, config.get('reset'))
|
||||
reset_fn(system)
|
||||
self._logger.log(
|
||||
"info",
|
||||
@@ -1439,67 +1442,58 @@ class Crew(FlowTrackable, BaseModel):
|
||||
|
||||
def _get_memory_systems(self):
|
||||
"""Get all available memory systems with their configuration.
|
||||
|
||||
|
||||
Returns:
|
||||
Dict containing all memory systems with their reset functions and display names.
|
||||
"""
|
||||
|
||||
def default_reset(memory):
|
||||
return memory.reset()
|
||||
|
||||
def knowledge_reset(memory):
|
||||
return self.reset_knowledge(memory)
|
||||
|
||||
# Get knowledge for agents
|
||||
agent_knowledges = [
|
||||
getattr(agent, "knowledge", None)
|
||||
for agent in self.agents
|
||||
if getattr(agent, "knowledge", None) is not None
|
||||
]
|
||||
|
||||
# Get knowledge for agents
|
||||
agent_knowledges = [getattr(agent, "knowledge", None) for agent in self.agents
|
||||
if getattr(agent, "knowledge", None) is not None]
|
||||
# Get knowledge for crew and agents
|
||||
crew_knowledge = getattr(self, "knowledge", None)
|
||||
crew_and_agent_knowledges = (
|
||||
[crew_knowledge] if crew_knowledge is not None else []
|
||||
) + agent_knowledges
|
||||
crew_and_agent_knowledges = ([crew_knowledge] if crew_knowledge is not None else []) + agent_knowledges
|
||||
|
||||
return {
|
||||
"short": {
|
||||
"system": getattr(self, "_short_term_memory", None),
|
||||
"reset": default_reset,
|
||||
"name": "Short Term",
|
||||
'short': {
|
||||
'system': getattr(self, "_short_term_memory", None),
|
||||
'reset': default_reset,
|
||||
'name': 'Short Term'
|
||||
},
|
||||
"entity": {
|
||||
"system": getattr(self, "_entity_memory", None),
|
||||
"reset": default_reset,
|
||||
"name": "Entity",
|
||||
'entity': {
|
||||
'system': getattr(self, "_entity_memory", None),
|
||||
'reset': default_reset,
|
||||
'name': 'Entity'
|
||||
},
|
||||
"external": {
|
||||
"system": getattr(self, "_external_memory", None),
|
||||
"reset": default_reset,
|
||||
"name": "External",
|
||||
'external': {
|
||||
'system': getattr(self, "_external_memory", None),
|
||||
'reset': default_reset,
|
||||
'name': 'External'
|
||||
},
|
||||
"long": {
|
||||
"system": getattr(self, "_long_term_memory", None),
|
||||
"reset": default_reset,
|
||||
"name": "Long Term",
|
||||
'long': {
|
||||
'system': getattr(self, "_long_term_memory", None),
|
||||
'reset': default_reset,
|
||||
'name': 'Long Term'
|
||||
},
|
||||
"kickoff_outputs": {
|
||||
"system": getattr(self, "_task_output_handler", None),
|
||||
"reset": default_reset,
|
||||
"name": "Task Output",
|
||||
'kickoff_outputs': {
|
||||
'system': getattr(self, "_task_output_handler", None),
|
||||
'reset': default_reset,
|
||||
'name': 'Task Output'
|
||||
},
|
||||
"knowledge": {
|
||||
"system": crew_and_agent_knowledges
|
||||
if crew_and_agent_knowledges
|
||||
else None,
|
||||
"reset": knowledge_reset,
|
||||
"name": "Crew Knowledge and Agent Knowledge",
|
||||
},
|
||||
"agent_knowledge": {
|
||||
"system": agent_knowledges if agent_knowledges else None,
|
||||
"reset": knowledge_reset,
|
||||
"name": "Agent Knowledge",
|
||||
'knowledge': {
|
||||
'system': crew_and_agent_knowledges if crew_and_agent_knowledges else None,
|
||||
'reset': knowledge_reset,
|
||||
'name': 'Crew Knowledge and Agent Knowledge'
|
||||
},
|
||||
'agent_knowledge': {
|
||||
'system': agent_knowledges if agent_knowledges else None,
|
||||
'reset': knowledge_reset,
|
||||
'name': 'Agent Knowledge'
|
||||
}
|
||||
}
|
||||
|
||||
def reset_knowledge(self, knowledges: List[Knowledge]) -> None:
|
||||
|
||||
@@ -1,96 +0,0 @@
|
||||
"""Hallucination Guardrail Placeholder for CrewAI.
|
||||
|
||||
This is a no-op version of the HallucinationGuardrail for the open-source repository.
|
||||
|
||||
Classes:
|
||||
HallucinationGuardrail: Placeholder guardrail that validates task outputs.
|
||||
"""
|
||||
|
||||
from typing import Any, Optional, Tuple
|
||||
|
||||
from crewai.llm import LLM
|
||||
from crewai.tasks.task_output import TaskOutput
|
||||
from crewai.utilities.logger import Logger
|
||||
|
||||
|
||||
class HallucinationGuardrail:
|
||||
"""Placeholder for the HallucinationGuardrail feature.
|
||||
|
||||
Attributes:
|
||||
context: The reference context that outputs would be checked against.
|
||||
llm: The language model that would be used for evaluation.
|
||||
threshold: Optional minimum faithfulness score that would be required to pass.
|
||||
tool_response: Optional tool response information that would be used in evaluation.
|
||||
|
||||
Examples:
|
||||
>>> # Basic usage with default verdict logic
|
||||
>>> guardrail = HallucinationGuardrail(
|
||||
... context="AI helps with various tasks including analysis and generation.",
|
||||
... llm=agent.llm
|
||||
... )
|
||||
|
||||
>>> # With custom threshold for stricter validation
|
||||
>>> strict_guardrail = HallucinationGuardrail(
|
||||
... context="Quantum computing uses qubits in superposition.",
|
||||
... llm=agent.llm,
|
||||
... threshold=8.0 # Would require score >= 8 to pass in enterprise version
|
||||
... )
|
||||
|
||||
>>> # With tool response for additional context
|
||||
>>> guardrail_with_tools = HallucinationGuardrail(
|
||||
... context="The current weather data",
|
||||
... llm=agent.llm,
|
||||
... tool_response="Weather API returned: Temperature 22°C, Humidity 65%"
|
||||
... )
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
context: str,
|
||||
llm: LLM,
|
||||
threshold: Optional[float] = None,
|
||||
tool_response: str = "",
|
||||
):
|
||||
"""Initialize the HallucinationGuardrail placeholder.
|
||||
|
||||
Args:
|
||||
context: The reference context that outputs would be checked against.
|
||||
llm: The language model that would be used for evaluation.
|
||||
threshold: Optional minimum faithfulness score that would be required to pass.
|
||||
tool_response: Optional tool response information that would be used in evaluation.
|
||||
"""
|
||||
self.context = context
|
||||
self.llm: LLM = llm
|
||||
self.threshold = threshold
|
||||
self.tool_response = tool_response
|
||||
self._logger = Logger(verbose=True)
|
||||
self._logger.log(
|
||||
"warning",
|
||||
"""Hallucination detection is a no-op in open source, use it for free at https://app.crewai.com\n""",
|
||||
color="red",
|
||||
)
|
||||
|
||||
@property
|
||||
def description(self) -> str:
|
||||
"""Generate a description of this guardrail for event logging."""
|
||||
return "HallucinationGuardrail (no-op)"
|
||||
|
||||
def __call__(self, task_output: TaskOutput) -> Tuple[bool, Any]:
|
||||
"""Validate a task output against hallucination criteria.
|
||||
|
||||
In the open source, this method always returns that the output is valid.
|
||||
|
||||
Args:
|
||||
task_output: The output to be validated.
|
||||
|
||||
Returns:
|
||||
A tuple containing:
|
||||
- True
|
||||
- The raw task output
|
||||
"""
|
||||
self._logger.log(
|
||||
"warning",
|
||||
"Premium hallucination detection skipped (use for free at https://app.crewai.com)\n",
|
||||
color="red",
|
||||
)
|
||||
return True, task_output.raw
|
||||
@@ -9,7 +9,6 @@ import warnings
|
||||
from contextlib import contextmanager
|
||||
from importlib.metadata import version
|
||||
from typing import TYPE_CHECKING, Any, Optional
|
||||
import threading
|
||||
|
||||
from opentelemetry import trace
|
||||
from opentelemetry.exporter.otlp.proto.http.trace_exporter import (
|
||||
@@ -65,17 +64,7 @@ class Telemetry:
|
||||
attribute in the Crew class.
|
||||
"""
|
||||
|
||||
_instance = None
|
||||
_lock = threading.Lock()
|
||||
|
||||
def __new__(cls):
|
||||
if cls._instance is None:
|
||||
with cls._lock:
|
||||
if cls._instance is None:
|
||||
cls._instance = super(Telemetry, cls).__new__(cls)
|
||||
return cls._instance
|
||||
|
||||
def __init__(self) -> None:
|
||||
def __init__(self):
|
||||
self.ready: bool = False
|
||||
self.trace_set: bool = False
|
||||
|
||||
|
||||
@@ -9,8 +9,7 @@ from crewai.agent import Agent
|
||||
from crewai.llm import BaseLLM
|
||||
from crewai.task import Task
|
||||
from crewai.tasks.task_output import TaskOutput
|
||||
from crewai.utilities.events import crewai_event_bus
|
||||
from crewai.utilities.events.crew_events import CrewTestResultEvent
|
||||
from crewai.telemetry import Telemetry
|
||||
|
||||
|
||||
class TaskEvaluationPydanticOutput(BaseModel):
|
||||
@@ -37,6 +36,7 @@ class CrewEvaluator:
|
||||
def __init__(self, crew, eval_llm: InstanceOf[BaseLLM]):
|
||||
self.crew = crew
|
||||
self.llm = eval_llm
|
||||
self._telemetry = Telemetry()
|
||||
self._setup_for_evaluating()
|
||||
|
||||
def _setup_for_evaluating(self) -> None:
|
||||
@@ -178,15 +178,11 @@ class CrewEvaluator:
|
||||
evaluation_result = evaluation_task.execute_sync()
|
||||
|
||||
if isinstance(evaluation_result.pydantic, TaskEvaluationPydanticOutput):
|
||||
crewai_event_bus.emit(
|
||||
self._test_result_span = self._telemetry.individual_test_result_span(
|
||||
self.crew,
|
||||
CrewTestResultEvent(
|
||||
quality=evaluation_result.pydantic.quality,
|
||||
execution_duration=current_task.execution_duration,
|
||||
model=self.llm.model,
|
||||
crew_name=self.crew.name,
|
||||
crew=self.crew,
|
||||
),
|
||||
evaluation_result.pydantic.quality,
|
||||
current_task.execution_duration,
|
||||
self.llm.model,
|
||||
)
|
||||
self.tasks_scores[self.iteration].append(evaluation_result.pydantic.quality)
|
||||
self.run_execution_times[self.iteration].append(
|
||||
|
||||
@@ -100,12 +100,3 @@ class CrewTestFailedEvent(CrewBaseEvent):
|
||||
|
||||
error: str
|
||||
type: str = "crew_test_failed"
|
||||
|
||||
|
||||
class CrewTestResultEvent(CrewBaseEvent):
|
||||
"""Event emitted when a crew test result is available"""
|
||||
|
||||
quality: float
|
||||
execution_duration: float
|
||||
model: str
|
||||
type: str = "crew_test_result"
|
||||
|
||||
@@ -37,7 +37,6 @@ from .crew_events import (
|
||||
CrewKickoffStartedEvent,
|
||||
CrewTestCompletedEvent,
|
||||
CrewTestFailedEvent,
|
||||
CrewTestResultEvent,
|
||||
CrewTestStartedEvent,
|
||||
CrewTrainCompletedEvent,
|
||||
CrewTrainFailedEvent,
|
||||
@@ -135,15 +134,6 @@ class EventListener(BaseEventListener):
|
||||
def on_crew_train_failed(source, event: CrewTrainFailedEvent):
|
||||
self.formatter.handle_crew_train_failed(event.crew_name or "Crew")
|
||||
|
||||
@crewai_event_bus.on(CrewTestResultEvent)
|
||||
def on_crew_test_result(source, event: CrewTestResultEvent):
|
||||
self._telemetry.individual_test_result_span(
|
||||
source.crew,
|
||||
event.quality,
|
||||
int(event.execution_duration),
|
||||
event.model,
|
||||
)
|
||||
|
||||
# ----------- TASK EVENTS -----------
|
||||
|
||||
@crewai_event_bus.on(TaskStartedEvent)
|
||||
|
||||
@@ -48,14 +48,6 @@ from .reasoning_events import (
|
||||
AgentReasoningCompletedEvent,
|
||||
AgentReasoningFailedEvent,
|
||||
)
|
||||
from .knowledge_events import (
|
||||
KnowledgeRetrievalStartedEvent,
|
||||
KnowledgeRetrievalCompletedEvent,
|
||||
KnowledgeQueryStartedEvent,
|
||||
KnowledgeQueryCompletedEvent,
|
||||
KnowledgeQueryFailedEvent,
|
||||
KnowledgeSearchQueryFailedEvent,
|
||||
)
|
||||
|
||||
EventTypes = Union[
|
||||
CrewKickoffStartedEvent,
|
||||
@@ -90,10 +82,4 @@ EventTypes = Union[
|
||||
AgentReasoningStartedEvent,
|
||||
AgentReasoningCompletedEvent,
|
||||
AgentReasoningFailedEvent,
|
||||
KnowledgeRetrievalStartedEvent,
|
||||
KnowledgeRetrievalCompletedEvent,
|
||||
KnowledgeQueryStartedEvent,
|
||||
KnowledgeQueryCompletedEvent,
|
||||
KnowledgeQueryFailedEvent,
|
||||
KnowledgeSearchQueryFailedEvent,
|
||||
]
|
||||
|
||||
@@ -19,13 +19,10 @@ class LLMGuardrailStartedEvent(BaseEvent):
|
||||
from inspect import getsource
|
||||
|
||||
from crewai.tasks.llm_guardrail import LLMGuardrail
|
||||
from crewai.tasks.hallucination_guardrail import HallucinationGuardrail
|
||||
|
||||
super().__init__(**data)
|
||||
|
||||
if isinstance(self.guardrail, LLMGuardrail) or isinstance(
|
||||
self.guardrail, HallucinationGuardrail
|
||||
):
|
||||
if isinstance(self.guardrail, LLMGuardrail):
|
||||
self.guardrail = self.guardrail.description.strip()
|
||||
elif isinstance(self.guardrail, Callable):
|
||||
self.guardrail = getsource(self.guardrail).strip()
|
||||
|
||||
@@ -4,7 +4,6 @@ from rich.console import Console
|
||||
from rich.panel import Panel
|
||||
from rich.text import Text
|
||||
from rich.tree import Tree
|
||||
from rich.live import Live
|
||||
|
||||
|
||||
class ConsoleFormatter:
|
||||
@@ -21,12 +20,6 @@ class ConsoleFormatter:
|
||||
def __init__(self, verbose: bool = False):
|
||||
self.console = Console(width=None)
|
||||
self.verbose = verbose
|
||||
# Live instance to dynamically update a Tree renderable (e.g. the Crew tree)
|
||||
# When multiple Tree objects are printed sequentially we reuse this Live
|
||||
# instance so the previous render is replaced instead of writing a new one.
|
||||
# Once any non-Tree renderable is printed we stop the Live session so the
|
||||
# final Tree persists on the terminal.
|
||||
self._live: Optional[Live] = None
|
||||
|
||||
def create_panel(self, content: Text, title: str, style: str = "blue") -> Panel:
|
||||
"""Create a standardized panel with consistent styling."""
|
||||
@@ -67,7 +60,7 @@ class ConsoleFormatter:
|
||||
label.append(f"{prefix} ", style=f"{style} bold")
|
||||
label.append(name, style=style)
|
||||
if status:
|
||||
label.append("\nStatus: ", style="white")
|
||||
label.append("\n Status: ", style="white")
|
||||
label.append(status, style=f"{style} bold")
|
||||
tree.label = label
|
||||
|
||||
@@ -76,46 +69,7 @@ class ConsoleFormatter:
|
||||
return parent.add(Text(text, style=style))
|
||||
|
||||
def print(self, *args, **kwargs) -> None:
|
||||
"""Custom print that replaces consecutive Tree renders.
|
||||
|
||||
* If the argument is a single ``Tree`` instance, we either start a
|
||||
``Live`` session (first tree) or update the existing one (subsequent
|
||||
trees). This results in the tree being rendered in-place instead of
|
||||
being appended repeatedly to the log.
|
||||
|
||||
* A blank call (no positional arguments) is ignored while a Live
|
||||
session is active so it does not prematurely terminate the tree
|
||||
rendering.
|
||||
|
||||
* Any other renderable will terminate the Live session (if one is
|
||||
active) so the last tree stays on screen and the new content is
|
||||
printed normally.
|
||||
"""
|
||||
|
||||
# Case 1: updating / starting live Tree rendering
|
||||
if len(args) == 1 and isinstance(args[0], Tree):
|
||||
tree = args[0]
|
||||
|
||||
if not self._live:
|
||||
# Start a new Live session for the first tree
|
||||
self._live = Live(tree, console=self.console, refresh_per_second=4)
|
||||
self._live.start()
|
||||
else:
|
||||
# Update existing Live session
|
||||
self._live.update(tree, refresh=True)
|
||||
return # Nothing else to do
|
||||
|
||||
# Case 2: blank line while a live session is running – ignore so we
|
||||
# don't break the in-place rendering behaviour
|
||||
if len(args) == 0 and self._live:
|
||||
return
|
||||
|
||||
# Case 3: printing something other than a Tree → terminate live session
|
||||
if self._live:
|
||||
self._live.stop()
|
||||
self._live = None
|
||||
|
||||
# Finally, pass through to the regular Console.print implementation
|
||||
"""Print to console with consistent formatting if verbose is enabled."""
|
||||
self.console.print(*args, **kwargs)
|
||||
|
||||
def print_panel(
|
||||
@@ -203,7 +157,7 @@ class ConsoleFormatter:
|
||||
|
||||
task_content = Text()
|
||||
task_content.append(f"📋 Task: {task_id}", style="yellow bold")
|
||||
task_content.append("\nStatus: ", style="white")
|
||||
task_content.append("\n Status: ", style="white")
|
||||
task_content.append("Executing Task...", style="yellow dim")
|
||||
|
||||
task_branch = None
|
||||
@@ -243,17 +197,11 @@ class ConsoleFormatter:
|
||||
# Update tree label
|
||||
for branch in crew_tree.children:
|
||||
if str(task_id) in str(branch.label):
|
||||
# Build label without introducing stray blank lines
|
||||
task_content = Text()
|
||||
# First line: Task ID
|
||||
task_content.append(f"📋 Task: {task_id}", style=f"{style} bold")
|
||||
|
||||
# Second line: Assigned to
|
||||
task_content.append("\nAssigned to: ", style="white")
|
||||
task_content.append("\n Assigned to: ", style="white")
|
||||
task_content.append(agent_role, style=style)
|
||||
|
||||
# Third line: Status
|
||||
task_content.append("Status: ", style="white")
|
||||
task_content.append("\n Status: ", style="white")
|
||||
task_content.append(status_text, style=f"{style} bold")
|
||||
branch.label = task_content
|
||||
self.print(crew_tree)
|
||||
@@ -272,16 +220,18 @@ class ConsoleFormatter:
|
||||
if not self.verbose or not task_branch or not crew_tree:
|
||||
return None
|
||||
|
||||
# Instead of creating a separate Agent node, we treat the task branch
|
||||
# itself as the logical agent branch so that Reasoning/Tool nodes are
|
||||
# nested under the task without an extra visual level.
|
||||
agent_branch = task_branch.add("")
|
||||
self.update_tree_label(
|
||||
agent_branch, "🤖 Agent:", agent_role, "green", "In Progress"
|
||||
)
|
||||
|
||||
# Store the task branch as the current_agent_branch for future nesting.
|
||||
self.current_agent_branch = task_branch
|
||||
self.print(crew_tree)
|
||||
self.print()
|
||||
|
||||
# No additional tree modification needed; return the task branch so
|
||||
# caller logic remains unchanged.
|
||||
return task_branch
|
||||
# Set the current_agent_branch attribute directly
|
||||
self.current_agent_branch = agent_branch
|
||||
|
||||
return agent_branch
|
||||
|
||||
def update_agent_status(
|
||||
self,
|
||||
@@ -291,10 +241,19 @@ class ConsoleFormatter:
|
||||
status: str = "completed",
|
||||
) -> None:
|
||||
"""Update agent status in the tree."""
|
||||
# We no longer render a separate agent branch, so this method simply
|
||||
# updates the stored branch reference (already the task branch) without
|
||||
# altering the tree. Keeping it a no-op avoids duplicate status lines.
|
||||
return
|
||||
if not self.verbose or agent_branch is None or crew_tree is None:
|
||||
return
|
||||
|
||||
self.update_tree_label(
|
||||
agent_branch,
|
||||
"🤖 Agent:",
|
||||
agent_role,
|
||||
"green",
|
||||
"✅ Completed" if status == "completed" else "❌ Failed",
|
||||
)
|
||||
|
||||
self.print(crew_tree)
|
||||
self.print()
|
||||
|
||||
def create_flow_tree(self, flow_name: str, flow_id: str) -> Optional[Tree]:
|
||||
"""Create and initialize a flow tree."""
|
||||
@@ -307,7 +266,7 @@ class ConsoleFormatter:
|
||||
flow_label = Text()
|
||||
flow_label.append("🌊 Flow: ", style="blue bold")
|
||||
flow_label.append(flow_name, style="blue")
|
||||
flow_label.append("\nID: ", style="white")
|
||||
flow_label.append("\n ID: ", style="white")
|
||||
flow_label.append(flow_id, style="blue")
|
||||
|
||||
flow_tree = Tree(flow_label)
|
||||
@@ -322,7 +281,7 @@ class ConsoleFormatter:
|
||||
flow_label = Text()
|
||||
flow_label.append("🌊 Flow: ", style="blue bold")
|
||||
flow_label.append(flow_name, style="blue")
|
||||
flow_label.append("\nID: ", style="white")
|
||||
flow_label.append("\n ID: ", style="white")
|
||||
flow_label.append(flow_id, style="blue")
|
||||
flow_tree.label = flow_label
|
||||
|
||||
@@ -436,15 +395,9 @@ class ConsoleFormatter:
|
||||
if not self.verbose:
|
||||
return None
|
||||
|
||||
# Parent for tool usage: LiteAgent > Agent > Task
|
||||
branch_to_use = (
|
||||
self.current_lite_agent_branch
|
||||
or agent_branch
|
||||
or self.current_task_branch
|
||||
)
|
||||
|
||||
# Render full crew tree when available for consistent live updates
|
||||
tree_to_use = self.current_crew_tree or crew_tree or branch_to_use
|
||||
# Use LiteAgent branch if available, otherwise use regular agent branch
|
||||
branch_to_use = self.current_lite_agent_branch or agent_branch
|
||||
tree_to_use = branch_to_use or crew_tree
|
||||
|
||||
if branch_to_use is None or tree_to_use is None:
|
||||
# If we don't have a valid branch, default to crew_tree if provided
|
||||
@@ -470,9 +423,10 @@ class ConsoleFormatter:
|
||||
"yellow",
|
||||
)
|
||||
|
||||
# Print updated tree immediately
|
||||
self.print(tree_to_use)
|
||||
self.print()
|
||||
# Only print if this is a new tool usage
|
||||
if tool_branch not in branch_to_use.children:
|
||||
self.print(tree_to_use)
|
||||
self.print()
|
||||
|
||||
return tool_branch
|
||||
|
||||
@@ -486,8 +440,8 @@ class ConsoleFormatter:
|
||||
if not self.verbose or tool_branch is None:
|
||||
return
|
||||
|
||||
# Decide which tree to render: prefer full crew tree, else parent branch
|
||||
tree_to_use = self.current_crew_tree or crew_tree or self.current_task_branch
|
||||
# Use LiteAgent branch if available, otherwise use crew tree
|
||||
tree_to_use = self.current_lite_agent_branch or crew_tree
|
||||
if tree_to_use is None:
|
||||
return
|
||||
|
||||
@@ -518,8 +472,8 @@ class ConsoleFormatter:
|
||||
if not self.verbose:
|
||||
return
|
||||
|
||||
# Decide which tree to render: prefer full crew tree, else parent branch
|
||||
tree_to_use = self.current_crew_tree or crew_tree or self.current_task_branch
|
||||
# Use LiteAgent branch if available, otherwise use crew tree
|
||||
tree_to_use = self.current_lite_agent_branch or crew_tree
|
||||
|
||||
if tool_branch:
|
||||
self.update_tree_label(
|
||||
@@ -547,15 +501,9 @@ class ConsoleFormatter:
|
||||
if not self.verbose:
|
||||
return None
|
||||
|
||||
# Parent for tool usage: LiteAgent > Agent > Task
|
||||
branch_to_use = (
|
||||
self.current_lite_agent_branch
|
||||
or agent_branch
|
||||
or self.current_task_branch
|
||||
)
|
||||
|
||||
# Render full crew tree when available for consistent live updates
|
||||
tree_to_use = self.current_crew_tree or crew_tree or branch_to_use
|
||||
# Use LiteAgent branch if available, otherwise use regular agent branch
|
||||
branch_to_use = self.current_lite_agent_branch or agent_branch
|
||||
tree_to_use = branch_to_use or crew_tree
|
||||
|
||||
if branch_to_use is None or tree_to_use is None:
|
||||
# If we don't have a valid branch, default to crew_tree if provided
|
||||
@@ -584,33 +532,24 @@ class ConsoleFormatter:
|
||||
if not self.verbose or tool_branch is None:
|
||||
return
|
||||
|
||||
# Decide which tree to render: prefer full crew tree, else parent branch
|
||||
tree_to_use = self.current_crew_tree or crew_tree or self.current_task_branch
|
||||
if tree_to_use is None:
|
||||
# Use LiteAgent branch if available, otherwise use regular agent branch
|
||||
branch_to_use = self.current_lite_agent_branch or agent_branch
|
||||
tree_to_use = branch_to_use or crew_tree
|
||||
|
||||
if branch_to_use is None or tree_to_use is None:
|
||||
return
|
||||
|
||||
# Remove the thinking status node when complete
|
||||
# Remove the thinking status node when complete, but only if it exists
|
||||
if "Thinking" in str(tool_branch.label):
|
||||
parents = [
|
||||
self.current_lite_agent_branch,
|
||||
self.current_agent_branch,
|
||||
self.current_task_branch,
|
||||
tree_to_use,
|
||||
]
|
||||
removed = False
|
||||
for parent in parents:
|
||||
if isinstance(parent, Tree) and tool_branch in parent.children:
|
||||
parent.children.remove(tool_branch)
|
||||
removed = True
|
||||
break
|
||||
|
||||
# Clear pointer if we just removed the current_tool_branch
|
||||
if self.current_tool_branch is tool_branch:
|
||||
self.current_tool_branch = None
|
||||
|
||||
if removed:
|
||||
self.print(tree_to_use)
|
||||
self.print()
|
||||
try:
|
||||
# Check if the node is actually in the children list
|
||||
if tool_branch in branch_to_use.children:
|
||||
branch_to_use.children.remove(tool_branch)
|
||||
self.print(tree_to_use)
|
||||
self.print()
|
||||
except Exception:
|
||||
# If any error occurs during removal, just continue without removing
|
||||
pass
|
||||
|
||||
def handle_llm_call_failed(
|
||||
self, tool_branch: Optional[Tree], error: str, crew_tree: Optional[Tree]
|
||||
@@ -619,8 +558,8 @@ class ConsoleFormatter:
|
||||
if not self.verbose:
|
||||
return
|
||||
|
||||
# Decide which tree to render: prefer full crew tree, else parent branch
|
||||
tree_to_use = self.current_crew_tree or crew_tree or self.current_task_branch
|
||||
# Use LiteAgent branch if available, otherwise use crew tree
|
||||
tree_to_use = self.current_lite_agent_branch or crew_tree
|
||||
|
||||
# Update tool branch if it exists
|
||||
if tool_branch:
|
||||
@@ -662,7 +601,7 @@ class ConsoleFormatter:
|
||||
test_label = Text()
|
||||
test_label.append("🧪 Test: ", style="blue bold")
|
||||
test_label.append(crew_name or "Crew", style="blue")
|
||||
test_label.append("\nStatus: ", style="white")
|
||||
test_label.append("\n Status: ", style="white")
|
||||
test_label.append("In Progress", style="yellow")
|
||||
|
||||
test_tree = Tree(test_label)
|
||||
@@ -684,7 +623,7 @@ class ConsoleFormatter:
|
||||
test_label = Text()
|
||||
test_label.append("✅ Test: ", style="green bold")
|
||||
test_label.append(crew_name or "Crew", style="green")
|
||||
test_label.append("\nStatus: ", style="white")
|
||||
test_label.append("\n Status: ", style="white")
|
||||
test_label.append("Completed", style="green bold")
|
||||
flow_tree.label = test_label
|
||||
|
||||
@@ -773,7 +712,7 @@ class ConsoleFormatter:
|
||||
lite_agent_label = Text()
|
||||
lite_agent_label.append("🤖 LiteAgent: ", style="cyan bold")
|
||||
lite_agent_label.append(lite_agent_role, style="cyan")
|
||||
lite_agent_label.append("\nStatus: ", style="white")
|
||||
lite_agent_label.append("\n Status: ", style="white")
|
||||
lite_agent_label.append("In Progress", style="yellow")
|
||||
|
||||
lite_agent_tree = Tree(lite_agent_label)
|
||||
@@ -812,7 +751,7 @@ class ConsoleFormatter:
|
||||
lite_agent_label = Text()
|
||||
lite_agent_label.append(f"{prefix} ", style=f"{style} bold")
|
||||
lite_agent_label.append(lite_agent_role, style=style)
|
||||
lite_agent_label.append("Status: ", style="white")
|
||||
lite_agent_label.append("\n Status: ", style="white")
|
||||
lite_agent_label.append(status_text, style=f"{style} bold")
|
||||
lite_agent_branch.label = lite_agent_label
|
||||
|
||||
@@ -1065,20 +1004,16 @@ class ConsoleFormatter:
|
||||
if not self.verbose:
|
||||
return None
|
||||
|
||||
# Prefer LiteAgent > Agent > Task branch as the parent for reasoning
|
||||
branch_to_use = (
|
||||
self.current_lite_agent_branch
|
||||
or agent_branch
|
||||
or self.current_task_branch
|
||||
)
|
||||
# Prefer LiteAgent branch if we are within a LiteAgent context
|
||||
branch_to_use = self.current_lite_agent_branch or agent_branch
|
||||
tree_to_use = branch_to_use or crew_tree
|
||||
|
||||
# We always want to render the full crew tree when possible so the
|
||||
# Live view updates coherently. Fallbacks: crew tree → branch itself.
|
||||
tree_to_use = self.current_crew_tree or crew_tree or branch_to_use
|
||||
|
||||
if branch_to_use is None:
|
||||
# Nothing to attach to, abort
|
||||
return None
|
||||
if branch_to_use is None or tree_to_use is None:
|
||||
# If we don't have a valid branch, default to crew_tree if provided
|
||||
if crew_tree is not None:
|
||||
branch_to_use = tree_to_use = crew_tree
|
||||
else:
|
||||
return None
|
||||
|
||||
# Reuse existing reasoning branch if present
|
||||
reasoning_branch = self.current_reasoning_branch
|
||||
@@ -1109,9 +1044,8 @@ class ConsoleFormatter:
|
||||
|
||||
reasoning_branch = self.current_reasoning_branch
|
||||
tree_to_use = (
|
||||
self.current_crew_tree
|
||||
or self.current_lite_agent_branch
|
||||
or self.current_task_branch
|
||||
self.current_lite_agent_branch
|
||||
or self.current_agent_branch
|
||||
or crew_tree
|
||||
)
|
||||
|
||||
@@ -1150,9 +1084,8 @@ class ConsoleFormatter:
|
||||
|
||||
reasoning_branch = self.current_reasoning_branch
|
||||
tree_to_use = (
|
||||
self.current_crew_tree
|
||||
or self.current_lite_agent_branch
|
||||
or self.current_task_branch
|
||||
self.current_lite_agent_branch
|
||||
or self.current_agent_branch
|
||||
or crew_tree
|
||||
)
|
||||
|
||||
|
||||
@@ -6,8 +6,6 @@ import pytest
|
||||
from crewai import Agent, Crew, Task
|
||||
from crewai.telemetry import Telemetry
|
||||
|
||||
from opentelemetry import trace
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"env_var,value,expected_ready",
|
||||
@@ -36,6 +34,9 @@ def test_telemetry_enabled_by_default():
|
||||
assert telemetry.ready is True
|
||||
|
||||
|
||||
from opentelemetry import trace
|
||||
|
||||
|
||||
@patch("crewai.telemetry.telemetry.logger.error")
|
||||
@patch(
|
||||
"opentelemetry.exporter.otlp.proto.http.trace_exporter.OTLPSpanExporter.export",
|
||||
@@ -66,32 +67,3 @@ def test_telemetry_fails_due_connect_timeout(export_mock, logger_mock):
|
||||
|
||||
export_mock.assert_called_once()
|
||||
logger_mock.assert_called_once_with(error)
|
||||
|
||||
|
||||
def test_telemetry_singleton_pattern():
|
||||
"""Test that Telemetry uses the singleton pattern correctly."""
|
||||
Telemetry._instance = None
|
||||
|
||||
telemetry1 = Telemetry()
|
||||
telemetry2 = Telemetry()
|
||||
|
||||
assert telemetry1 is telemetry2
|
||||
|
||||
setattr(telemetry1, "test_attribute", "test_value")
|
||||
assert hasattr(telemetry2, "test_attribute")
|
||||
assert getattr(telemetry2, "test_attribute") == "test_value"
|
||||
|
||||
import threading
|
||||
|
||||
instances = []
|
||||
|
||||
def create_instance():
|
||||
instances.append(Telemetry())
|
||||
|
||||
threads = [threading.Thread(target=create_instance) for _ in range(5)]
|
||||
for thread in threads:
|
||||
thread.start()
|
||||
for thread in threads:
|
||||
thread.join()
|
||||
|
||||
assert all(instance is telemetry1 for instance in instances)
|
||||
|
||||
@@ -1,108 +0,0 @@
|
||||
from unittest.mock import Mock
|
||||
|
||||
import pytest
|
||||
|
||||
from crewai.llm import LLM
|
||||
from crewai.tasks.hallucination_guardrail import HallucinationGuardrail
|
||||
from crewai.tasks.task_output import TaskOutput
|
||||
|
||||
|
||||
def test_hallucination_guardrail_initialization():
|
||||
"""Test that the hallucination guardrail initializes correctly with all parameters."""
|
||||
mock_llm = Mock(spec=LLM)
|
||||
|
||||
guardrail = HallucinationGuardrail(context="Test reference context", llm=mock_llm)
|
||||
|
||||
assert guardrail.context == "Test reference context"
|
||||
assert guardrail.llm == mock_llm
|
||||
assert guardrail.threshold is None
|
||||
assert guardrail.tool_response == ""
|
||||
|
||||
guardrail = HallucinationGuardrail(
|
||||
context="Test reference context",
|
||||
llm=mock_llm,
|
||||
threshold=8.5,
|
||||
tool_response="Sample tool response",
|
||||
)
|
||||
|
||||
assert guardrail.context == "Test reference context"
|
||||
assert guardrail.llm == mock_llm
|
||||
assert guardrail.threshold == 8.5
|
||||
assert guardrail.tool_response == "Sample tool response"
|
||||
|
||||
|
||||
def test_hallucination_guardrail_no_op_behavior():
|
||||
"""Test that the guardrail always returns True in the open-source version."""
|
||||
mock_llm = Mock(spec=LLM)
|
||||
guardrail = HallucinationGuardrail(
|
||||
context="Test reference context",
|
||||
llm=mock_llm,
|
||||
threshold=9.0,
|
||||
)
|
||||
|
||||
task_output = TaskOutput(
|
||||
raw="Sample task output",
|
||||
description="Test task",
|
||||
expected_output="Expected output",
|
||||
agent="Test Agent",
|
||||
)
|
||||
|
||||
result, output = guardrail(task_output)
|
||||
|
||||
assert result is True
|
||||
assert output == "Sample task output"
|
||||
|
||||
|
||||
def test_hallucination_guardrail_description():
|
||||
"""Test that the guardrail provides the correct description for event logging."""
|
||||
guardrail = HallucinationGuardrail(
|
||||
context="Test reference context", llm=Mock(spec=LLM)
|
||||
)
|
||||
|
||||
assert guardrail.description == "HallucinationGuardrail (no-op)"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"context,task_output_text,threshold,tool_response",
|
||||
[
|
||||
(
|
||||
"Earth orbits the Sun once every 365.25 days.",
|
||||
"It takes Earth approximately one year to go around the Sun.",
|
||||
None,
|
||||
"",
|
||||
),
|
||||
(
|
||||
"Python was created by Guido van Rossum in 1991.",
|
||||
"Python is a programming language developed by Guido van Rossum.",
|
||||
7.5,
|
||||
"",
|
||||
),
|
||||
(
|
||||
"The capital of France is Paris.",
|
||||
"Paris is the largest city and capital of France.",
|
||||
9.0,
|
||||
"Geographic API returned: France capital is Paris",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_hallucination_guardrail_always_passes(
|
||||
context, task_output_text, threshold, tool_response
|
||||
):
|
||||
"""Test that the guardrail always passes regardless of configuration in open-source version."""
|
||||
mock_llm = Mock(spec=LLM)
|
||||
|
||||
guardrail = HallucinationGuardrail(
|
||||
context=context, llm=mock_llm, threshold=threshold, tool_response=tool_response
|
||||
)
|
||||
|
||||
task_output = TaskOutput(
|
||||
raw=task_output_text,
|
||||
description="Test task",
|
||||
expected_output="Expected output",
|
||||
agent="Test Agent",
|
||||
)
|
||||
|
||||
result, output = guardrail(task_output)
|
||||
|
||||
assert result is True
|
||||
assert output == task_output_text
|
||||
@@ -1,10 +1,9 @@
|
||||
from unittest.mock import Mock, patch
|
||||
from unittest.mock import ANY, Mock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from crewai import Agent, Task
|
||||
from crewai.llm import LLM
|
||||
from crewai.tasks.hallucination_guardrail import HallucinationGuardrail
|
||||
from crewai.tasks.llm_guardrail import LLMGuardrail
|
||||
from crewai.tasks.task_output import TaskOutput
|
||||
from crewai.utilities.events import (
|
||||
@@ -268,37 +267,3 @@ def test_guardrail_when_an_error_occurs(sample_agent, task_output):
|
||||
max_retries=0,
|
||||
)
|
||||
task.execute_sync(agent=sample_agent)
|
||||
|
||||
|
||||
def test_hallucination_guardrail_integration():
|
||||
"""Test that HallucinationGuardrail integrates properly with the task system."""
|
||||
agent = Mock()
|
||||
agent.role = "test_agent"
|
||||
agent.execute_task.return_value = "test result"
|
||||
agent.crew = None
|
||||
|
||||
mock_llm = Mock(spec=LLM)
|
||||
guardrail = HallucinationGuardrail(
|
||||
context="Test reference context for validation", llm=mock_llm, threshold=8.0
|
||||
)
|
||||
|
||||
task = Task(
|
||||
description="Test task with hallucination guardrail",
|
||||
expected_output="Valid output",
|
||||
guardrail=guardrail,
|
||||
)
|
||||
|
||||
result = task.execute_sync(agent=agent)
|
||||
assert isinstance(result, TaskOutput)
|
||||
assert result.raw == "test result"
|
||||
|
||||
|
||||
def test_hallucination_guardrail_description_in_events():
|
||||
"""Test that HallucinationGuardrail description appears correctly in events."""
|
||||
mock_llm = Mock(spec=LLM)
|
||||
guardrail = HallucinationGuardrail(context="Test context", llm=mock_llm)
|
||||
|
||||
assert guardrail.description == "HallucinationGuardrail (no-op)"
|
||||
|
||||
event = LLMGuardrailStartedEvent(guardrail=guardrail, retry_count=0)
|
||||
assert event.guardrail == "HallucinationGuardrail (no-op)"
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -1,3 +1,4 @@
|
||||
import os
|
||||
from datetime import datetime
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
@@ -21,7 +22,6 @@ from crewai.utilities.events.crew_events import (
|
||||
CrewKickoffFailedEvent,
|
||||
CrewKickoffStartedEvent,
|
||||
CrewTestCompletedEvent,
|
||||
CrewTestResultEvent,
|
||||
CrewTestStartedEvent,
|
||||
)
|
||||
from crewai.utilities.events.crewai_event_bus import crewai_event_bus
|
||||
@@ -38,6 +38,7 @@ from crewai.utilities.events.llm_events import (
|
||||
LLMCallCompletedEvent,
|
||||
LLMCallFailedEvent,
|
||||
LLMCallStartedEvent,
|
||||
LLMCallType,
|
||||
LLMStreamChunkEvent,
|
||||
)
|
||||
from crewai.utilities.events.task_events import (
|
||||
@@ -131,10 +132,6 @@ def test_crew_emits_test_kickoff_type_event():
|
||||
def handle_crew_test_end(source, event):
|
||||
received_events.append(event)
|
||||
|
||||
@crewai_event_bus.on(CrewTestResultEvent)
|
||||
def handle_crew_test_result(source, event):
|
||||
received_events.append(event)
|
||||
|
||||
eval_llm = LLM(model="gpt-4o-mini")
|
||||
with (
|
||||
patch.object(
|
||||
@@ -152,16 +149,13 @@ def test_crew_emits_test_kickoff_type_event():
|
||||
assert args[2] is None
|
||||
assert args[3] == eval_llm
|
||||
|
||||
assert len(received_events) == 3
|
||||
assert len(received_events) == 2
|
||||
assert received_events[0].crew_name == "TestCrew"
|
||||
assert isinstance(received_events[0].timestamp, datetime)
|
||||
assert received_events[0].type == "crew_test_started"
|
||||
assert received_events[1].crew_name == "TestCrew"
|
||||
assert isinstance(received_events[1].timestamp, datetime)
|
||||
assert received_events[1].type == "crew_test_result"
|
||||
assert received_events[2].crew_name == "TestCrew"
|
||||
assert isinstance(received_events[2].timestamp, datetime)
|
||||
assert received_events[2].type == "crew_test_completed"
|
||||
assert received_events[1].type == "crew_test_completed"
|
||||
|
||||
|
||||
@pytest.mark.vcr(filter_headers=["authorization"])
|
||||
@@ -315,7 +309,7 @@ def test_agent_emits_execution_error_event():
|
||||
) as invoke_mock:
|
||||
invoke_mock.side_effect = Exception(error_message)
|
||||
|
||||
with pytest.raises(Exception):
|
||||
with pytest.raises(Exception) as e:
|
||||
base_agent.execute_task(
|
||||
task=base_task,
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user