mirror of
https://github.com/crewAIInc/crewAI.git
synced 2026-05-12 20:48:15 +00:00
Compare commits
12 Commits
lg-agent-e
...
devin/1751
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d86259b0b9 | ||
|
|
1f106015ea | ||
|
|
eec1262d4f | ||
|
|
c6caa763d7 | ||
|
|
08fa3797ca | ||
|
|
f388890971 | ||
|
|
3220575d29 | ||
|
|
6be376f804 | ||
|
|
5b548d618d | ||
|
|
2ffed3ccf0 | ||
|
|
1ea3fc44fa | ||
|
|
6e91a26785 |
@@ -32,6 +32,7 @@ A crew in crewAI represents a collaborative group of agents working together to
|
||||
| **Prompt File** _(optional)_ | `prompt_file` | Path to the prompt JSON file to be used for the crew. |
|
||||
| **Planning** *(optional)* | `planning` | Adds planning ability to the Crew. When activated before each Crew iteration, all Crew data is sent to an AgentPlanner that will plan the tasks and this plan will be added to each task description. |
|
||||
| **Planning LLM** *(optional)* | `planning_llm` | The language model used by the AgentPlanner in a planning process. |
|
||||
| **Knowledge Sources** _(optional)_ | `knowledge_sources` | Knowledge sources available at the crew level, accessible to all the agents. |
|
||||
|
||||
<Tip>
|
||||
**Crew Max RPM**: The `max_rpm` attribute sets the maximum number of requests per minute the crew can perform to avoid rate limits and will override individual agents' `max_rpm` settings if you set it.
|
||||
|
||||
@@ -57,6 +57,7 @@ crew = Crew(
|
||||
| **Output JSON** _(optional)_ | `output_json` | `Optional[Type[BaseModel]]` | A Pydantic model to structure the JSON output. |
|
||||
| **Output Pydantic** _(optional)_ | `output_pydantic` | `Optional[Type[BaseModel]]` | A Pydantic model for task output. |
|
||||
| **Callback** _(optional)_ | `callback` | `Optional[Any]` | Function/object to be executed after task completion. |
|
||||
| **Guardrail** _(optional)_ | `guardrail` | `Optional[Union[Callable, str]]` | Function or string description to validate task output before proceeding to next task. |
|
||||
|
||||
## Creating Tasks
|
||||
|
||||
@@ -86,6 +87,7 @@ research_task:
|
||||
expected_output: >
|
||||
A list with 10 bullet points of the most relevant information about {topic}
|
||||
agent: researcher
|
||||
guardrail: ensure each bullet contains a minimum of 100 words
|
||||
|
||||
reporting_task:
|
||||
description: >
|
||||
@@ -332,9 +334,13 @@ Task guardrails provide a way to validate and transform task outputs before they
|
||||
are passed to the next task. This feature helps ensure data quality and provides
|
||||
feedback to agents when their output doesn't meet specific criteria.
|
||||
|
||||
### Using Task Guardrails
|
||||
**Guardrails can be defined in two ways:**
|
||||
1. **Function-based guardrails**: Python functions that implement custom validation logic
|
||||
2. **String-based guardrails**: Natural language descriptions that are automatically converted to LLM-powered validation
|
||||
|
||||
To add a guardrail to a task, provide a validation function through the `guardrail` parameter:
|
||||
### Function-Based Guardrails
|
||||
|
||||
To add a function-based guardrail to a task, provide a validation function through the `guardrail` parameter:
|
||||
|
||||
```python Code
|
||||
from typing import Tuple, Union, Dict, Any
|
||||
@@ -372,9 +378,82 @@ blog_task = Task(
|
||||
- On success: it returns a tuple of `(bool, Any)`. For example: `(True, validated_result)`
|
||||
- On Failure: it returns a tuple of `(bool, str)`. For example: `(False, "Error message explain the failure")`
|
||||
|
||||
### LLMGuardrail
|
||||
### String-Based Guardrails
|
||||
|
||||
The `LLMGuardrail` class offers a robust mechanism for validating task outputs.
|
||||
String-based guardrails allow you to describe validation criteria in natural language. When you provide a string instead of a function, CrewAI automatically converts it to an `LLMGuardrail` that uses an AI agent to validate the task output.
|
||||
|
||||
#### Using String Guardrails in Python
|
||||
|
||||
```python Code
|
||||
from crewai import Task
|
||||
|
||||
# Simple string-based guardrail
|
||||
blog_task = Task(
|
||||
description="Write a blog post about AI",
|
||||
expected_output="A blog post under 200 words",
|
||||
agent=blog_agent,
|
||||
guardrail="Ensure the blog post is under 200 words and includes practical examples"
|
||||
)
|
||||
|
||||
# More complex validation criteria
|
||||
research_task = Task(
|
||||
description="Research AI trends for 2025",
|
||||
expected_output="A comprehensive research report",
|
||||
agent=research_agent,
|
||||
guardrail="Ensure each finding includes a credible source and is backed by recent data from 2024-2025"
|
||||
)
|
||||
```
|
||||
|
||||
#### Using String Guardrails in YAML
|
||||
|
||||
```yaml
|
||||
research_task:
|
||||
description: Research the latest AI developments
|
||||
expected_output: A list of 10 bullet points about AI
|
||||
agent: researcher
|
||||
guardrail: ensure each bullet contains a minimum of 100 words
|
||||
|
||||
validation_task:
|
||||
description: Validate the research findings
|
||||
expected_output: A validation report
|
||||
agent: validator
|
||||
guardrail: confirm all sources are from reputable publications and published within the last 2 years
|
||||
```
|
||||
|
||||
#### How String Guardrails Work
|
||||
|
||||
When you provide a string guardrail, CrewAI automatically:
|
||||
1. Creates an `LLMGuardrail` instance using the string as validation criteria
|
||||
2. Uses the task's agent LLM to power the validation
|
||||
3. Creates a temporary validation agent that checks the output against your criteria
|
||||
4. Returns detailed feedback if validation fails
|
||||
|
||||
This approach is ideal when you want to use natural language to describe validation rules without writing custom validation functions.
|
||||
|
||||
### LLMGuardrail Class
|
||||
|
||||
The `LLMGuardrail` class is the underlying mechanism that powers string-based guardrails. You can also use it directly for more advanced control:
|
||||
|
||||
```python Code
|
||||
from crewai import Task
|
||||
from crewai.tasks.llm_guardrail import LLMGuardrail
|
||||
from crewai.llm import LLM
|
||||
|
||||
# Create a custom LLMGuardrail with specific LLM
|
||||
custom_guardrail = LLMGuardrail(
|
||||
description="Ensure the response contains exactly 5 bullet points with proper citations",
|
||||
llm=LLM(model="gpt-4o-mini")
|
||||
)
|
||||
|
||||
task = Task(
|
||||
description="Research AI safety measures",
|
||||
expected_output="A detailed analysis with bullet points",
|
||||
agent=research_agent,
|
||||
guardrail=custom_guardrail
|
||||
)
|
||||
```
|
||||
|
||||
**Note**: When you use a string guardrail, CrewAI automatically creates an `LLMGuardrail` instance using your task's agent LLM. Using `LLMGuardrail` directly gives you more control over the validation process and LLM selection.
|
||||
|
||||
### Error Handling Best Practices
|
||||
|
||||
@@ -798,166 +877,7 @@ While creating and executing tasks, certain validation mechanisms are in place t
|
||||
|
||||
These validations help in maintaining the consistency and reliability of task executions within the crewAI framework.
|
||||
|
||||
## Task Guardrails
|
||||
|
||||
Task guardrails provide a powerful way to validate, transform, or filter task outputs before they are passed to the next task. Guardrails are optional functions that execute before the next task starts, allowing you to ensure that task outputs meet specific requirements or formats.
|
||||
|
||||
### Basic Usage
|
||||
|
||||
#### Define your own logic to validate
|
||||
|
||||
```python Code
|
||||
from typing import Tuple, Union
|
||||
from crewai import Task
|
||||
|
||||
def validate_json_output(result: str) -> Tuple[bool, Union[dict, str]]:
|
||||
"""Validate that the output is valid JSON."""
|
||||
try:
|
||||
json_data = json.loads(result)
|
||||
return (True, json_data)
|
||||
except json.JSONDecodeError:
|
||||
return (False, "Output must be valid JSON")
|
||||
|
||||
task = Task(
|
||||
description="Generate JSON data",
|
||||
expected_output="Valid JSON object",
|
||||
guardrail=validate_json_output
|
||||
)
|
||||
```
|
||||
|
||||
#### Leverage a no-code approach for validation
|
||||
|
||||
```python Code
|
||||
from crewai import Task
|
||||
|
||||
task = Task(
|
||||
description="Generate JSON data",
|
||||
expected_output="Valid JSON object",
|
||||
guardrail="Ensure the response is a valid JSON object"
|
||||
)
|
||||
```
|
||||
|
||||
#### Using YAML
|
||||
|
||||
```yaml
|
||||
research_task:
|
||||
...
|
||||
guardrail: make sure each bullet contains a minimum of 100 words
|
||||
...
|
||||
```
|
||||
|
||||
```python Code
|
||||
@CrewBase
|
||||
class InternalCrew:
|
||||
agents_config = "config/agents.yaml"
|
||||
tasks_config = "config/tasks.yaml"
|
||||
|
||||
...
|
||||
@task
|
||||
def research_task(self):
|
||||
return Task(config=self.tasks_config["research_task"]) # type: ignore[index]
|
||||
...
|
||||
```
|
||||
|
||||
|
||||
#### Use custom models for code generation
|
||||
|
||||
```python Code
|
||||
from crewai import Task
|
||||
from crewai.llm import LLM
|
||||
|
||||
task = Task(
|
||||
description="Generate JSON data",
|
||||
expected_output="Valid JSON object",
|
||||
guardrail=LLMGuardrail(
|
||||
description="Ensure the response is a valid JSON object",
|
||||
llm=LLM(model="gpt-4o-mini"),
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
### How Guardrails Work
|
||||
|
||||
1. **Optional Attribute**: Guardrails are an optional attribute at the task level, allowing you to add validation only where needed.
|
||||
2. **Execution Timing**: The guardrail function is executed before the next task starts, ensuring valid data flow between tasks.
|
||||
3. **Return Format**: Guardrails must return a tuple of `(success, data)`:
|
||||
- If `success` is `True`, `data` is the validated/transformed result
|
||||
- If `success` is `False`, `data` is the error message
|
||||
4. **Result Routing**:
|
||||
- On success (`True`), the result is automatically passed to the next task
|
||||
- On failure (`False`), the error is sent back to the agent to generate a new answer
|
||||
|
||||
### Common Use Cases
|
||||
|
||||
#### Data Format Validation
|
||||
```python Code
|
||||
def validate_email_format(result: str) -> Tuple[bool, Union[str, str]]:
|
||||
"""Ensure the output contains a valid email address."""
|
||||
import re
|
||||
email_pattern = r'^[\w\.-]+@[\w\.-]+\.\w+$'
|
||||
if re.match(email_pattern, result.strip()):
|
||||
return (True, result.strip())
|
||||
return (False, "Output must be a valid email address")
|
||||
```
|
||||
|
||||
#### Content Filtering
|
||||
```python Code
|
||||
def filter_sensitive_info(result: str) -> Tuple[bool, Union[str, str]]:
|
||||
"""Remove or validate sensitive information."""
|
||||
sensitive_patterns = ['SSN:', 'password:', 'secret:']
|
||||
for pattern in sensitive_patterns:
|
||||
if pattern.lower() in result.lower():
|
||||
return (False, f"Output contains sensitive information ({pattern})")
|
||||
return (True, result)
|
||||
```
|
||||
|
||||
#### Data Transformation
|
||||
```python Code
|
||||
def normalize_phone_number(result: str) -> Tuple[bool, Union[str, str]]:
|
||||
"""Ensure phone numbers are in a consistent format."""
|
||||
import re
|
||||
digits = re.sub(r'\D', '', result)
|
||||
if len(digits) == 10:
|
||||
formatted = f"({digits[:3]}) {digits[3:6]}-{digits[6:]}"
|
||||
return (True, formatted)
|
||||
return (False, "Output must be a 10-digit phone number")
|
||||
```
|
||||
|
||||
### Advanced Features
|
||||
|
||||
#### Chaining Multiple Validations
|
||||
```python Code
|
||||
def chain_validations(*validators):
|
||||
"""Chain multiple validators together."""
|
||||
def combined_validator(result):
|
||||
for validator in validators:
|
||||
success, data = validator(result)
|
||||
if not success:
|
||||
return (False, data)
|
||||
result = data
|
||||
return (True, result)
|
||||
return combined_validator
|
||||
|
||||
# Usage
|
||||
task = Task(
|
||||
description="Get user contact info",
|
||||
expected_output="Email and phone",
|
||||
guardrail=chain_validations(
|
||||
validate_email_format,
|
||||
filter_sensitive_info
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
#### Custom Retry Logic
|
||||
```python Code
|
||||
task = Task(
|
||||
description="Generate data",
|
||||
expected_output="Valid data",
|
||||
guardrail=validate_data,
|
||||
max_retries=5 # Override default retry limit
|
||||
)
|
||||
```
|
||||
|
||||
## Creating Directories when Saving Files
|
||||
|
||||
|
||||
@@ -76,6 +76,7 @@ Exemplo:
|
||||
crewai train -n 10 -f my_training_data.pkl
|
||||
```
|
||||
|
||||
```python
|
||||
# Exemplo de uso programático do comando train
|
||||
n_iterations = 2
|
||||
inputs = {"topic": "Treinamento CrewAI"}
|
||||
@@ -89,6 +90,7 @@ try:
|
||||
)
|
||||
except Exception as e:
|
||||
raise Exception(f"Ocorreu um erro ao treinar a crew: {e}")
|
||||
```
|
||||
|
||||
### 4. Replay
|
||||
|
||||
|
||||
@@ -57,6 +57,7 @@ crew = Crew(
|
||||
| **Saída JSON** _(opcional)_ | `output_json` | `Optional[Type[BaseModel]]` | Um modelo Pydantic para estruturar a saída em JSON. |
|
||||
| **Output Pydantic** _(opcional)_ | `output_pydantic` | `Optional[Type[BaseModel]]` | Um modelo Pydantic para a saída da tarefa. |
|
||||
| **Callback** _(opcional)_ | `callback` | `Optional[Any]` | Função/objeto a ser executado após a conclusão da tarefa. |
|
||||
| **Guardrail** _(opcional)_ | `guardrail` | `Optional[Union[Callable, str]]` | Função ou descrição em string para validar a saída da tarefa antes de prosseguir para a próxima tarefa. |
|
||||
|
||||
## Criando Tarefas
|
||||
|
||||
@@ -86,6 +87,7 @@ research_task:
|
||||
expected_output: >
|
||||
Uma lista com 10 tópicos em bullet points das informações mais relevantes sobre {topic}
|
||||
agent: researcher
|
||||
guardrail: garanta que cada bullet point contenha no mínimo 100 palavras
|
||||
|
||||
reporting_task:
|
||||
description: >
|
||||
@@ -330,9 +332,13 @@ analysis_task = Task(
|
||||
|
||||
Guardrails (trilhas de proteção) de tarefas fornecem uma maneira de validar e transformar as saídas das tarefas antes que elas sejam passadas para a próxima tarefa. Esse recurso assegura a qualidade dos dados e oferece feedback aos agentes quando sua saída não atende a critérios específicos.
|
||||
|
||||
### Usando Guardrails em Tarefas
|
||||
**Guardrails podem ser definidos de duas maneiras:**
|
||||
1. **Guardrails baseados em função**: Funções Python que implementam lógica de validação customizada
|
||||
2. **Guardrails baseados em string**: Descrições em linguagem natural que são automaticamente convertidas em validação baseada em LLM
|
||||
|
||||
Para adicionar um guardrail a uma tarefa, forneça uma função de validação por meio do parâmetro `guardrail`:
|
||||
### Guardrails Baseados em Função
|
||||
|
||||
Para adicionar um guardrail baseado em função a uma tarefa, forneça uma função de validação por meio do parâmetro `guardrail`:
|
||||
|
||||
```python Code
|
||||
from typing import Tuple, Union, Dict, Any
|
||||
@@ -370,9 +376,82 @@ blog_task = Task(
|
||||
- Em caso de sucesso: retorna uma tupla `(True, resultado_validado)`
|
||||
- Em caso de falha: retorna uma tupla `(False, "mensagem de erro explicando a falha")`
|
||||
|
||||
### LLMGuardrail
|
||||
### Guardrails Baseados em String
|
||||
|
||||
A classe `LLMGuardrail` oferece um mecanismo robusto para validação das saídas das tarefas.
|
||||
Guardrails baseados em string permitem que você descreva critérios de validação em linguagem natural. Quando você fornece uma string em vez de uma função, o CrewAI automaticamente a converte em um `LLMGuardrail` que usa um agente de IA para validar a saída da tarefa.
|
||||
|
||||
#### Usando Guardrails de String em Python
|
||||
|
||||
```python Code
|
||||
from crewai import Task
|
||||
|
||||
# Guardrail simples baseado em string
|
||||
blog_task = Task(
|
||||
description="Escreva um post de blog sobre IA",
|
||||
expected_output="Um post de blog com menos de 200 palavras",
|
||||
agent=blog_agent,
|
||||
guardrail="Garanta que o post do blog tenha menos de 200 palavras e inclua exemplos práticos"
|
||||
)
|
||||
|
||||
# Critérios de validação mais complexos
|
||||
research_task = Task(
|
||||
description="Pesquise tendências de IA para 2025",
|
||||
expected_output="Um relatório abrangente de pesquisa",
|
||||
agent=research_agent,
|
||||
guardrail="Garanta que cada descoberta inclua uma fonte confiável e seja respaldada por dados recentes de 2024-2025"
|
||||
)
|
||||
```
|
||||
|
||||
#### Usando Guardrails de String em YAML
|
||||
|
||||
```yaml
|
||||
research_task:
|
||||
description: Pesquise os últimos desenvolvimentos em IA
|
||||
expected_output: Uma lista de 10 bullet points sobre IA
|
||||
agent: researcher
|
||||
guardrail: garanta que cada bullet point contenha no mínimo 100 palavras
|
||||
|
||||
validation_task:
|
||||
description: Valide os achados da pesquisa
|
||||
expected_output: Um relatório de validação
|
||||
agent: validator
|
||||
guardrail: confirme que todas as fontes são de publicações respeitáveis e publicadas nos últimos 2 anos
|
||||
```
|
||||
|
||||
#### Como Funcionam os Guardrails de String
|
||||
|
||||
Quando você fornece um guardrail de string, o CrewAI automaticamente:
|
||||
1. Cria uma instância `LLMGuardrail` usando a string como critério de validação
|
||||
2. Usa o LLM do agente da tarefa para alimentar a validação
|
||||
3. Cria um agente temporário de validação que verifica a saída contra seus critérios
|
||||
4. Retorna feedback detalhado se a validação falhar
|
||||
|
||||
Esta abordagem é ideal quando você quer usar linguagem natural para descrever regras de validação sem escrever funções de validação customizadas.
|
||||
|
||||
### Classe LLMGuardrail
|
||||
|
||||
A classe `LLMGuardrail` é o mecanismo subjacente que alimenta os guardrails baseados em string. Você também pode usá-la diretamente para maior controle avançado:
|
||||
|
||||
```python Code
|
||||
from crewai import Task
|
||||
from crewai.tasks.llm_guardrail import LLMGuardrail
|
||||
from crewai.llm import LLM
|
||||
|
||||
# Crie um LLMGuardrail customizado com LLM específico
|
||||
custom_guardrail = LLMGuardrail(
|
||||
description="Garanta que a resposta contenha exatamente 5 bullet points com citações adequadas",
|
||||
llm=LLM(model="gpt-4o-mini")
|
||||
)
|
||||
|
||||
task = Task(
|
||||
description="Pesquise medidas de segurança em IA",
|
||||
expected_output="Uma análise detalhada com bullet points",
|
||||
agent=research_agent,
|
||||
guardrail=custom_guardrail
|
||||
)
|
||||
```
|
||||
|
||||
**Nota**: Quando você usa um guardrail de string, o CrewAI automaticamente cria uma instância `LLMGuardrail` usando o LLM do agente da sua tarefa. Usar `LLMGuardrail` diretamente lhe dá mais controle sobre o processo de validação e seleção de LLM.
|
||||
|
||||
### Melhores Práticas de Tratamento de Erros
|
||||
|
||||
|
||||
@@ -210,7 +210,6 @@ class Agent(BaseAgent):
|
||||
sources=self.knowledge_sources,
|
||||
embedder=self.embedder,
|
||||
collection_name=self.role,
|
||||
storage=self.knowledge_storage or None,
|
||||
)
|
||||
self.knowledge.add_sources()
|
||||
except (TypeError, ValueError) as e:
|
||||
@@ -341,7 +340,8 @@ class Agent(BaseAgent):
|
||||
self.knowledge_config.model_dump() if self.knowledge_config else {}
|
||||
)
|
||||
|
||||
if self.knowledge:
|
||||
|
||||
if self.knowledge or (self.crew and self.crew.knowledge):
|
||||
crewai_event_bus.emit(
|
||||
self,
|
||||
event=KnowledgeRetrievalStartedEvent(
|
||||
@@ -353,25 +353,28 @@ class Agent(BaseAgent):
|
||||
task_prompt
|
||||
)
|
||||
if self.knowledge_search_query:
|
||||
agent_knowledge_snippets = self.knowledge.query(
|
||||
[self.knowledge_search_query], **knowledge_config
|
||||
)
|
||||
if agent_knowledge_snippets:
|
||||
self.agent_knowledge_context = extract_knowledge_context(
|
||||
agent_knowledge_snippets
|
||||
)
|
||||
if self.agent_knowledge_context:
|
||||
task_prompt += self.agent_knowledge_context
|
||||
if self.crew:
|
||||
knowledge_snippets = self.crew.query_knowledge(
|
||||
# Quering agent specific knowledge
|
||||
if self.knowledge:
|
||||
agent_knowledge_snippets = self.knowledge.query(
|
||||
[self.knowledge_search_query], **knowledge_config
|
||||
)
|
||||
if knowledge_snippets:
|
||||
self.crew_knowledge_context = extract_knowledge_context(
|
||||
knowledge_snippets
|
||||
if agent_knowledge_snippets:
|
||||
self.agent_knowledge_context = extract_knowledge_context(
|
||||
agent_knowledge_snippets
|
||||
)
|
||||
if self.crew_knowledge_context:
|
||||
task_prompt += self.crew_knowledge_context
|
||||
if self.agent_knowledge_context:
|
||||
task_prompt += self.agent_knowledge_context
|
||||
|
||||
# Quering crew specific knowledge
|
||||
knowledge_snippets = self.crew.query_knowledge(
|
||||
[self.knowledge_search_query], **knowledge_config
|
||||
)
|
||||
if knowledge_snippets:
|
||||
self.crew_knowledge_context = extract_knowledge_context(
|
||||
knowledge_snippets
|
||||
)
|
||||
if self.crew_knowledge_context:
|
||||
task_prompt += self.crew_knowledge_context
|
||||
|
||||
crewai_event_bus.emit(
|
||||
self,
|
||||
|
||||
@@ -41,6 +41,7 @@ from crewai.agents.parser import (
|
||||
)
|
||||
from crewai.flow.flow_trackable import FlowTrackable
|
||||
from crewai.llm import LLM
|
||||
from crewai.llms.base_llm import BaseLLM
|
||||
from crewai.tools.base_tool import BaseTool
|
||||
from crewai.tools.structured_tool import CrewStructuredTool
|
||||
from crewai.utilities import I18N
|
||||
@@ -209,8 +210,8 @@ class LiteAgent(FlowTrackable, BaseModel):
|
||||
def setup_llm(self):
|
||||
"""Set up the LLM and other components after initialization."""
|
||||
self.llm = create_llm(self.llm)
|
||||
if not isinstance(self.llm, LLM):
|
||||
raise ValueError("Unable to create LLM instance")
|
||||
if not isinstance(self.llm, BaseLLM):
|
||||
raise ValueError(f"Expected LLM instance of type BaseLLM, got {type(self.llm).__name__}")
|
||||
|
||||
# Initialize callbacks
|
||||
token_callback = TokenCalcHandler(token_cost_process=self._token_process)
|
||||
@@ -232,7 +233,8 @@ class LiteAgent(FlowTrackable, BaseModel):
|
||||
elif isinstance(self.guardrail, str):
|
||||
from crewai.tasks.llm_guardrail import LLMGuardrail
|
||||
|
||||
assert isinstance(self.llm, LLM)
|
||||
if not isinstance(self.llm, BaseLLM):
|
||||
raise TypeError(f"Guardrail requires LLM instance of type BaseLLM, got {type(self.llm).__name__}")
|
||||
|
||||
self._guardrail = LLMGuardrail(description=self.guardrail, llm=self.llm)
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ Classes:
|
||||
|
||||
from typing import Any, Optional, Tuple
|
||||
|
||||
from crewai.llm import LLM
|
||||
from crewai.llms.base_llm import BaseLLM
|
||||
from crewai.tasks.task_output import TaskOutput
|
||||
from crewai.utilities.logger import Logger
|
||||
|
||||
@@ -47,7 +47,7 @@ class HallucinationGuardrail:
|
||||
def __init__(
|
||||
self,
|
||||
context: str,
|
||||
llm: LLM,
|
||||
llm: BaseLLM,
|
||||
threshold: Optional[float] = None,
|
||||
tool_response: str = "",
|
||||
):
|
||||
@@ -60,7 +60,7 @@ class HallucinationGuardrail:
|
||||
tool_response: Optional tool response information that would be used in evaluation.
|
||||
"""
|
||||
self.context = context
|
||||
self.llm: LLM = llm
|
||||
self.llm: BaseLLM = llm
|
||||
self.threshold = threshold
|
||||
self.tool_response = tool_response
|
||||
self._logger = Logger(verbose=True)
|
||||
|
||||
@@ -1,10 +1,9 @@
|
||||
from typing import Any, Optional, Tuple
|
||||
from typing import Any, Tuple
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from crewai.agent import Agent, LiteAgentOutput
|
||||
from crewai.llm import LLM
|
||||
from crewai.task import Task
|
||||
from crewai.llms.base_llm import BaseLLM
|
||||
from crewai.tasks.task_output import TaskOutput
|
||||
|
||||
|
||||
@@ -32,11 +31,11 @@ class LLMGuardrail:
|
||||
def __init__(
|
||||
self,
|
||||
description: str,
|
||||
llm: LLM,
|
||||
llm: BaseLLM,
|
||||
):
|
||||
self.description = description
|
||||
|
||||
self.llm: LLM = llm
|
||||
self.llm: BaseLLM = llm
|
||||
|
||||
def _validate_output(self, task_output: TaskOutput) -> LiteAgentOutput:
|
||||
agent = Agent(
|
||||
|
||||
@@ -1896,6 +1896,80 @@ def test_agent_with_knowledge_sources_generate_search_query():
|
||||
assert "red" in result.raw.lower()
|
||||
|
||||
|
||||
@pytest.mark.vcr(record_mode='none', filter_headers=["authorization"])
|
||||
def test_agent_with_knowledge_with_no_crewai_knowledge():
|
||||
mock_knowledge = MagicMock(spec=Knowledge)
|
||||
|
||||
agent = Agent(
|
||||
role="Information Agent",
|
||||
goal="Provide information based on knowledge sources",
|
||||
backstory="You have access to specific knowledge sources.",
|
||||
llm=LLM(model="openrouter/openai/gpt-4o-mini",api_key=os.getenv('OPENROUTER_API_KEY')),
|
||||
knowledge=mock_knowledge
|
||||
)
|
||||
|
||||
# Create a task that requires the agent to use the knowledge
|
||||
task = Task(
|
||||
description="What is Vidit's favorite color?",
|
||||
expected_output="Vidit's favorclearite color.",
|
||||
agent=agent,
|
||||
)
|
||||
|
||||
crew = Crew(agents=[agent], tasks=[task])
|
||||
crew.kickoff()
|
||||
mock_knowledge.query.assert_called_once()
|
||||
|
||||
|
||||
@pytest.mark.vcr(record_mode='none', filter_headers=["authorization"])
|
||||
def test_agent_with_only_crewai_knowledge():
|
||||
mock_knowledge = MagicMock(spec=Knowledge)
|
||||
|
||||
agent = Agent(
|
||||
role="Information Agent",
|
||||
goal="Provide information based on knowledge sources",
|
||||
backstory="You have access to specific knowledge sources.",
|
||||
llm=LLM(model="openrouter/openai/gpt-4o-mini",api_key=os.getenv('OPENROUTER_API_KEY'))
|
||||
)
|
||||
|
||||
# Create a task that requires the agent to use the knowledge
|
||||
task = Task(
|
||||
description="What is Vidit's favorite color?",
|
||||
expected_output="Vidit's favorclearite color.",
|
||||
agent=agent
|
||||
)
|
||||
|
||||
crew = Crew(agents=[agent], tasks=[task],knowledge=mock_knowledge)
|
||||
crew.kickoff()
|
||||
mock_knowledge.query.assert_called_once()
|
||||
|
||||
|
||||
@pytest.mark.vcr(record_mode='none', filter_headers=["authorization"])
|
||||
def test_agent_knowledege_with_crewai_knowledge():
|
||||
crew_knowledge = MagicMock(spec=Knowledge)
|
||||
agent_knowledge = MagicMock(spec=Knowledge)
|
||||
|
||||
|
||||
agent = Agent(
|
||||
role="Information Agent",
|
||||
goal="Provide information based on knowledge sources",
|
||||
backstory="You have access to specific knowledge sources.",
|
||||
llm=LLM(model="openrouter/openai/gpt-4o-mini",api_key=os.getenv('OPENROUTER_API_KEY')),
|
||||
knowledge=agent_knowledge
|
||||
)
|
||||
|
||||
# Create a task that requires the agent to use the knowledge
|
||||
task = Task(
|
||||
description="What is Vidit's favorite color?",
|
||||
expected_output="Vidit's favorclearite color.",
|
||||
agent=agent,
|
||||
)
|
||||
|
||||
crew = Crew(agents=[agent],tasks=[task],knowledge=crew_knowledge)
|
||||
crew.kickoff()
|
||||
agent_knowledge.query.assert_called_once()
|
||||
crew_knowledge.query.assert_called_once()
|
||||
|
||||
|
||||
@pytest.mark.vcr(filter_headers=["authorization"])
|
||||
def test_litellm_auth_error_handling():
|
||||
"""Test that LiteLLM authentication errors are handled correctly and not retried."""
|
||||
|
||||
150
tests/cassettes/test_agent_knowledege_with_crewai_knowledge.yaml
Normal file
150
tests/cassettes/test_agent_knowledege_with_crewai_knowledge.yaml
Normal file
@@ -0,0 +1,150 @@
|
||||
interactions:
|
||||
- request:
|
||||
body: '{"model": "openai/gpt-4o-mini", "messages": [{"role": "system", "content":
|
||||
"Your goal is to rewrite the user query so that it is optimized for retrieval
|
||||
from a vector database. Consider how the query will be used to find relevant
|
||||
documents, and aim to make it more specific and context-aware. \n\n Do not include
|
||||
any other text than the rewritten query, especially any preamble or postamble
|
||||
and only add expected output format if its relevant to the rewritten query.
|
||||
\n\n Focus on the key words of the intended task and to retrieve the most relevant
|
||||
information. \n\n There will be some extra context provided that might need
|
||||
to be removed such as expected_output formats structured_outputs and other instructions."},
|
||||
{"role": "user", "content": "The original query is: What is Vidit''s favorite
|
||||
color?\n\nThis is the expected criteria for your final answer: Vidit''s favorclearite
|
||||
color.\nyou MUST return the actual complete content as the final answer, not
|
||||
a summary.."}], "stream": false, "stop": ["\nObservation:"]}'
|
||||
headers:
|
||||
accept:
|
||||
- '*/*'
|
||||
accept-encoding:
|
||||
- gzip, deflate
|
||||
connection:
|
||||
- keep-alive
|
||||
content-length:
|
||||
- '1017'
|
||||
content-type:
|
||||
- application/json
|
||||
host:
|
||||
- openrouter.ai
|
||||
http-referer:
|
||||
- https://litellm.ai
|
||||
user-agent:
|
||||
- litellm/1.68.0
|
||||
x-title:
|
||||
- liteLLM
|
||||
method: POST
|
||||
uri: https://openrouter.ai/api/v1/chat/completions
|
||||
response:
|
||||
body:
|
||||
string: !!binary |
|
||||
H4sIAAAAAAAAAwAAAP//4lKAAS4AAAAA//90kE1vE0EMhv9K9V64TMrmgyadG8ceECAhhIrQarrj
|
||||
3bidHY/GTgSK9r+jpUpaJLja78djn8ARHgPlxXK72a6X6+12szhq7Id72d2V8b58/nbzQb98gkOp
|
||||
cuRIFR4fC+X3d3AYJVKChxTKgd8OxRYbWYycGQ7y8EidwaPbB7vuZCyJjCXDoasUjCL8S61Dtxfu
|
||||
SOG/n5BkKFUeFD4fUnLoObPu20pBJcNDTQoccjA+UvufLedIP+Ebh5FUw0DwJ1RJBI+gymoh20wj
|
||||
2SjPpF85sr3Rqz4cpbLRVSdJ6jUcKvUHDenM81zFeXgeTNMPB/2lRuMMM1Atlf8k9qVt1rer3WrV
|
||||
3DZwOJw5SpWxWGvyRFnnR7ybQc4/usxvHEwspBfhbun+NreRLHDSObUL3Z7iRdxM/wh9rb/c8coy
|
||||
Tb8BAAD//wMAqVt3JyMCAAA=
|
||||
headers:
|
||||
Access-Control-Allow-Origin:
|
||||
- '*'
|
||||
CF-RAY:
|
||||
- 9402cb503aec46c0-BOM
|
||||
Connection:
|
||||
- keep-alive
|
||||
Content-Encoding:
|
||||
- gzip
|
||||
Content-Type:
|
||||
- application/json
|
||||
Date:
|
||||
- Thu, 15 May 2025 12:56:14 GMT
|
||||
Server:
|
||||
- cloudflare
|
||||
Transfer-Encoding:
|
||||
- chunked
|
||||
Vary:
|
||||
- Accept-Encoding
|
||||
x-clerk-auth-message:
|
||||
- Invalid JWT form. A JWT consists of three parts separated by dots. (reason=token-invalid,
|
||||
token-carrier=header)
|
||||
x-clerk-auth-reason:
|
||||
- token-invalid
|
||||
x-clerk-auth-status:
|
||||
- signed-out
|
||||
status:
|
||||
code: 200
|
||||
message: OK
|
||||
- request:
|
||||
body: '{"model": "openai/gpt-4o-mini", "messages": [{"role": "system", "content":
|
||||
"You are Information Agent. You have access to specific knowledge sources.\nYour
|
||||
personal goal is: Provide information based on knowledge sources\nTo give my
|
||||
best complete final answer to the task respond using the exact following format:\n\nThought:
|
||||
I now can give a great answer\nFinal Answer: Your final answer must be the great
|
||||
and the most complete as possible, it must be outcome described.\n\nI MUST use
|
||||
these formats, my job depends on it!"}, {"role": "user", "content": "\nCurrent
|
||||
Task: What is Vidit''s favorite color?\n\nThis is the expected criteria for
|
||||
your final answer: Vidit''s favorclearite color.\nyou MUST return the actual
|
||||
complete content as the final answer, not a summary.\n\nBegin! This is VERY
|
||||
important to you, use the tools available and give your best Final Answer, your
|
||||
job depends on it!\n\nThought:"}], "stream": false, "stop": ["\nObservation:"]}'
|
||||
headers:
|
||||
accept:
|
||||
- '*/*'
|
||||
accept-encoding:
|
||||
- gzip, deflate
|
||||
connection:
|
||||
- keep-alive
|
||||
content-length:
|
||||
- '951'
|
||||
content-type:
|
||||
- application/json
|
||||
host:
|
||||
- openrouter.ai
|
||||
http-referer:
|
||||
- https://litellm.ai
|
||||
user-agent:
|
||||
- litellm/1.68.0
|
||||
x-title:
|
||||
- liteLLM
|
||||
method: POST
|
||||
uri: https://openrouter.ai/api/v1/chat/completions
|
||||
response:
|
||||
body:
|
||||
string: !!binary |
|
||||
H4sIAAAAAAAAAwAAAP//4lKAAS4AAAAA///iQjABAAAA//90kE9rG0EMxb/K8C69jNON7WJ7boFS
|
||||
CD2ENm2g/1jGs/Ja7aw0zIydBuPvXjbBcQrtUU9P0u/pAO7g0JNMLhfzxexytli8mdy8r7c6/3Lb
|
||||
v13eff00088fPj7AImXdc0cZDjeJ5OoaFoN2FOGgicTz6z7VyVwnAwvDQtc/KVQ4hK2vF0GHFKmy
|
||||
CixCJl+pgzuftQhb5UAF7tsBUfuUdV3gZBejxYaFy7bN5IsKHErVBAvxlffU/qfL0tFvuMZioFJ8
|
||||
T3AHZI0EB18Kl+qljjQqlWQkvTai9yZ4MT3vyXjTj6DGS7mnbMx3ecfio7l6rJ25447rq2I2fq+Z
|
||||
K5mgUbPhYtZxRxewyLTZFR9PMZ4IWfon4Xj8YVEeSqVhzNBTTpkfQTapbWar6XI6bVYNLHYn/JR1
|
||||
SLWt+oukjP9rRv7Ta8/6yqJq9fGsLFf27+m2o+o5lnFt8GFL3bO5Of5j60v/c5AXI8fjHwAAAP//
|
||||
AwDEkP8dZgIAAA==
|
||||
headers:
|
||||
Access-Control-Allow-Origin:
|
||||
- '*'
|
||||
CF-RAY:
|
||||
- 9402cb55c9fe46c0-BOM
|
||||
Connection:
|
||||
- keep-alive
|
||||
Content-Encoding:
|
||||
- gzip
|
||||
Content-Type:
|
||||
- application/json
|
||||
Date:
|
||||
- Thu, 15 May 2025 12:56:15 GMT
|
||||
Server:
|
||||
- cloudflare
|
||||
Transfer-Encoding:
|
||||
- chunked
|
||||
Vary:
|
||||
- Accept-Encoding
|
||||
x-clerk-auth-message:
|
||||
- Invalid JWT form. A JWT consists of three parts separated by dots. (reason=token-invalid,
|
||||
token-carrier=header)
|
||||
x-clerk-auth-reason:
|
||||
- token-invalid
|
||||
x-clerk-auth-status:
|
||||
- signed-out
|
||||
status:
|
||||
code: 200
|
||||
message: OK
|
||||
version: 1
|
||||
@@ -0,0 +1,151 @@
|
||||
interactions:
|
||||
- request:
|
||||
body: '{"model": "openai/gpt-4o-mini", "messages": [{"role": "system", "content":
|
||||
"Your goal is to rewrite the user query so that it is optimized for retrieval
|
||||
from a vector database. Consider how the query will be used to find relevant
|
||||
documents, and aim to make it more specific and context-aware. \n\n Do not include
|
||||
any other text than the rewritten query, especially any preamble or postamble
|
||||
and only add expected output format if its relevant to the rewritten query.
|
||||
\n\n Focus on the key words of the intended task and to retrieve the most relevant
|
||||
information. \n\n There will be some extra context provided that might need
|
||||
to be removed such as expected_output formats structured_outputs and other instructions."},
|
||||
{"role": "user", "content": "The original query is: What is Vidit''s favorite
|
||||
color?\n\nThis is the expected criteria for your final answer: Vidit''s favorclearite
|
||||
color.\nyou MUST return the actual complete content as the final answer, not
|
||||
a summary.."}], "stream": false, "stop": ["\nObservation:"]}'
|
||||
headers:
|
||||
accept:
|
||||
- '*/*'
|
||||
accept-encoding:
|
||||
- gzip, deflate
|
||||
connection:
|
||||
- keep-alive
|
||||
content-length:
|
||||
- '1017'
|
||||
content-type:
|
||||
- application/json
|
||||
host:
|
||||
- openrouter.ai
|
||||
http-referer:
|
||||
- https://litellm.ai
|
||||
user-agent:
|
||||
- litellm/1.68.0
|
||||
x-title:
|
||||
- liteLLM
|
||||
method: POST
|
||||
uri: https://openrouter.ai/api/v1/chat/completions
|
||||
response:
|
||||
body:
|
||||
string: !!binary |
|
||||
H4sIAAAAAAAAAwAAAP//4lKAAS4AAAAA//90kE1vE0EMhv9K9V64TGCbNGQ7N46gIg6IXhBaTWed
|
||||
Xbez49HYiaii/e9oqRKKBFf7/XjsE7iHx0B5db272W2uN++b3ep585k+jcmo/XqnYXvX5m/3cChV
|
||||
jtxThceXQvnDRzhM0lOChxTKgd8NxVY3spo4Mxzk4ZGiwSOOwd5GmUoiY8lwiJWCUQ9/qW0d4igc
|
||||
SeG/n5BkKFUeFD4fUnLYc2Ydu0pBJcNDTQoccjA+UvefLeeefsI3DhOphoHgT6iSCB5BldVCtoVG
|
||||
slFeSO+5Z3ujV/twlMpGV1GSVDhU2h80pDPOSxPn4WUwzz8c9FmNpoVloFoq/w7cl67Z3K7b9bq5
|
||||
beBwOGOUKlOxzuSJsi5/2C4c5xdd5lsHEwvpj7Bt3N/mricLnHRJjSGO1F/EzfyP0Nf6yx2vLPP8
|
||||
CwAA//8DAOHu/cIiAgAA
|
||||
headers:
|
||||
Access-Control-Allow-Origin:
|
||||
- '*'
|
||||
CF-RAY:
|
||||
- 9402c73df9d8859c-BOM
|
||||
Connection:
|
||||
- keep-alive
|
||||
Content-Encoding:
|
||||
- gzip
|
||||
Content-Type:
|
||||
- application/json
|
||||
Date:
|
||||
- Thu, 15 May 2025 12:53:27 GMT
|
||||
Server:
|
||||
- cloudflare
|
||||
Transfer-Encoding:
|
||||
- chunked
|
||||
Vary:
|
||||
- Accept-Encoding
|
||||
x-clerk-auth-message:
|
||||
- Invalid JWT form. A JWT consists of three parts separated by dots. (reason=token-invalid,
|
||||
token-carrier=header)
|
||||
x-clerk-auth-reason:
|
||||
- token-invalid
|
||||
x-clerk-auth-status:
|
||||
- signed-out
|
||||
status:
|
||||
code: 200
|
||||
message: OK
|
||||
- request:
|
||||
body: '{"model": "openai/gpt-4o-mini", "messages": [{"role": "system", "content":
|
||||
"You are Information Agent. You have access to specific knowledge sources.\nYour
|
||||
personal goal is: Provide information based on knowledge sources\nTo give my
|
||||
best complete final answer to the task respond using the exact following format:\n\nThought:
|
||||
I now can give a great answer\nFinal Answer: Your final answer must be the great
|
||||
and the most complete as possible, it must be outcome described.\n\nI MUST use
|
||||
these formats, my job depends on it!"}, {"role": "user", "content": "\nCurrent
|
||||
Task: What is Vidit''s favorite color?\n\nThis is the expected criteria for
|
||||
your final answer: Vidit''s favorclearite color.\nyou MUST return the actual
|
||||
complete content as the final answer, not a summary.\n\nBegin! This is VERY
|
||||
important to you, use the tools available and give your best Final Answer, your
|
||||
job depends on it!\n\nThought:"}], "stream": false, "stop": ["\nObservation:"]}'
|
||||
headers:
|
||||
accept:
|
||||
- '*/*'
|
||||
accept-encoding:
|
||||
- gzip, deflate
|
||||
connection:
|
||||
- keep-alive
|
||||
content-length:
|
||||
- '951'
|
||||
content-type:
|
||||
- application/json
|
||||
host:
|
||||
- openrouter.ai
|
||||
http-referer:
|
||||
- https://litellm.ai
|
||||
user-agent:
|
||||
- litellm/1.68.0
|
||||
x-title:
|
||||
- liteLLM
|
||||
method: POST
|
||||
uri: https://openrouter.ai/api/v1/chat/completions
|
||||
response:
|
||||
body:
|
||||
string: !!binary |
|
||||
H4sIAAAAAAAAAwAAAP//4lKAAS4AAAAA///iQjABAAAA//90kUGPEzEMhf+K5QuXdJmlpbvkthIg
|
||||
emFXQoIDoMpNPFNDJo6STLul6n9H09KyIDjmxc9+/rxH8Wix4zi5vpndTK+n8+Z2wo9vXj28fHff
|
||||
vW4+PNT5j1l6/wkNpqwb8ZzR4n3ieLdAg716DmhRE0eS512qk5lOeomCBnX1jV1Fi25N9cppnwJX
|
||||
0YgGXWaq7NH+HmvQrVUcF7Sf9xi0S1lXBW0cQjDYSpSyXmamohEtlqoJDUaqsuHlf34len5E2xjs
|
||||
uRTqGO0eswZGi1SKlEqxjmk0Vo5j0gVE3YKjCJ1sGAi6MShQLFvOAF/iW4kU4O74tvBRvNRnBVra
|
||||
aJbK4DRoBikQtcJWPIcdeHVDz7GyB4mQhlUQF3ZAG5JAq8BQdMiOi4GisBiHj+ZftIHA87hePeY5
|
||||
5cjcUfYSO1hLgZLYSSvurxRXaDBzOxQKZ4gnPhK7k3A4fDVYdqVyPxLsOKcsRwxtWvoVOZo3vm3Q
|
||||
4HCGl7L2qS6rfudYxus1I73zYS/69NZg1UrhorwYD/yHe+m5koQytnXk1uwvxc3hH12f1l8WeWI5
|
||||
HH4CAAD//wMAhZKqO+QCAAA=
|
||||
headers:
|
||||
Access-Control-Allow-Origin:
|
||||
- '*'
|
||||
CF-RAY:
|
||||
- 9402c7459f3f859c-BOM
|
||||
Connection:
|
||||
- keep-alive
|
||||
Content-Encoding:
|
||||
- gzip
|
||||
Content-Type:
|
||||
- application/json
|
||||
Date:
|
||||
- Thu, 15 May 2025 12:53:28 GMT
|
||||
Server:
|
||||
- cloudflare
|
||||
Transfer-Encoding:
|
||||
- chunked
|
||||
Vary:
|
||||
- Accept-Encoding
|
||||
x-clerk-auth-message:
|
||||
- Invalid JWT form. A JWT consists of three parts separated by dots. (reason=token-invalid,
|
||||
token-carrier=header)
|
||||
x-clerk-auth-reason:
|
||||
- token-invalid
|
||||
x-clerk-auth-status:
|
||||
- signed-out
|
||||
status:
|
||||
code: 200
|
||||
message: OK
|
||||
version: 1
|
||||
150
tests/cassettes/test_agent_with_only_crewai_knowledge.yaml
Normal file
150
tests/cassettes/test_agent_with_only_crewai_knowledge.yaml
Normal file
@@ -0,0 +1,150 @@
|
||||
interactions:
|
||||
- request:
|
||||
body: '{"model": "openai/gpt-4o-mini", "messages": [{"role": "system", "content":
|
||||
"Your goal is to rewrite the user query so that it is optimized for retrieval
|
||||
from a vector database. Consider how the query will be used to find relevant
|
||||
documents, and aim to make it more specific and context-aware. \n\n Do not include
|
||||
any other text than the rewritten query, especially any preamble or postamble
|
||||
and only add expected output format if its relevant to the rewritten query.
|
||||
\n\n Focus on the key words of the intended task and to retrieve the most relevant
|
||||
information. \n\n There will be some extra context provided that might need
|
||||
to be removed such as expected_output formats structured_outputs and other instructions."},
|
||||
{"role": "user", "content": "The original query is: What is Vidit''s favorite
|
||||
color?\n\nThis is the expected criteria for your final answer: Vidit''s favorclearite
|
||||
color.\nyou MUST return the actual complete content as the final answer, not
|
||||
a summary.."}], "stream": false, "stop": ["\nObservation:"]}'
|
||||
headers:
|
||||
accept:
|
||||
- '*/*'
|
||||
accept-encoding:
|
||||
- gzip, deflate
|
||||
connection:
|
||||
- keep-alive
|
||||
content-length:
|
||||
- '1017'
|
||||
content-type:
|
||||
- application/json
|
||||
host:
|
||||
- openrouter.ai
|
||||
http-referer:
|
||||
- https://litellm.ai
|
||||
user-agent:
|
||||
- litellm/1.68.0
|
||||
x-title:
|
||||
- liteLLM
|
||||
method: POST
|
||||
uri: https://openrouter.ai/api/v1/chat/completions
|
||||
response:
|
||||
body:
|
||||
string: !!binary |
|
||||
H4sIAAAAAAAAAwAAAP//4lKAAS4AAAAA//90kE1PIzEMhv8Kei97Sdnplwq5gTgAF8ShcFitRmnG
|
||||
nTFk4ihxq11V899Xs6gFJLja78djH8ANLFqKk+lqsZpP56vpYqJhublfP1eP65v1i79Lt9fdMwxS
|
||||
lj03lGHxkChe3cGgl4YCLCRRdPyzTTpZyKTnyDCQzQt5hYXvnJ576VMgZYkw8JmcUgP7XmvgO2FP
|
||||
BfbXAUHalGVTYOMuBIMtRy5dnckVibAoKgkG0Snvqf5my7GhP7CVQU+luJZgD8gSCBauFC7qoo40
|
||||
EpXiSPrEDeuPcrZ1e8msdOYlSIZBpu2uuHDEeWvi2L4NhuG3QflblPqRpaWcMv8P3Ka6ml/OLmaz
|
||||
6rKCwe6IkbL0SWuVV4pl/MNy5Di+6DRfGqioC+/Ci8p8NtcNqeNQxlTvfEfNSVwNX4R+1J/u+GAZ
|
||||
hn8AAAD//wMAIwJ79CICAAA=
|
||||
headers:
|
||||
Access-Control-Allow-Origin:
|
||||
- '*'
|
||||
CF-RAY:
|
||||
- 9402c9db99ec4722-BOM
|
||||
Connection:
|
||||
- keep-alive
|
||||
Content-Encoding:
|
||||
- gzip
|
||||
Content-Type:
|
||||
- application/json
|
||||
Date:
|
||||
- Thu, 15 May 2025 12:55:14 GMT
|
||||
Server:
|
||||
- cloudflare
|
||||
Transfer-Encoding:
|
||||
- chunked
|
||||
Vary:
|
||||
- Accept-Encoding
|
||||
x-clerk-auth-message:
|
||||
- Invalid JWT form. A JWT consists of three parts separated by dots. (reason=token-invalid,
|
||||
token-carrier=header)
|
||||
x-clerk-auth-reason:
|
||||
- token-invalid
|
||||
x-clerk-auth-status:
|
||||
- signed-out
|
||||
status:
|
||||
code: 200
|
||||
message: OK
|
||||
- request:
|
||||
body: '{"model": "openai/gpt-4o-mini", "messages": [{"role": "system", "content":
|
||||
"You are Information Agent. You have access to specific knowledge sources.\nYour
|
||||
personal goal is: Provide information based on knowledge sources\nTo give my
|
||||
best complete final answer to the task respond using the exact following format:\n\nThought:
|
||||
I now can give a great answer\nFinal Answer: Your final answer must be the great
|
||||
and the most complete as possible, it must be outcome described.\n\nI MUST use
|
||||
these formats, my job depends on it!"}, {"role": "user", "content": "\nCurrent
|
||||
Task: What is Vidit''s favorite color?\n\nThis is the expected criteria for
|
||||
your final answer: Vidit''s favorclearite color.\nyou MUST return the actual
|
||||
complete content as the final answer, not a summary.\n\nBegin! This is VERY
|
||||
important to you, use the tools available and give your best Final Answer, your
|
||||
job depends on it!\n\nThought:"}], "stream": false, "stop": ["\nObservation:"]}'
|
||||
headers:
|
||||
accept:
|
||||
- '*/*'
|
||||
accept-encoding:
|
||||
- gzip, deflate
|
||||
connection:
|
||||
- keep-alive
|
||||
content-length:
|
||||
- '951'
|
||||
content-type:
|
||||
- application/json
|
||||
host:
|
||||
- openrouter.ai
|
||||
http-referer:
|
||||
- https://litellm.ai
|
||||
user-agent:
|
||||
- litellm/1.68.0
|
||||
x-title:
|
||||
- liteLLM
|
||||
method: POST
|
||||
uri: https://openrouter.ai/api/v1/chat/completions
|
||||
response:
|
||||
body:
|
||||
string: !!binary |
|
||||
H4sIAAAAAAAAAwAAAP//4lKAAS4AAAAA///iQjABAAAA//90kN1qGzEQRl9FfNdyul4nday73ARy
|
||||
VUpLE2jLIu+O15NoZ4QkOy1moa/R1+uTlE1wnEB7qU/zc84cwB0cepLZfHm+XMwXy/nF7II/3d7V
|
||||
H+tOPvsS3le3d+keFjHpnjtKcPgQSa5uYDFoRwEOGkk8v+tjmZ3rbGBhWOj6ntoCh3bry1mrQwxU
|
||||
WAUWbSJfqIM7rbVot8otZbivBwTtY9J1hpNdCBYbFs7bJpHPKnDIRSMsxBfeU/OfX5aOfsBVFgPl
|
||||
7HuCOyBpIDj4nDkXL2WiUSkkE+mNEX00rRfT856MN/0EarzkR0rGfJNrFh/M1dPbmS/ccfnz63c2
|
||||
G7/XxIVMq0GT4WzWYUdnsEi02WUfjiLPjCz9czCO3y3yz1xomCx6SjHxE8omNtViVV/WdbWqYLE7
|
||||
CsSkQyxN0QeSPF2wmgyOxz3lK4uixYdTcrmyb7ubjornkKexrW+31L0UV+M/pr6ufxF51TKOfwEA
|
||||
AP//AwBybekMaAIAAA==
|
||||
headers:
|
||||
Access-Control-Allow-Origin:
|
||||
- '*'
|
||||
CF-RAY:
|
||||
- 9402c9e1b94a4722-BOM
|
||||
Connection:
|
||||
- keep-alive
|
||||
Content-Encoding:
|
||||
- gzip
|
||||
Content-Type:
|
||||
- application/json
|
||||
Date:
|
||||
- Thu, 15 May 2025 12:55:15 GMT
|
||||
Server:
|
||||
- cloudflare
|
||||
Transfer-Encoding:
|
||||
- chunked
|
||||
Vary:
|
||||
- Accept-Encoding
|
||||
x-clerk-auth-message:
|
||||
- Invalid JWT form. A JWT consists of three parts separated by dots. (reason=token-invalid,
|
||||
token-carrier=header)
|
||||
x-clerk-auth-reason:
|
||||
- token-invalid
|
||||
x-clerk-auth-status:
|
||||
- signed-out
|
||||
status:
|
||||
code: 200
|
||||
message: OK
|
||||
version: 1
|
||||
@@ -146,12 +146,12 @@ def test_lite_agent_with_tools():
|
||||
"What is the population of Tokyo and how many people would that be per square kilometer if Tokyo's area is 2,194 square kilometers?"
|
||||
)
|
||||
|
||||
assert (
|
||||
"21 million" in result.raw or "37 million" in result.raw
|
||||
), "Agent should find Tokyo's population"
|
||||
assert (
|
||||
"per square kilometer" in result.raw
|
||||
), "Agent should calculate population density"
|
||||
assert "21 million" in result.raw or "37 million" in result.raw, (
|
||||
"Agent should find Tokyo's population"
|
||||
)
|
||||
assert "per square kilometer" in result.raw, (
|
||||
"Agent should calculate population density"
|
||||
)
|
||||
|
||||
received_events = []
|
||||
|
||||
@@ -316,11 +316,17 @@ def test_sets_parent_flow_when_inside_flow():
|
||||
flow.kickoff()
|
||||
assert captured_agent.parent_flow is flow
|
||||
|
||||
|
||||
@pytest.mark.vcr(filter_headers=["authorization"])
|
||||
def test_guardrail_is_called_using_string():
|
||||
guardrail_events = defaultdict(list)
|
||||
from crewai.utilities.events import LLMGuardrailCompletedEvent, LLMGuardrailStartedEvent
|
||||
from crewai.utilities.events import (
|
||||
LLMGuardrailCompletedEvent,
|
||||
LLMGuardrailStartedEvent,
|
||||
)
|
||||
|
||||
with crewai_event_bus.scoped_handlers():
|
||||
|
||||
@crewai_event_bus.on(LLMGuardrailStartedEvent)
|
||||
def capture_guardrail_started(source, event):
|
||||
guardrail_events["started"].append(event)
|
||||
@@ -338,17 +344,26 @@ def test_guardrail_is_called_using_string():
|
||||
|
||||
result = agent.kickoff(messages="Top 10 best players in the world?")
|
||||
|
||||
assert len(guardrail_events['started']) == 2
|
||||
assert len(guardrail_events['completed']) == 2
|
||||
assert not guardrail_events['completed'][0].success
|
||||
assert guardrail_events['completed'][1].success
|
||||
assert "Here are the top 10 best soccer players in the world, focusing exclusively on Brazilian players" in result.raw
|
||||
assert len(guardrail_events["started"]) == 2
|
||||
assert len(guardrail_events["completed"]) == 2
|
||||
assert not guardrail_events["completed"][0].success
|
||||
assert guardrail_events["completed"][1].success
|
||||
assert (
|
||||
"Here are the top 10 best soccer players in the world, focusing exclusively on Brazilian players"
|
||||
in result.raw
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.vcr(filter_headers=["authorization"])
|
||||
def test_guardrail_is_called_using_callable():
|
||||
guardrail_events = defaultdict(list)
|
||||
from crewai.utilities.events import LLMGuardrailCompletedEvent, LLMGuardrailStartedEvent
|
||||
from crewai.utilities.events import (
|
||||
LLMGuardrailCompletedEvent,
|
||||
LLMGuardrailStartedEvent,
|
||||
)
|
||||
|
||||
with crewai_event_bus.scoped_handlers():
|
||||
|
||||
@crewai_event_bus.on(LLMGuardrailStartedEvent)
|
||||
def capture_guardrail_started(source, event):
|
||||
guardrail_events["started"].append(event)
|
||||
@@ -366,16 +381,22 @@ def test_guardrail_is_called_using_callable():
|
||||
|
||||
result = agent.kickoff(messages="Top 1 best players in the world?")
|
||||
|
||||
assert len(guardrail_events['started']) == 1
|
||||
assert len(guardrail_events['completed']) == 1
|
||||
assert guardrail_events['completed'][0].success
|
||||
assert len(guardrail_events["started"]) == 1
|
||||
assert len(guardrail_events["completed"]) == 1
|
||||
assert guardrail_events["completed"][0].success
|
||||
assert "Pelé - Santos, 1958" in result.raw
|
||||
|
||||
|
||||
@pytest.mark.vcr(filter_headers=["authorization"])
|
||||
def test_guardrail_reached_attempt_limit():
|
||||
guardrail_events = defaultdict(list)
|
||||
from crewai.utilities.events import LLMGuardrailCompletedEvent, LLMGuardrailStartedEvent
|
||||
from crewai.utilities.events import (
|
||||
LLMGuardrailCompletedEvent,
|
||||
LLMGuardrailStartedEvent,
|
||||
)
|
||||
|
||||
with crewai_event_bus.scoped_handlers():
|
||||
|
||||
@crewai_event_bus.on(LLMGuardrailStartedEvent)
|
||||
def capture_guardrail_started(source, event):
|
||||
guardrail_events["started"].append(event)
|
||||
@@ -388,18 +409,23 @@ def test_guardrail_reached_attempt_limit():
|
||||
role="Sports Analyst",
|
||||
goal="Gather information about the best soccer players",
|
||||
backstory="""You are an expert at gathering and organizing information. You carefully collect details and present them in a structured way.""",
|
||||
guardrail=lambda output: (False, "You are not allowed to include Brazilian players"),
|
||||
guardrail=lambda output: (
|
||||
False,
|
||||
"You are not allowed to include Brazilian players",
|
||||
),
|
||||
guardrail_max_retries=2,
|
||||
)
|
||||
|
||||
with pytest.raises(Exception, match="Agent's guardrail failed validation after 2 retries"):
|
||||
with pytest.raises(
|
||||
Exception, match="Agent's guardrail failed validation after 2 retries"
|
||||
):
|
||||
agent.kickoff(messages="Top 10 best players in the world?")
|
||||
|
||||
assert len(guardrail_events['started']) == 3 # 2 retries + 1 initial call
|
||||
assert len(guardrail_events['completed']) == 3 # 2 retries + 1 initial call
|
||||
assert not guardrail_events['completed'][0].success
|
||||
assert not guardrail_events['completed'][1].success
|
||||
assert not guardrail_events['completed'][2].success
|
||||
assert len(guardrail_events["started"]) == 3 # 2 retries + 1 initial call
|
||||
assert len(guardrail_events["completed"]) == 3 # 2 retries + 1 initial call
|
||||
assert not guardrail_events["completed"][0].success
|
||||
assert not guardrail_events["completed"][1].success
|
||||
assert not guardrail_events["completed"][2].success
|
||||
|
||||
|
||||
@pytest.mark.vcr(filter_headers=["authorization"])
|
||||
@@ -412,9 +438,100 @@ def test_agent_output_when_guardrail_returns_base_model():
|
||||
role="Sports Analyst",
|
||||
goal="Gather information about the best soccer players",
|
||||
backstory="""You are an expert at gathering and organizing information. You carefully collect details and present them in a structured way.""",
|
||||
guardrail=lambda output: (True, Player(name="Lionel Messi", country="Argentina")),
|
||||
guardrail=lambda output: (
|
||||
True,
|
||||
Player(name="Lionel Messi", country="Argentina"),
|
||||
),
|
||||
)
|
||||
|
||||
result = agent.kickoff(messages="Top 10 best players in the world?")
|
||||
|
||||
assert result.pydantic == Player(name="Lionel Messi", country="Argentina")
|
||||
|
||||
|
||||
@pytest.mark.vcr(filter_headers=["authorization"])
|
||||
def test_lite_agent_with_custom_llm_and_guardrails():
|
||||
"""Test that CustomLLM (inheriting from BaseLLM) works with guardrails."""
|
||||
from crewai.llms.base_llm import BaseLLM
|
||||
|
||||
class CustomLLM(BaseLLM):
|
||||
def __init__(self, response: str = "Custom response"):
|
||||
super().__init__(model="custom-model")
|
||||
self.response = response
|
||||
self.call_count = 0
|
||||
|
||||
def call(
|
||||
self,
|
||||
messages,
|
||||
tools=None,
|
||||
callbacks=None,
|
||||
available_functions=None,
|
||||
from_task=None,
|
||||
from_agent=None,
|
||||
) -> str:
|
||||
self.call_count += 1
|
||||
|
||||
if "valid" in str(messages) and "feedback" in str(messages):
|
||||
return '{"valid": true, "feedback": null}'
|
||||
|
||||
if "Thought:" in str(messages):
|
||||
return f"Thought: I will analyze soccer players\nFinal Answer: {self.response}"
|
||||
|
||||
return self.response
|
||||
|
||||
def supports_function_calling(self) -> bool:
|
||||
return False
|
||||
|
||||
def supports_stop_words(self) -> bool:
|
||||
return False
|
||||
|
||||
def get_context_window_size(self) -> int:
|
||||
return 4096
|
||||
|
||||
custom_llm = CustomLLM(response="Brazilian soccer players are the best!")
|
||||
|
||||
agent = Agent(
|
||||
role="Sports Analyst",
|
||||
goal="Analyze soccer players",
|
||||
backstory="You analyze soccer players and their performance.",
|
||||
llm=custom_llm,
|
||||
guardrail="Only include Brazilian players",
|
||||
)
|
||||
|
||||
result = agent.kickoff("Tell me about the best soccer players")
|
||||
|
||||
assert custom_llm.call_count > 0
|
||||
assert "Brazilian" in result.raw
|
||||
|
||||
custom_llm2 = CustomLLM(response="Original response")
|
||||
|
||||
def test_guardrail(output):
|
||||
return (True, "Modified by guardrail")
|
||||
|
||||
agent2 = Agent(
|
||||
role="Test Agent",
|
||||
goal="Test goal",
|
||||
backstory="Test backstory",
|
||||
llm=custom_llm2,
|
||||
guardrail=test_guardrail,
|
||||
)
|
||||
|
||||
result2 = agent2.kickoff("Test message")
|
||||
assert result2.raw == "Modified by guardrail"
|
||||
|
||||
|
||||
@pytest.mark.vcr(filter_headers=["authorization"])
|
||||
def test_lite_agent_with_invalid_llm():
|
||||
"""Test that LiteAgent raises proper error when create_llm returns None."""
|
||||
from unittest.mock import patch
|
||||
|
||||
with patch("crewai.lite_agent.create_llm", return_value=None):
|
||||
agent = Agent(
|
||||
role="Test Agent",
|
||||
goal="Test goal",
|
||||
backstory="Test backstory",
|
||||
llm="invalid-model",
|
||||
)
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
agent.kickoff("Test message")
|
||||
assert "Expected LLM instance of type BaseLLM" in str(exc_info.value)
|
||||
|
||||
Reference in New Issue
Block a user