- Update validate_tools method in BaseAgent to accept all documented tool patterns:
* Function tools (with or without @tool decorator)
* Dict-based tool definitions
* BaseTool class inheritance
* Direct function assignment
- Change tools field type annotation from List[BaseTool] to List[Any] to allow Pydantic validation
- Update parse_tools function to accept all BaseTool instances (not just CrewAITool)
- Add comprehensive tests covering all custom tool patterns from issue #3226
- Add reproduction script to verify all patterns work correctly
Fixes#3226
Co-Authored-By: João <joao@crewai.com>
* feat: add exchanged messages in LLMCallCompletedEvent
* feat: add GoalAlignment metric for Agent evaluation
* feat: add SemanticQuality metric for Agent evaluation
* feat: add Tool Metrics for Agent evaluation
* feat: add Reasoning Metrics for Agent evaluation, still in progress
* feat: add AgentEvaluator class
This class will evaluate Agent' results and report to user
* fix: do not evaluate Agent by default
This is a experimental feature we still need refine it further
* test: add Agent eval tests
* fix: render all feedback per iteration
* style: resolve linter issues
* style: fix mypy issues
* fix: allow messages be empty on LLMCallCompletedEvent
Add crew context tracking using OpenTelemetry baggage for thread-safe propagation. Context is set during kickoff and cleaned up in finally block. Added thread safety tests with mocked agent execution.
* feat: add capability to track LLM calls by task and agent
This makes it possible to filter or scope LLM events by specific agents or tasks, which can be very useful for debugging or analytics in real-time application
* feat: add docs about LLM tracking by Agents and Tasks
* fix incompatible BaseLLM.call method signature
* feat: support to filter LLM Events from Lite Agent
* Added Union of List of Task, None, NotSpecified
* Seems like a flaky test
* Fixed run time issue
* Fixed Linting issues
* fix pydantic error
* aesthetic changes
---------
Co-authored-by: Lucas Gomide <lucaslg200@gmail.com>
* fix: possible fix for Thinking stuck
* feat: add agent logging events for execution tracking
- Introduced AgentLogsStartedEvent and AgentLogsExecutionEvent to enhance logging capabilities during agent execution.
- Updated CrewAgentExecutor to emit these events at the start and during execution, respectively.
- Modified EventListener to handle the new logging events and format output accordingly in the console.
- Enhanced ConsoleFormatter to display agent logs in a structured format, improving visibility of agent actions and outputs.
* drop emoji
* refactor: improve code structure and logging in LiteAgent and ConsoleFormatter
- Refactored imports in lite_agent.py for better readability.
- Enhanced guardrail property initialization in LiteAgent.
- Updated logging functionality to emit AgentLogsExecutionEvent for better tracking.
- Modified ConsoleFormatter to include tool arguments and final output in status updates.
- Improved output formatting for long text in ConsoleFormatter.
* fix tests
---------
Co-authored-by: Eduardo Chiarotti <dudumelgaco@hotmail.com>
* Fix issue 2993: Prevent Flow status logs from hiding human input
- Add pause_live_updates() and resume_live_updates() methods to ConsoleFormatter
- Modify _ask_human_input() to pause Flow status updates during human input
- Add comprehensive tests for pause/resume functionality and integration
- Ensure Live session is properly managed during human input prompts
- Fix prevents Flow status logs from overwriting user input prompts
Fixes#2993
Co-Authored-By: João <joao@crewai.com>
* Fix lint: Remove unused pytest import
- Remove unused pytest import from test_console_formatter_pause_resume.py
- Fixes F401 lint error identified in CI
Co-Authored-By: João <joao@crewai.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: João <joao@crewai.com>
* feat: add guardrail support for Agents when using direct kickoff calls
* refactor: expose guardrail func in a proper utils file
* fix: resolve Self import on python 3.10
* docs: add organization management in our CLI docs
* feat: improve user feedback when user is not authenticated
* feat: improve logging about current organization while publishing/install a Tool
* feat: improve logging when Agent repository is not found during fetch
* fix linter offences
* test: fix auth token error
Previously, we only supported tools from the crewai-tools open-source repository. Now, we're introducing improved support for private tool repositories.
- Add `HallucinationGuardrail` class as enterprise feature placeholder
- Update LLM guardrail events to support `HallucinationGuardrail` instances
- Add comprehensive tests for `HallucinationGuardrail` initialization and behavior
- Add integration tests for `HallucinationGuardrail` with task execution system
- Ensure no-op behavior always returns True
* Refactor Crew class memory initialization and enhance event handling
- Simplified the initialization of the external memory attribute in the Crew class.
- Updated memory system retrieval logic for consistency in key usage.
- Introduced a singleton pattern for the Telemetry class to ensure a single instance.
- Replaced telemetry usage in CrewEvaluator with event bus emissions for test results.
- Added new CrewTestResultEvent to handle crew test results more effectively.
- Updated event listener to process CrewTestResultEvent and log telemetry data accordingly.
- Enhanced tests to validate the singleton pattern in Telemetry and the new event handling logic.
* linted
* Remove unused telemetry attribute from Crew class memory initialization
* fix ordering of test
* Implement thread-safe singleton pattern in Telemetry class
- Introduced a threading lock to ensure safe instantiation of the Telemetry singleton.
- Updated the __new__ method to utilize double-checked locking for instance creation.
* Add reasoning attribute to Agent class
Co-Authored-By: Joe Moura <joao@crewai.com>
* Address PR feedback: improve type hints, error handling, refactor reasoning handler, and enhance tests and docs
Co-Authored-By: Joe Moura <joao@crewai.com>
* Implement function calling for reasoning and move prompts to translations
Co-Authored-By: Joe Moura <joao@crewai.com>
* Simplify function calling implementation with better error handling
Co-Authored-By: Joe Moura <joao@crewai.com>
* Enhance system prompts to leverage agent context (role, goal, backstory)
Co-Authored-By: Joe Moura <joao@crewai.com>
* Fix lint and type-checker issues
Co-Authored-By: Joe Moura <joao@crewai.com>
* Enhance system prompts to better leverage agent context
Co-Authored-By: Joe Moura <joao@crewai.com>
* Fix backstory access in reasoning handler for Python 3.12 compatibility
Co-Authored-By: Joe Moura <joao@crewai.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Joe Moura <joao@crewai.com>
Co-authored-by: João Moura <joaomdmoura@gmail.com>
* Enhance string interpolation to support hyphens in variable names and add corresponding test cases. Update existing tests for consistency and formatting.
* Refactor tests in task_test.py by removing unused Task instances to streamline test cases for the interpolate_only method and related functions.
* feat: implement knowledge retrieval events in Agent
This commit introduces a series of knowledge retrieval events in the Agent class, enhancing its ability to handle knowledge queries. New events include KnowledgeRetrievalStartedEvent, KnowledgeRetrievalCompletedEvent, KnowledgeQueryGeneratedEvent, KnowledgeQueryFailedEvent, and KnowledgeSearchQueryCompletedEvent. The Agent now emits these events during knowledge retrieval processes, allowing for better tracking and handling of knowledge queries. Additionally, the console formatter has been updated to handle these new events, providing visual feedback during knowledge retrieval operations.
* refactor: update knowledge query handling in Agent
This commit refines the knowledge query processing in the Agent class by renaming variables for clarity and optimizing the query rewriting logic. The system prompt has been updated in the translation file to enhance clarity and context for the query rewriting process. These changes aim to improve the overall readability and maintainability of the code.
* fix: add missing newline at end of en.json file
* fix broken tests
* refactor: rename knowledge query events and enhance retrieval handling
This commit renames the KnowledgeQueryGeneratedEvent to KnowledgeQueryStartedEvent to better reflect its purpose. It also updates the event handling in the EventListener and ConsoleFormatter classes to accommodate the new event structure. Additionally, the retrieval knowledge is now included in the KnowledgeRetrievalCompletedEvent, improving the overall knowledge retrieval process.
* docs for transparancy
* refactor: improve error handling in knowledge query processing
This commit refactors the knowledge query handling in the Agent class by changing the order of checks for LLM compatibility. It now logs a warning and emits a failure event if the LLM is not an instance of BaseLLM before attempting to call the LLM. Additionally, the task_prompt attribute has been removed from the KnowledgeQueryFailedEvent, simplifying the event structure.
* test: add unit test for knowledge search query and VCR cassette
This commit introduces a new test, `test_get_knowledge_search_query`, to verify that the `_get_knowledge_search_query` method in the Agent class correctly interacts with the LLM using the appropriate prompts. Additionally, a VCR cassette is added to record the interactions with the OpenAI API for this test, ensuring consistent and reliable test results.
* build(dev): add pytest-randomly dependency
By randomizing the test execution order, this helps identify tests
that unintentionally depend on shared state or specific execution
order, which can lead to flaky or unreliable test behavior.
* build(dev): add pytest-timeout
This will prevent a test from running indefinitely
* test: block external requests in CI and set default 10s timeout per test
* test: adding missing cassettes
We notice that those cassettes are missing after enabling block-network on CI
* test: increase tests timeout on CI
* test: fix flaky test ValueError: Circular reference detected (id repeated)
* fix: prevent crash when event handler raises exception
Previously, if a registered event handler raised an exception during execution,
it could crash the entire application or interrupt the event dispatch process.
This change wraps handler execution in a try/except block within the `emit` method,
ensuring that exceptions are caught and logged without affecting other handlers or flow.
This improves the resilience of the event bus, especially when handling third-party
or temporary listeners.
* feat: support to define a guardrail task no-code
* feat: add auto-discovery for Guardrail code execution mode
* feat: handle malformed or invalid response from CodeInterpreterTool
* feat: allow to set unsafe_mode from Guardrail task
* feat: renaming GuardrailTask to TaskGuardrail
* feat: ensure guardrail is callable while initializing Task
* feat: remove Docker availability check from TaskGuardrail
The CodeInterpreterTool already ensures compliance with this requirement.
* refactor: replace if/raise with assert
For this use case `assert` is more appropriate choice
* test: remove useless or duplicated test
* fix: attempt to fix type-checker
* feat: support to define a task guardrail using YAML config
* refactor: simplify TaskGuardrail to use LLM for validation, no code generation
* docs: update TaskGuardrail doc strings
* refactor: drop task paramenter from TaskGuardrail
This parameter was used to get the model from the `task.agent` which is a quite bit redudant since we could propagate the llm directly
Add `__init__.py` files to 20 directories to conform with Python package standards. This ensures directories are properly recognized as packages, enabling cleaner imports.
* Fix issue #2402: Handle missing templates gracefully
Co-Authored-By: Joe Moura <joao@crewai.com>
* Fix import sorting in test files
Co-Authored-By: Joe Moura <joao@crewai.com>
* Bluit in top of devin-ai integration
* Fixed test cases
* Fixed test cases
* fixed linting issue
* Added docs
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Joe Moura <joao@crewai.com>
* feat: add OpenAI agent adapter implementation
- Introduced OpenAIAgentAdapter class to facilitate interaction with OpenAI Assistants.
- Implemented methods for task execution, tool configuration, and response processing.
- Added support for converting CrewAI tools to OpenAI format and handling delegation tools.
* created an adapter for the delegate and ask_question tools
* delegate and ask_questions work and it delegates to crewai agents*
* refactor: introduce OpenAIAgentToolAdapter for tool management
- Created OpenAIAgentToolAdapter class to encapsulate tool configuration and conversion for OpenAI Assistant.
- Removed tool configuration logic from OpenAIAgentAdapter and integrated it into the new adapter.
- Enhanced the tool conversion process to ensure compatibility with OpenAI's requirements.
* feat: implement BaseAgentAdapter for agent integration
- Introduced BaseAgentAdapter as an abstract base class for agent adapters in CrewAI.
- Defined common interface and methods for configuring tools and structured output.
- Updated OpenAIAgentAdapter to inherit from BaseAgentAdapter, enhancing its structure and functionality.
* feat: add LangGraph agent and tool adapter for CrewAI integration
- Introduced LangGraphAgentAdapter to facilitate interaction with LangGraph agents.
- Implemented methods for task execution, context handling, and tool configuration.
- Created LangGraphToolAdapter to convert CrewAI tools into LangGraph-compatible format.
- Enhanced error handling and logging for task execution and streaming processes.
* feat: enhance LangGraphToolAdapter and improve conversion instructions
- Added type hints for better clarity and type checking in LangGraphToolAdapter.
- Updated conversion instructions to ensure compatibility with optional LLM checks.
* feat: integrate structured output handling in LangGraph and OpenAI agents
- Added LangGraphConverterAdapter for managing structured output in LangGraph agents.
- Enhanced LangGraphAgentAdapter to utilize the new converter for system prompt and task execution.
- Updated LangGraphToolAdapter to use StructuredTool for better compatibility.
- Introduced OpenAIConverterAdapter for structured output management in OpenAI agents.
- Improved task execution flow in OpenAIAgentAdapter to incorporate structured output configuration and post-processing.
* feat: implement BaseToolAdapter for tool integration
- Introduced BaseToolAdapter as an abstract base class for tool adapters in CrewAI.
- Updated LangGraphToolAdapter and OpenAIAgentToolAdapter to inherit from BaseToolAdapter, enhancing their structure and functionality.
- Improved tool configuration methods to support better integration with various frameworks.
- Added type hints and documentation for clarity and maintainability.
* feat: enhance OpenAIAgentAdapter with configurable agent properties
- Refactored OpenAIAgentAdapter to accept agent configuration as an argument.
- Introduced a method to build a system prompt for the OpenAI agent, improving task execution context.
- Updated initialization to utilize role, goal, and backstory from kwargs, enhancing flexibility in agent setup.
- Improved tool handling and integration within the adapter.
* feat: enhance agent adapters with structured output support
- Introduced BaseConverterAdapter as an abstract class for structured output handling.
- Implemented LangGraphConverterAdapter and OpenAIConverterAdapter to manage structured output in their respective agents.
- Updated BaseAgentAdapter to accept an agent configuration dictionary during initialization.
- Enhanced LangGraphAgentAdapter to utilize the new converter and improved tool handling.
- Added methods for configuring structured output and enhancing system prompts in converter adapters.
* refactor: remove _parse_tools method from OpenAIAgentAdapter and BaseAgent
- Eliminated the _parse_tools method from OpenAIAgentAdapter and its abstract declaration in BaseAgent.
- Cleaned up related test code in MockAgent to reflect the removal of the method.
* also removed _parse_tools here as not used
* feat: add dynamic import handling for LangGraph dependencies
- Implemented conditional imports for LangGraph components to handle ImportError gracefully.
- Updated LangGraphAgentAdapter initialization to check for LangGraph availability and raise an informative error if dependencies are missing.
- Enhanced the agent adapter's robustness by ensuring it only initializes components when the required libraries are present.
* fix: improve error handling for agent adapters
- Updated LangGraphAgentAdapter to raise an ImportError with a clear message if LangGraph dependencies are not installed.
- Refactored OpenAIAgentAdapter to include a similar check for OpenAI dependencies, ensuring robust initialization and user guidance for missing libraries.
- Enhanced overall error handling in agent adapters to prevent runtime issues when dependencies are unavailable.
* refactor: enhance tool handling in agent adapters
- Updated BaseToolAdapter to initialize original and converted tools in the constructor.
- Renamed method `all_tools` to `tools` for clarity in BaseToolAdapter.
- Added `sanitize_tool_name` method to ensure tool names are API compatible.
- Modified LangGraphAgentAdapter to utilize the updated tool handling and ensure proper tool configuration.
- Refactored LangGraphToolAdapter to streamline tool conversion and ensure consistent naming conventions.
* feat: emit AgentExecutionCompletedEvent in agent adapters
- Added emission of AgentExecutionCompletedEvent in both LangGraphAgentAdapter and OpenAIAgentAdapter to signal task completion.
- Enhanced event handling to include agent, task, and output details for better tracking of execution results.
* docs: Enhance BaseConverterAdapter documentation
- Added a detailed docstring to the BaseConverterAdapter class, outlining its purpose and the expected functionality for all converter adapters.
- Updated the post_process_result method's docstring to specify the expected format of the result as a string.
* docs: Add comprehensive guide for bringing custom agents into CrewAI
- Introduced a new documentation file detailing the process of integrating custom agents using the BaseAgentAdapter, BaseToolAdapter, and BaseConverter.
- Included step-by-step instructions for creating custom adapters, configuring tools, and handling structured output.
- Provided examples for implementing adapters for various frameworks, enhancing the usability of CrewAI for developers.
* feat: Introduce adapted_agent flag in BaseAgent and update BaseAgentAdapter initialization
- Added an `adapted_agent` boolean field to the BaseAgent class to indicate if the agent is adapted.
- Updated the BaseAgentAdapter's constructor to pass `adapted_agent=True` to the superclass, ensuring proper initialization of the new field.
* feat: Enhance LangGraphAgentAdapter to support optional agent configuration
- Updated LangGraphAgentAdapter to conditionally apply agent configuration when creating the agent graph, allowing for more flexible initialization.
- Modified LangGraphToolAdapter to ensure only instances of BaseTool are converted, improving tool compatibility and handling.
* feat: Introduce OpenAIConverterAdapter for structured output handling
- Added OpenAIConverterAdapter to manage structured output conversion for OpenAI agents, enhancing their ability to process and format results.
- Updated OpenAIAgentAdapter to utilize the new converter for configuring structured output and post-processing results.
- Removed the deprecated get_output_converter method from OpenAIAgentAdapter.
- Added unit tests for BaseAgentAdapter and BaseToolAdapter to ensure proper functionality and integration of new features.
* feat: Enhance tool adapters to support asynchronous execution
- Updated LangGraphToolAdapter and OpenAIAgentToolAdapter to handle asynchronous tool execution by checking if the output is awaitable.
- Introduced `inspect` import to facilitate the awaitability check.
- Refactored tool wrapper functions to ensure proper handling of both synchronous and asynchronous tool results.
* fix: Correct method definition syntax and enhance tool adapter implementation
- Updated the method definition for `configure_structured_output` to include the `def` keyword for clarity.
- Added an asynchronous tool wrapper to ensure tools can operate in both synchronous and asynchronous contexts.
- Modified the constructor of the custom converter adapter to directly assign the agent adapter, improving clarity and functionality.
* linted
* refactor: Improve tool processing logic in BaseAgent
- Added a check to return an empty list if no tools are provided.
- Simplified the tool attribute validation by using a list of required attributes.
- Removed commented-out abstract method definition for clarity.
* refactor: Simplify tool handling in agent adapters
- Changed default value of `tools` parameter in LangGraphAgentAdapter to None for better handling of empty tool lists.
- Updated tool initialization in both LangGraphAgentAdapter and OpenAIAgentAdapter to directly pass the `tools` parameter, removing unnecessary list handling.
- Cleaned up commented-out code in OpenAIConverterAdapter to improve readability.
* refactor: Remove unused stream_task method from LangGraphAgentAdapter
- Deleted the `stream_task` method from LangGraphAgentAdapter to streamline the code and eliminate unnecessary complexity.
- This change enhances maintainability by focusing on essential functionalities within the agent adapter.
* feat: unblock LLM(stream=True) to work with tools
* feat: replace pytest-vcr by pytest-recording
1. pytest-vcr does not support httpx - which LiteLLM uses for streaming responses.
2. pytest-vcr is no longer maintained, last commit 6 years ago :fist::skin-tone-4:
3. pytest-recording supports modern request libraries (including httpx) and actively maintained
* refactor: remove @skip_streaming_in_ci
Since we have fixed streaming response issue we can remove this @skip_streaming_in_ci
---------
Co-authored-by: Lorenze Jay <63378463+lorenzejay@users.noreply.github.com>
* WIP
* WIP
* wip
* wip
* WIP
* More WIP
* Its working but needs a massive clean up
* output type works now
* Usage metrics fixed
* more testing
* WIP
* cleaning up
* Update logger
* 99% done. Need to make docs match new example
* cleanup
* drop hard coded examples
* docs
* Clean up
* Fix errors
* Trying to fix CI issues
* more type checker fixes
* More type checking fixes
* Update LiteAgent documentation for clarity and consistency; replace WebsiteSearchTool with SerperDevTool, and improve formatting in examples.
* fix fingerprinting issues
* fix type-checker
* Fix type-checker issue by adding type ignore comment for cache read in ToolUsage class
* Add optional agent parameter to CrewAgentParser and enhance action handling logic
* Remove unused parameters from ToolUsage instantiation in tests and clean up debug print statement in CrewAgentParser.
* Remove deprecated test files and examples for LiteAgent; add comprehensive tests for LiteAgent functionality, including tool usage and structured output handling.
* Remove unused variable 'result' from ToolUsage class to clean up code.
* Add initialization for 'result' variable in ToolUsage class to resolve type-checker warnings
* Refactor agent_utils.py by removing unused event imports and adding missing commas in function definitions. Update test_events.py to reflect changes in expected event counts and adjust assertions accordingly. Modify test_tools_emits_error_events.yaml to include new headers and update response content for consistency with recent API changes.
* Enhance tests in crew_test.py by verifying cache behavior in test_tools_with_custom_caching and ensuring proper agent initialization with added commas in test_crew_kickoff_for_each_works_with_manager_agent_copy.
* Update agent tests to reflect changes in expected call counts and improve response formatting in YAML cassette. Adjusted mock call count from 2 to 3 and refined interaction formats for clarity and consistency.
* Refactor agent tests to update model versions and improve response formatting in YAML cassettes. Changed model references from 'o1-preview' to 'o3-mini' and adjusted interaction formats for consistency. Enhanced error handling in context length tests and refined mock setups for better clarity.
* Update tool usage logging to ensure tool arguments are consistently formatted as strings. Adjust agent test cases to reflect changes in maximum iterations and expected outputs, enhancing clarity in assertions. Update YAML cassettes to align with new response formats and improve overall consistency across tests.
* Update YAML cassette for LLM tests to reflect changes in response structure and model version. Adjusted request and response headers, including updated content length and user agent. Enhanced token limits and request counts for improved testing accuracy.
* Update tool usage logging to store tool arguments as native types instead of strings, enhancing data integrity and usability.
* Refactor agent tests by removing outdated test cases and updating YAML cassettes to reflect changes in tool usage and response formats. Adjusted request and response headers, including user agent and content length, for improved accuracy in testing. Enhanced interaction formats for consistency across tests.
* Add Excalidraw diagram file for visual representation of input-output flow
Created a new Excalidraw file that includes a diagram illustrating the input box, database, and output box with connecting arrows. This visual aid enhances understanding of the data flow within the application.
* Remove redundant error handling for action and final answer in CrewAgentParser. Update tests to reflect this change by deleting the corresponding test case.
---------
Co-authored-by: Lorenze Jay <63378463+lorenzejay@users.noreply.github.com>
Co-authored-by: Lorenze Jay <lorenzejaytech@gmail.com>
- Renamed `CrewEvent` to `BaseEvent` across the codebase for consistency
- Created a `CrewBaseEvent` that automatically identifies fingerprints for DRY
- Added a new `to_json()` method for serializing events
