* Add telemetry mocking for pytest tests
- Mock telemetry by default for all tests except telemetry-specific tests
- Add @pytest.mark.telemetry marker for real telemetry tests
- Reduce test overhead and improve isolation
* Fix telemetry test isolation
- Properly isolate telemetry tests from mocking environment
- Preserve API keys and other necessary environment variables
- Ensure telemetry tests can run with real telemetry instances
* for ephemeral traces
* default false
* simpler and consolidated
* keep raising exception but catch it and continue if its for trace batches
* cleanup
* more cleanup
* not using logger
* refactor: rename TEMP_TRACING_RESOURCE to EPHEMERAL_TRACING_RESOURCE for clarity and consistency in PlusAPI; update related method calls accordingly
* default true
* drop print
* feat: add tracing support to Crew and Flow classes
- Introduced a new `tracing` optional field in both the `Crew` and `Flow` classes to enable tracing functionality.
- Updated the initialization logic to conditionally set up the `TraceCollectionListener` based on the `tracing` flag or the `CREWAI_TRACING_ENABLED` environment variable.
- Removed the obsolete `interfaces.py` file related to tracing.
- Enhanced the `TraceCollectionListener` to accept a `tracing` parameter and adjusted its internal logic accordingly.
- Added tests to verify the correct setup of the trace listener when tracing is enabled.
This change improves the observability of the crew execution process and allows for better debugging and performance monitoring.
* fix flow name
* refactor: replace _send_batch method with finalize_batch calls in TraceCollectionListener
- Updated the TraceCollectionListener to use the batch_manager's finalize_batch method instead of the deprecated _send_batch method for handling trace events.
- This change improves the clarity of the code and ensures that batch finalization is consistently managed through the batch manager.
- Removed the obsolete _send_batch method to streamline the listener's functionality.
* removed comments
* refactor: enhance tracing functionality by introducing utility for tracing checks
- Added a new utility function `is_tracing_enabled` to streamline the logic for checking if tracing is enabled based on the `CREWAI_TRACING_ENABLED` environment variable.
- Updated the `Crew` and `Flow` classes to utilize this utility for improved readability and maintainability.
- Refactored the `TraceCollectionListener` to simplify tracing checks and ensure consistent behavior across components.
- Introduced a new module for tracing utilities to encapsulate related functions, enhancing code organization.
* refactor: remove unused imports from crew and flow modules
- Removed unnecessary `os` imports from both `crew.py` and `flow.py` files to enhance code cleanliness and maintainability.
* initial setup
* feat: enhance CrewKickoffCompletedEvent to include total token usage
- Added total_tokens attribute to CrewKickoffCompletedEvent for better tracking of token usage during crew execution.
- Updated Crew class to emit total token usage upon kickoff completion.
- Removed obsolete context handler and execution context tracker files to streamline event handling.
* cleanup
* remove print statements for loggers
* feat: add CrewAI base URL and improve logging in tracing
- Introduced `CREWAI_BASE_URL` constant for easy access to the CrewAI application URL.
- Replaced print statements with logging in the `TraceSender` class for better error tracking.
- Enhanced the `TraceBatchManager` to provide default values for flow names and removed unnecessary comments.
- Implemented singleton pattern in `TraceCollectionListener` to ensure a single instance is used.
- Added a new test case to verify that the trace listener correctly collects events during crew execution.
* clear
* fix: update datetime serialization in tracing interfaces
- Removed the 'Z' suffix from datetime serialization in TraceSender and TraceEvent to ensure consistent ISO format.
- Added new test cases to validate the functionality of the TraceBatchManager and event collection during crew execution.
- Introduced fixtures to clear event bus listeners before each test to maintain isolation.
* test: enhance tracing tests with mock authentication token
- Added a mock authentication token to the tracing tests to ensure proper setup and event collection.
- Updated test methods to include the mock token, improving isolation and reliability of tests related to the TraceListener and BatchManager.
- Ensured that the tests validate the correct behavior of event collection during crew execution.
* test: refactor tracing tests to improve mock usage
- Moved the mock authentication token patching inside the test class to enhance readability and maintainability.
- Updated test methods to remove unnecessary mock parameters, streamlining the test signatures.
- Ensured that the tests continue to validate the correct behavior of event collection during crew execution while improving isolation.
* test: refactor tracing tests for improved mock usage and consistency
- Moved mock authentication token patching into individual test methods for better clarity and maintainability.
- Corrected the backstory string in the `Agent` instantiation to fix a typo.
- Ensured that all tests validate the correct behavior of event collection during crew execution while enhancing isolation and readability.
* test: add new tracing test for disabled trace listener
- Introduced a new test case to verify that the trace listener does not make HTTP calls when tracing is disabled via environment variables.
- Enhanced existing tests by mocking PlusAPI HTTP calls to avoid authentication and network requests, improving test isolation and reliability.
- Updated the test setup to ensure proper initialization of the trace listener and its components during crew execution.
* refactor: update LLM class to utilize new completion function and improve cost calculation
- Replaced direct calls to `litellm.completion` with a new import for better clarity and maintainability.
- Introduced a new optional attribute `completion_cost` in the LLM class to track the cost of completions.
- Updated the handling of completion responses to ensure accurate cost calculations and improved error handling.
- Removed outdated test cassettes for gemini models to streamline test suite and avoid redundancy.
- Enhanced existing tests to reflect changes in the LLM class and ensure proper functionality.
* test: enhance tracing tests with additional request and response scenarios
- Added new test cases to validate the behavior of the trace listener and batch manager when handling 404 responses from the tracing API.
- Updated existing test cassettes to include detailed request and response structures, ensuring comprehensive coverage of edge cases.
- Improved mock setup to avoid unnecessary network calls and enhance test reliability.
- Ensured that the tests validate the correct behavior of event collection during crew execution, particularly in scenarios where the tracing service is unavailable.
* feat: enable conditional tracing based on environment variable
- Added support for enabling or disabling the trace listener based on the `CREWAI_TRACING_ENABLED` environment variable.
- Updated the `Crew` class to conditionally set up the trace listener only when tracing is enabled, improving performance and resource management.
- Refactored test cases to ensure proper cleanup of event bus listeners before and after each test, enhancing test reliability and isolation.
- Improved mock setup in tracing tests to validate the behavior of the trace listener when tracing is disabled.
* fix: downgrade litellm version from 1.74.9 to 1.74.3
- Updated the `pyproject.toml` and `uv.lock` files to reflect the change in the `litellm` dependency version.
- This downgrade addresses compatibility issues and ensures stability in the project environment.
* refactor: improve tracing test setup by moving mock authentication token patching
- Removed the module-level patch for the authentication token and implemented a fixture to mock the token for all tests in the class, enhancing test isolation and readability.
- Updated the event bus clearing logic to ensure original handlers are restored after tests, improving reliability of the test environment.
- This refactor streamlines the test setup and ensures consistent behavior across tracing tests.
* test: enhance tracing test setup with comprehensive mock authentication
- Expanded the mock authentication token patching to cover all instances where `get_auth_token` is used across different modules, ensuring consistent behavior in tests.
- Introduced a new fixture to reset tracing singleton instances between tests, improving test isolation and reliability.
- This update enhances the overall robustness of the tracing tests by ensuring that all necessary components are properly mocked and reset, leading to more reliable test outcomes.
* just drop the test for now
* refactor: comment out completion-related code in LLM and LLM event classes
- Commented out the `completion` and `completion_cost` imports and their usage in the `LLM` class to prevent potential issues during execution.
- Updated the `LLMCallCompletedEvent` class to comment out the `response_cost` attribute, ensuring consistency with the changes in the LLM class.
- This refactor aims to streamline the code and prepare for future updates without affecting current functionality.
* refactor: update LLM response handling in LiteAgent
- Commented out the `response_cost` attribute in the LLM response handling to align with recent refactoring in the LLM class.
- This change aims to maintain consistency in the codebase and prepare for future updates without affecting current functionality.
* refactor: remove commented-out response cost attributes in LLM and LiteAgent
- Commented out the `response_cost` attribute in both the `LiteAgent` and `LLM` classes to maintain consistency with recent refactoring efforts.
- This change aligns with previous updates aimed at streamlining the codebase and preparing for future enhancements without impacting current functionality.
* bring back litellm upgrade version
* Dropping User Memory
* Dropping checks for user memory
* changed memory.mdx documentation removed user memory.
* Flaky Test Case Maybe
* Drop memory_config
* Fixed test cases
* Fixed some test cases
* Changed docs
* Changed BR docs
* Docs fixing
* Fix minor doc
* Fix minor doc
* Fix minor doc
* Added fallback mechanism in Mem0
* feat: support oauth2 config for authentication
* refactor: improve OAuth2 settings management
The CLI now supports seamless integration with other authentication providers, since the client_id, issue, domain are now manage by the user
* feat: support okta Device Authorization flow
* chore: resolve linter issues
* test: fix tests
* test: adding tests for auth providers
* test: fix broken test
* refator: adding WorkOS paramenters as default settings auth
* chore: improve oauth2 attributes description
* refactor: simplify WorkOS getting values
* fix: ensure Auth0 parameters is set when overrinding default auth provider
* chore: remove TODO Auth0 no longer provides default values
---------
Co-authored-by: Heitor Carvalho <heitor.scz@gmail.com>
* fix: support to add memories to Mem0 with agent_id
* feat: removing memory_type checkings from Mem0Storage
* feat: ensure agent_id is always present while saving memory into Mem0
* fix: use OR operator when querying Mem0 memories with both user_id and agent_id
* Fix issue #2421: Handle missing google.genai dependency gracefully
Co-Authored-By: Joe Moura <joao@crewai.com>
* Fix import sorting in test file
Co-Authored-By: Joe Moura <joao@crewai.com>
* Fix import sorting with ruff
Co-Authored-By: Joe Moura <joao@crewai.com>
* Removed unwatned test case
* Added dynamic catching for all the embedder function
* Dropped the comment
* Added test case
* Fixed Linting Issue
* Flaky test case in 3.13
* Test Case fixed
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Joe Moura <joao@crewai.com>
Co-authored-by: Lucas Gomide <lucaslg200@gmail.com>
- Added an optional `name` attribute to the Flow class for better identification.
- Updated event emissions to utilize the new `name` attribute, ensuring accurate flow naming in events.
- Added tests to verify the correct flow name is set and emitted during flow execution.
* Changed v1.1 -> v2
* Fixed Test Cases:
* Fixed linting issues
* Changed docs
* Refractored the storage
* Fixed test cases
* Fixing run-time checks
* Fixed Test Case
* Updated docs and added test case for custom categories
* Add the TODO back
* Minor Changes
* Added output_format in search
* Minor changes
* Added output_format and version in both search and save
* Small change
* Minor bugs
* Fixed test cases
* Changed docs
---------
Co-authored-by: Lucas Gomide <lucaslg200@gmail.com>
* feat: add exchanged messages in LLMCallCompletedEvent
* feat: add GoalAlignment metric for Agent evaluation
* feat: add SemanticQuality metric for Agent evaluation
* feat: add Tool Metrics for Agent evaluation
* feat: add Reasoning Metrics for Agent evaluation, still in progress
* feat: add AgentEvaluator class
This class will evaluate Agent' results and report to user
* fix: do not evaluate Agent by default
This is a experimental feature we still need refine it further
* test: add Agent eval tests
* fix: render all feedback per iteration
* style: resolve linter issues
* style: fix mypy issues
* fix: allow messages be empty on LLMCallCompletedEvent
* feat: add Experiment evaluation framework with baseline comparison
* fix: reset evaluator for each experiement iteraction
* fix: fix track of new test cases
* chore: split Experimental evaluation classes
* refactor: remove unused method
* refactor: isolate Console print in a dedicated class
* fix: make crew required to run an experiment
* fix: use time-aware to define experiment result
* test: add tests for Evaluator Experiment
* style: fix linter issues
* fix: encode string before hashing
* style: resolve linter issues
* feat: add experimental folder for beta features (#3141)
* test: move tests to experimental folder
* Fix#3149: Add missing create_directory parameter to Task class
- Add create_directory field with default value True for backward compatibility
- Update _save_file method to respect create_directory parameter
- Add comprehensive tests covering all scenarios
- Maintain existing behavior when create_directory=True (default)
The create_directory parameter was documented but missing from implementation.
Users can now control directory creation behavior:
- create_directory=True (default): Creates directories if they don't exist
- create_directory=False: Raises RuntimeError if directory doesn't exist
Fixes issue where users got TypeError when trying to use the documented
create_directory parameter.
Co-Authored-By: Jo\u00E3o <joao@crewai.com>
* Fix lint: Remove unused import os from test_create_directory_true
- Removes F401 lint error: 'os' imported but unused
- All lint checks should now pass
Co-Authored-By: Jo\u00E3o <joao@crewai.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Jo\u00E3o <joao@crewai.com>
* Compaing BaseLLM class instead of LLM
* Fixed test cases
* Fixed Linting Issues
* removed last line
---------
Co-authored-by: Lucas Gomide <lucaslg200@gmail.com>
* Added add_sources()
* Fixed the agent knowledge querying
* Added test cases
* Fixed linting issue
* Fixed logic
* Seems like a falky test case
* Minor changes
* Added knowledge attriute to the crew documentation
* Flaky test
* fixed spaces
* Flaky Test Case
* Seems like a flaky test case
---------
Co-authored-by: Lucas Gomide <lucaslg200@gmail.com>
* feat: add exchanged messages in LLMCallCompletedEvent
* feat: add GoalAlignment metric for Agent evaluation
* feat: add SemanticQuality metric for Agent evaluation
* feat: add Tool Metrics for Agent evaluation
* feat: add Reasoning Metrics for Agent evaluation, still in progress
* feat: add AgentEvaluator class
This class will evaluate Agent' results and report to user
* fix: do not evaluate Agent by default
This is a experimental feature we still need refine it further
* test: add Agent eval tests
* fix: render all feedback per iteration
* style: resolve linter issues
* style: fix mypy issues
* fix: allow messages be empty on LLMCallCompletedEvent
Add crew context tracking using OpenTelemetry baggage for thread-safe propagation. Context is set during kickoff and cleaned up in finally block. Added thread safety tests with mocked agent execution.
* fix: clean up whitespace and update dependencies
* Removed unnecessary whitespace in multiple files for consistency.
* Updated `crewai-tools` dependency version to `0.49.0` in `pyproject.toml` and related template files.
* Bumped CrewAI version to `0.140.0` in `__init__.py` for alignment with updated dependencies.
* chore: update pyproject.toml to exclude documentation from build targets
* Added exclusions for the `docs` directory in both wheel and sdist build targets to streamline the build process and reduce unnecessary file inclusion.
* chore: update uv.lock for dependency resolution and Python version compatibility
* Incremented revision to 2.
* Updated resolution markers to include support for Python 3.13 and adjusted platform checks for better compatibility.
* Added new wheel URLs for zstandard version 0.23.0 to ensure availability across various platforms.
* chore: pin json-repair dependency version in pyproject.toml and uv.lock
* Updated json-repair dependency from a range to a specific version (0.25.2) for consistency and to avoid potential compatibility issues.
* Adjusted related entries in uv.lock to reflect the pinned version, ensuring alignment across project files.
* chore: pin agentops dependency version in pyproject.toml and uv.lock
* Updated agentops dependency from a range to a specific version (0.3.18) for consistency and to avoid potential compatibility issues.
* Adjusted related entries in uv.lock to reflect the pinned version, ensuring alignment across project files.
* test: enhance cache call assertions in crew tests
* Improved the test for cache hitting between agents by filtering mock calls to ensure they include the expected 'tool' and 'input' keywords.
* Added assertions to verify the number of cache calls and their expected arguments, enhancing the reliability of the test.
* Cleaned up whitespace and improved readability in various test cases for better maintainability.
* feat: add capability to track LLM calls by task and agent
This makes it possible to filter or scope LLM events by specific agents or tasks, which can be very useful for debugging or analytics in real-time application
* feat: add docs about LLM tracking by Agents and Tasks
* fix incompatible BaseLLM.call method signature
* feat: support to filter LLM Events from Lite Agent
When creating a Crew via the CLI and selecting the Azure provider, the generated .env file had environment variables in lowercase.
This commit ensures that all environment variables are written in uppercase.
* fix: normalize project names by stripping trailing slashes in crew creation
- Strip trailing slashes from project names in create_folder_structure
- Add comprehensive tests for trailing slash scenarios
- Fixes#3059
The issue occurred because trailing slashes in project names like 'hello/'
were directly incorporated into pyproject.toml, creating invalid package
names and script entries. This fix silently normalizes project names by
stripping trailing slashes before processing, maintaining backward
compatibility while fixing the invalid template generation.
Co-Authored-By: João <joao@crewai.com>
* trigger CI re-run to check for flaky test issue
Co-Authored-By: João <joao@crewai.com>
* fix: resolve circular import in CLI authentication module
- Move ToolCommand import to be local inside _poll_for_token method
- Update test mock to patch ToolCommand at correct location
- Resolves Python 3.11 test collection failure in CI
Co-Authored-By: João <joao@crewai.com>
* feat: add comprehensive class name validation for Python identifiers
- Ensure generated class names are always valid Python identifiers
- Handle edge cases: names starting with numbers, special characters, keywords, built-ins
- Add sanitization logic to remove invalid characters and prefix with 'Crew' when needed
- Add comprehensive test coverage for class name validation edge cases
- Addresses GitHub PR comment from lucasgomide about class name validity
Fixes include:
- Names starting with numbers: '123project' -> 'Crew123Project'
- Python built-ins: 'True' -> 'TrueCrew', 'False' -> 'FalseCrew'
- Special characters: 'hello@world' -> 'HelloWorld'
- Empty/whitespace: ' ' -> 'DefaultCrew'
- All generated class names pass isidentifier() and keyword checks
Co-Authored-By: João <joao@crewai.com>
* refactor: change class name validation to raise errors instead of generating defaults
- Remove default value generation (Crew prefix/suffix, DefaultCrew fallback)
- Raise ValueError with descriptive messages for invalid class names
- Update tests to expect validation errors instead of default corrections
- Addresses GitHub comment feedback from lucasgomide about strict validation
Co-Authored-By: João <joao@crewai.com>
* fix: add working directory safety checks to prevent test interference
Co-Authored-By: João <joao@crewai.com>
* fix: standardize working directory handling in tests to prevent corruption
Co-Authored-By: João <joao@crewai.com>
* fix: eliminate os.chdir() usage in tests to prevent working directory corruption
- Replace os.chdir() with parent_folder parameter for create_folder_structure tests
- Mock create_folder_structure directly for create_crew tests to avoid directory changes
- All 12 tests now pass locally without working directory corruption
- Should resolve the 103 failing tests in Python 3.12 CI
Co-Authored-By: João <joao@crewai.com>
* fix: remove unused os import to resolve lint failure
- Remove unused 'import os' statement from test_create_crew.py
- All tests still pass locally after removing unused import
- Should resolve F401 lint error in CI
Co-Authored-By: João <joao@crewai.com>
* feat: add folder name validation for Python module names
- Implement validation to ensure folder_name is valid Python identifier
- Check that folder names don't start with digits
- Validate folder names are not Python keywords
- Sanitize invalid characters from folder names
- Raise ValueError with descriptive messages for invalid cases
- Update tests to validate both folder and class name requirements
- Addresses GitHub comment requiring folder names to be valid Python module names
Co-Authored-By: João <joao@crewai.com>
* fix: correct folder name validation logic to match test expectations
- Fix validation regex to catch names starting with invalid characters like '@#/'
- Ensure validation properly raises ValueError for cases expected by tests
- Maintain support for valid cases like 'my.project/' -> 'myproject'
- Address lucasgomide's comment about valid Python module names
Co-Authored-By: João <joao@crewai.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: João <joao@crewai.com>
Co-authored-by: Lucas Gomide <lucaslg200@gmail.com>
* test: add tests to test get_crews
* feat: improve Crew search while resetting their memories
Some memories couldn't be reset due to their reliance on relative external sources like `PDFKnowledge`. This was caused by the need to run the reset memories command from the `src` directory, which could break when external files weren't accessible from that path.
This commit allows the reset command to be executed from the root of the project — the same location typically used to run a crew — improving compatibility and reducing friction.
* feat: skip cli/templates folder while looking for Crew
* refactor: use console.print instead of print
* fix: possible fix for Thinking stuck
* feat: add agent logging events for execution tracking
- Introduced AgentLogsStartedEvent and AgentLogsExecutionEvent to enhance logging capabilities during agent execution.
- Updated CrewAgentExecutor to emit these events at the start and during execution, respectively.
- Modified EventListener to handle the new logging events and format output accordingly in the console.
- Enhanced ConsoleFormatter to display agent logs in a structured format, improving visibility of agent actions and outputs.
* drop emoji
* refactor: improve code structure and logging in LiteAgent and ConsoleFormatter
- Refactored imports in lite_agent.py for better readability.
- Enhanced guardrail property initialization in LiteAgent.
- Updated logging functionality to emit AgentLogsExecutionEvent for better tracking.
- Modified ConsoleFormatter to include tool arguments and final output in status updates.
- Improved output formatting for long text in ConsoleFormatter.
* fix tests
---------
Co-authored-by: Eduardo Chiarotti <dudumelgaco@hotmail.com>
* Fix issue 2993: Prevent Flow status logs from hiding human input
- Add pause_live_updates() and resume_live_updates() methods to ConsoleFormatter
- Modify _ask_human_input() to pause Flow status updates during human input
- Add comprehensive tests for pause/resume functionality and integration
- Ensure Live session is properly managed during human input prompts
- Fix prevents Flow status logs from overwriting user input prompts
Fixes#2993
Co-Authored-By: João <joao@crewai.com>
* Fix lint: Remove unused pytest import
- Remove unused pytest import from test_console_formatter_pause_resume.py
- Fixes F401 lint error identified in CI
Co-Authored-By: João <joao@crewai.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: João <joao@crewai.com>
* Fix telemetry singleton pattern to respect dynamic environment variables
- Modified Telemetry.__init__ to prevent re-initialization with _initialized flag
- Updated _safe_telemetry_operation to check _is_telemetry_disabled() dynamically
- Added comprehensive tests for environment variables set after singleton creation
- Fixed singleton contamination in existing tests by adding proper reset
- Resolves issue #2945 where CREWAI_DISABLE_TELEMETRY=true was ignored when set after import
Co-Authored-By: João <joao@crewai.com>
* Implement code review improvements
- Move _initialized flag to __new__ method for better encapsulation
- Add type hints to _safe_telemetry_operation method
- Consolidate telemetry execution checks into _should_execute_telemetry helper
- Add pytest fixtures to reduce test setup redundancy
- Enhanced documentation for singleton behavior
Co-Authored-By: João <joao@crewai.com>
* Fix mypy type-checker errors
- Add explicit bool type annotation to _initialized field
- Fix return value in task_started method to not return _safe_telemetry_operation result
- Simplify initialization logic to set _initialized once in __init__
Co-Authored-By: João <joao@crewai.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: João <joao@crewai.com>
Co-authored-by: Lorenze Jay <63378463+lorenzejay@users.noreply.github.com>
* feat: add guardrail support for Agents when using direct kickoff calls
* refactor: expose guardrail func in a proper utils file
* fix: resolve Self import on python 3.10
* test: fix structured tool tests
No tests were being executed from this file
* feat: support to run async tool
Some Tool requires async execution. This commit allow us to collect tool result from coroutines
* docs: add docs about asynchronous tool support
