8 Commits

Author	SHA1	Message	Date
Lorenze Jay	a0b757a12c	Lorenze/traces mark as failed (#3586) ... * marking trace batch as failed if its failed * fix test	2025-09-23 22:02:27 -07:00
Lorenze Jay	578fa8c2e4	Lorenze/ephemeral trace ask (#3530) ... * feat(tracing): implement first-time trace handling and improve event management - Added FirstTimeTraceHandler for managing first-time user trace collection and display. - Enhanced TraceBatchManager to support ephemeral trace URLs and improved event buffering. - Updated TraceCollectionListener to utilize the new FirstTimeTraceHandler. - Refactored type annotations across multiple files for consistency and clarity. - Improved error handling and logging for trace-related operations. - Introduced utility functions for trace viewing prompts and first execution checks. * brought back crew finalize batch events * refactor(trace): move instance variables to __init__ in TraceBatchManager - Refactored TraceBatchManager to initialize instance variables in the constructor instead of as class variables. - Improved clarity and encapsulation of the class state. * fix(tracing): improve error handling in user data loading and saving - Enhanced error handling in _load_user_data and _save_user_data functions to log warnings for JSON decoding and file access issues. - Updated documentation for trace usage to clarify the addition of tracing parameters in Crew and Flow initialization. - Refined state management in Flow class to ensure proper handling of state IDs when persistence is enabled. * add some tests * fix test * fix tests * refactor(tracing): enhance user input handling for trace viewing - Replaced signal-based timeout handling with threading for user input in prompt_user_for_trace_viewing function. - Improved user experience by allowing a configurable timeout for viewing execution traces. - Updated tests to mock threading behavior and verify timeout handling correctly. * fix(tracing): improve machine ID retrieval with error handling - Added error handling to the _get_machine_id function to log warnings when retrieving the machine ID fails. - Ensured that the function continues to provide a stable, privacy-preserving machine fingerprint even in case of errors. * refactor(flow): streamline state ID assignment in Flow class - Replaced direct attribute assignment with setattr for improved flexibility in handling state IDs. - Enhanced code readability by simplifying the logic for setting the state ID when persistence is enabled.	2025-09-18 10:17:34 -07:00
Greyson LaLonde	878c1a649a	refactor: Move events module to crewai.events (#3425) ... refactor(events): relocate events module & update imports - Move events from utilities/ to top-level events/ with types/, listeners/, utils/ structure - Update all source/tests/docs to new import paths - Add backwards compatibility stubs in crewai.utilities.events with deprecation warnings - Restore test mocks and fix related test imports	2025-09-02 10:06:42 -04:00
Greyson LaLonde	641c156c17	fix: address flaky tests (#3363) ... fix: resolve flaky tests and race conditions in test suite - Fix telemetry/event tests by patching class methods instead of instances - Use unique temp files/directories to prevent CI race conditions - Reset singleton state between tests - Mock embedchain.Client.setup() to prevent JSON corruption - Rename test files to test_*.py convention - Move agent tests to tests/agents directory - Fix repeated tool usage detection - Remove database-dependent tools causing initialization errors	2025-08-20 13:34:09 -04:00
Lorenze Jay	a1b3edd79c	Refactor tracing logic to consolidate conditions for enabling tracing… (#3347) ... * Refactor tracing logic to consolidate conditions for enabling tracing in Crew class and update TraceBatchManager to handle ephemeral batches more effectively. Added tests for trace listener handling of both ephemeral and authenticated user batches. * drop print * linted * refactor: streamline ephemeral handling in TraceBatchManager This commit removes the ephemeral parameter from the _send_events_to_backend and _finalize_backend_batch methods, replacing it with internal logic that checks the current batch's ephemeral status. This change simplifies the method signatures and enhances the clarity of the code by directly using the is_current_batch_ephemeral attribute for conditional logic.	2025-08-18 14:16:51 -07:00
Lorenze Jay	04a03d332f	Lorenze/emphemeral tracing (#3323) ... * for ephemeral traces * default false * simpler and consolidated * keep raising exception but catch it and continue if its for trace batches * cleanup * more cleanup * not using logger * refactor: rename TEMP_TRACING_RESOURCE to EPHEMERAL_TRACING_RESOURCE for clarity and consistency in PlusAPI; update related method calls accordingly * default true * drop print	2025-08-15 13:37:16 -07:00
Lorenze Jay	251ae00b8b	Lorenze/tracing-improvements-cleanup (#3291) ... * feat: add tracing support to Crew and Flow classes - Introduced a new `tracing` optional field in both the `Crew` and `Flow` classes to enable tracing functionality. - Updated the initialization logic to conditionally set up the `TraceCollectionListener` based on the `tracing` flag or the `CREWAI_TRACING_ENABLED` environment variable. - Removed the obsolete `interfaces.py` file related to tracing. - Enhanced the `TraceCollectionListener` to accept a `tracing` parameter and adjusted its internal logic accordingly. - Added tests to verify the correct setup of the trace listener when tracing is enabled. This change improves the observability of the crew execution process and allows for better debugging and performance monitoring. * fix flow name * refactor: replace _send_batch method with finalize_batch calls in TraceCollectionListener - Updated the TraceCollectionListener to use the batch_manager's finalize_batch method instead of the deprecated _send_batch method for handling trace events. - This change improves the clarity of the code and ensures that batch finalization is consistently managed through the batch manager. - Removed the obsolete _send_batch method to streamline the listener's functionality. * removed comments * refactor: enhance tracing functionality by introducing utility for tracing checks - Added a new utility function `is_tracing_enabled` to streamline the logic for checking if tracing is enabled based on the `CREWAI_TRACING_ENABLED` environment variable. - Updated the `Crew` and `Flow` classes to utilize this utility for improved readability and maintainability. - Refactored the `TraceCollectionListener` to simplify tracing checks and ensure consistent behavior across components. - Introduced a new module for tracing utilities to encapsulate related functions, enhancing code organization. * refactor: remove unused imports from crew and flow modules - Removed unnecessary `os` imports from both `crew.py` and `flow.py` files to enhance code cleanliness and maintainability.	2025-08-08 13:42:25 -07:00
Lorenze Jay	8f4a6cc61c	Lorenze/tracing v1 (#3279) ... * initial setup * feat: enhance CrewKickoffCompletedEvent to include total token usage - Added total_tokens attribute to CrewKickoffCompletedEvent for better tracking of token usage during crew execution. - Updated Crew class to emit total token usage upon kickoff completion. - Removed obsolete context handler and execution context tracker files to streamline event handling. * cleanup * remove print statements for loggers * feat: add CrewAI base URL and improve logging in tracing - Introduced `CREWAI_BASE_URL` constant for easy access to the CrewAI application URL. - Replaced print statements with logging in the `TraceSender` class for better error tracking. - Enhanced the `TraceBatchManager` to provide default values for flow names and removed unnecessary comments. - Implemented singleton pattern in `TraceCollectionListener` to ensure a single instance is used. - Added a new test case to verify that the trace listener correctly collects events during crew execution. * clear * fix: update datetime serialization in tracing interfaces - Removed the 'Z' suffix from datetime serialization in TraceSender and TraceEvent to ensure consistent ISO format. - Added new test cases to validate the functionality of the TraceBatchManager and event collection during crew execution. - Introduced fixtures to clear event bus listeners before each test to maintain isolation. * test: enhance tracing tests with mock authentication token - Added a mock authentication token to the tracing tests to ensure proper setup and event collection. - Updated test methods to include the mock token, improving isolation and reliability of tests related to the TraceListener and BatchManager. - Ensured that the tests validate the correct behavior of event collection during crew execution. * test: refactor tracing tests to improve mock usage - Moved the mock authentication token patching inside the test class to enhance readability and maintainability. - Updated test methods to remove unnecessary mock parameters, streamlining the test signatures. - Ensured that the tests continue to validate the correct behavior of event collection during crew execution while improving isolation. * test: refactor tracing tests for improved mock usage and consistency - Moved mock authentication token patching into individual test methods for better clarity and maintainability. - Corrected the backstory string in the `Agent` instantiation to fix a typo. - Ensured that all tests validate the correct behavior of event collection during crew execution while enhancing isolation and readability. * test: add new tracing test for disabled trace listener - Introduced a new test case to verify that the trace listener does not make HTTP calls when tracing is disabled via environment variables. - Enhanced existing tests by mocking PlusAPI HTTP calls to avoid authentication and network requests, improving test isolation and reliability. - Updated the test setup to ensure proper initialization of the trace listener and its components during crew execution. * refactor: update LLM class to utilize new completion function and improve cost calculation - Replaced direct calls to `litellm.completion` with a new import for better clarity and maintainability. - Introduced a new optional attribute `completion_cost` in the LLM class to track the cost of completions. - Updated the handling of completion responses to ensure accurate cost calculations and improved error handling. - Removed outdated test cassettes for gemini models to streamline test suite and avoid redundancy. - Enhanced existing tests to reflect changes in the LLM class and ensure proper functionality. * test: enhance tracing tests with additional request and response scenarios - Added new test cases to validate the behavior of the trace listener and batch manager when handling 404 responses from the tracing API. - Updated existing test cassettes to include detailed request and response structures, ensuring comprehensive coverage of edge cases. - Improved mock setup to avoid unnecessary network calls and enhance test reliability. - Ensured that the tests validate the correct behavior of event collection during crew execution, particularly in scenarios where the tracing service is unavailable. * feat: enable conditional tracing based on environment variable - Added support for enabling or disabling the trace listener based on the `CREWAI_TRACING_ENABLED` environment variable. - Updated the `Crew` class to conditionally set up the trace listener only when tracing is enabled, improving performance and resource management. - Refactored test cases to ensure proper cleanup of event bus listeners before and after each test, enhancing test reliability and isolation. - Improved mock setup in tracing tests to validate the behavior of the trace listener when tracing is disabled. * fix: downgrade litellm version from 1.74.9 to 1.74.3 - Updated the `pyproject.toml` and `uv.lock` files to reflect the change in the `litellm` dependency version. - This downgrade addresses compatibility issues and ensures stability in the project environment. * refactor: improve tracing test setup by moving mock authentication token patching - Removed the module-level patch for the authentication token and implemented a fixture to mock the token for all tests in the class, enhancing test isolation and readability. - Updated the event bus clearing logic to ensure original handlers are restored after tests, improving reliability of the test environment. - This refactor streamlines the test setup and ensures consistent behavior across tracing tests. * test: enhance tracing test setup with comprehensive mock authentication - Expanded the mock authentication token patching to cover all instances where `get_auth_token` is used across different modules, ensuring consistent behavior in tests. - Introduced a new fixture to reset tracing singleton instances between tests, improving test isolation and reliability. - This update enhances the overall robustness of the tracing tests by ensuring that all necessary components are properly mocked and reset, leading to more reliable test outcomes. * just drop the test for now * refactor: comment out completion-related code in LLM and LLM event classes - Commented out the `completion` and `completion_cost` imports and their usage in the `LLM` class to prevent potential issues during execution. - Updated the `LLMCallCompletedEvent` class to comment out the `response_cost` attribute, ensuring consistency with the changes in the LLM class. - This refactor aims to streamline the code and prepare for future updates without affecting current functionality. * refactor: update LLM response handling in LiteAgent - Commented out the `response_cost` attribute in the LLM response handling to align with recent refactoring in the LLM class. - This change aims to maintain consistency in the codebase and prepare for future updates without affecting current functionality. * refactor: remove commented-out response cost attributes in LLM and LiteAgent - Commented out the `response_cost` attribute in both the `LiteAgent` and `LLM` classes to maintain consistency with recent refactoring efforts. - This change aligns with previous updates aimed at streamlining the codebase and preparing for future enhancements without impacting current functionality. * bring back litellm upgrade version	2025-08-06 14:05:14 -07:00