Commit Graph

813 Commits

Author SHA1 Message Date
Lucas Gomide
3fee619798 test: add tests for Evaluator Experiment 2025-07-11 14:09:22 -03:00
Lucas Gomide
bb2a0f95ba fix: use time-aware to define experiment result 2025-07-11 14:09:22 -03:00
Lucas Gomide
22d46999fe fix: make crew required to run an experiment 2025-07-11 14:09:22 -03:00
Lucas Gomide
9f00760437 refactor: isolate Console print in a dedicated class 2025-07-11 14:09:22 -03:00
Lucas Gomide
ffab51ce2c refactor: remove unused method 2025-07-11 14:09:22 -03:00
Lucas Gomide
007f396f4d chore: split Experimental evaluation classes 2025-07-11 14:09:22 -03:00
Lucas Gomide
cf91084f21 fix: fix track of new test cases 2025-07-11 14:09:22 -03:00
Lucas Gomide
ccea8870d7 fix: reset evaluator for each experiement iteraction 2025-07-11 14:09:22 -03:00
Lucas Gomide
15f1912c2e feat: add Experiment evaluation framework with baseline comparison 2025-07-11 14:09:22 -03:00
Lucas Gomide
064997464e fix: allow messages be empty on LLMCallCompletedEvent 2025-07-11 14:05:25 -03:00
Lucas Gomide
6f0ed6642b style: fix mypy issues 2025-07-11 13:02:34 -03:00
Lucas Gomide
43f339fa84 style: resolve linter issues 2025-07-11 13:02:34 -03:00
Lucas Gomide
5ea221e54e fix: render all feedback per iteration 2025-07-11 13:02:34 -03:00
Lucas Gomide
37cfbe7389 fix: do not evaluate Agent by default
This is a experimental feature we still need refine it further
2025-07-11 13:02:34 -03:00
Lucas Gomide
6d7c7d940e feat: add AgentEvaluator class
This class will evaluate Agent' results and report to user
2025-07-11 13:02:34 -03:00
Lucas Gomide
80bd23a8a9 feat: add Reasoning Metrics for Agent evaluation, still in progress 2025-07-11 13:02:34 -03:00
Lucas Gomide
50593d1485 feat: add Tool Metrics for Agent evaluation 2025-07-11 13:02:34 -03:00
Lucas Gomide
60084af745 feat: add SemanticQuality metric for Agent evaluation 2025-07-11 13:02:34 -03:00
Lucas Gomide
be4ade8c45 feat: add GoalAlignment metric for Agent evaluation 2025-07-11 13:02:34 -03:00
Lucas Gomide
6a49a24810 feat: add exchanged messages in LLMCallCompletedEvent 2025-07-11 13:02:34 -03:00
Heitor Carvalho
a6e60a5d42 fix: use production workos environment id (#3129)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
2025-07-09 17:09:01 -04:00
Lorenze Jay
7b0f3aabd9 chore: update crewAI and dependencies to version 0.141.0 and 0.51.0 (#3128)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
- Bump crewAI version to 0.141.0 in __init__.py for alignment with updated dependencies.
- Update `crewai-tools` dependency version to 0.51.0 in pyproject.toml and related template files.
- Add new testing dependencies: pytest-split and pytest-xdist for improved test execution.
- Ensure compatibility with the latest package versions in uv.lock and template files.
2025-07-09 10:37:06 -07:00
Greyson LaLonde
34a03f882c feat: add crew context tracking for LLM guardrail events (#3111)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
Add crew context tracking using OpenTelemetry baggage for thread-safe propagation. Context is set during kickoff and cleaned up in finally block. Added thread safety tests with mocked agent execution.
2025-07-07 16:33:07 -04:00
Lorenze Jay
748c25451c Lorenze/new version 0.140.0 (#3106)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
* fix: clean up whitespace and update dependencies

* Removed unnecessary whitespace in multiple files for consistency.
* Updated `crewai-tools` dependency version to `0.49.0` in `pyproject.toml` and related template files.
* Bumped CrewAI version to `0.140.0` in `__init__.py` for alignment with updated dependencies.

* chore: update pyproject.toml to exclude documentation from build targets

* Added exclusions for the `docs` directory in both wheel and sdist build targets to streamline the build process and reduce unnecessary file inclusion.

* chore: update uv.lock for dependency resolution and Python version compatibility

* Incremented revision to 2.
* Updated resolution markers to include support for Python 3.13 and adjusted platform checks for better compatibility.
* Added new wheel URLs for zstandard version 0.23.0 to ensure availability across various platforms.

* chore: pin json-repair dependency version in pyproject.toml and uv.lock

* Updated json-repair dependency from a range to a specific version (0.25.2) for consistency and to avoid potential compatibility issues.
* Adjusted related entries in uv.lock to reflect the pinned version, ensuring alignment across project files.

* chore: pin agentops dependency version in pyproject.toml and uv.lock

* Updated agentops dependency from a range to a specific version (0.3.18) for consistency and to avoid potential compatibility issues.
* Adjusted related entries in uv.lock to reflect the pinned version, ensuring alignment across project files.

* test: enhance cache call assertions in crew tests

* Improved the test for cache hitting between agents by filtering mock calls to ensure they include the expected 'tool' and 'input' keywords.
* Added assertions to verify the number of cache calls and their expected arguments, enhancing the reliability of the test.
* Cleaned up whitespace and improved readability in various test cases for better maintainability.
2025-07-02 15:22:18 -07:00
Heitor Carvalho
a77dcdd419 feat: add multiple provider support (#3089)
* Remove `crewai signup` command, update docs

* Add `Settings.clear()` and clear settings before each login

* Add pyjwt

* Remove print statement from ToolCommand.login()

* Remove auth0 dependency

* Update docs
2025-07-02 16:44:47 -04:00
Greyson LaLonde
68f5bdf0d9 feat: add console logging for LLM guardrail events (#3105)
* feat: add console logging for memory events

* fix: emit guardrail events in correct order and handle exceptions

* fix: remove unreachable elif conditions in guardrail event listener

* fix: resolve mypy type errors in guardrail event handler
2025-07-02 16:19:22 -04:00
Lucas Gomide
ae57e5723c feat: add console logging for memory system usage (#3103) 2025-07-02 11:00:26 -04:00
Lucas Gomide
ab39753a75 Introduce MemoryEvents to monitor their usage (#3098)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
* feat: emit events about memory usage

* test: add tests about memory events usage

* fixed linter issues

* test: use scoped_handlers while listener Memory events
2025-07-01 22:50:39 -04:00
Lorenze Jay
e544ff8ba3 refactor: streamline collection handling in RAGStorage (#3097)
Replaced the try-except block for collection retrieval with a single call to get_or_create_collection, simplifying the code and improving readability. Added logging to confirm whether the collection was found or created.
2025-07-01 10:14:39 -07:00
Lucas Gomide
49c0144154 feat: improve data training for models up to 7B parameters (#3085)
* feat: improve data training for models up to 7B parameters.

* docs: training considerations for small models to the documentation
2025-07-01 11:47:47 -04:00
Lucas Gomide
b7bf15681e feat: add capability to track LLM calls by task and agent (#3087)
* feat: add capability to track LLM calls by task and agent

This makes it possible to filter or scope LLM events by specific agents or tasks, which can be very useful for debugging or analytics in real-time application

* feat: add docs about LLM tracking by Agents and Tasks

* fix incompatible BaseLLM.call method signature

* feat: support to filter LLM Events from Lite Agent
2025-07-01 09:30:16 -04:00
Tony Kipkemboi
af9c01f5d3 Add Scarf analytics tracking (#3086)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
* Add Scarf analytics tracking

* Fix bandit security warning for urlopen

* Fix linting errors

* Refactor telemetry: reuse existing logic and simplify exceptions
2025-06-30 17:48:45 -04:00
Lucas Gomide
b35c3e8024 fix: ensure env-vars are written in upper case (#3072)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
When creating a Crew via the CLI and selecting the Azure provider, the generated .env file had environment variables in lowercase.
This commit ensures that all environment variables are written in uppercase.
2025-06-26 12:29:06 -04:00
devin-ai-integration[bot]
e0b46492fa Fix: Normalize project names by stripping trailing slashes in crew creation (#3060)
* fix: normalize project names by stripping trailing slashes in crew creation

- Strip trailing slashes from project names in create_folder_structure
- Add comprehensive tests for trailing slash scenarios
- Fixes #3059

The issue occurred because trailing slashes in project names like 'hello/'
were directly incorporated into pyproject.toml, creating invalid package
names and script entries. This fix silently normalizes project names by
stripping trailing slashes before processing, maintaining backward
compatibility while fixing the invalid template generation.

Co-Authored-By: João <joao@crewai.com>

* trigger CI re-run to check for flaky test issue

Co-Authored-By: João <joao@crewai.com>

* fix: resolve circular import in CLI authentication module

- Move ToolCommand import to be local inside _poll_for_token method
- Update test mock to patch ToolCommand at correct location
- Resolves Python 3.11 test collection failure in CI

Co-Authored-By: João <joao@crewai.com>

* feat: add comprehensive class name validation for Python identifiers

- Ensure generated class names are always valid Python identifiers
- Handle edge cases: names starting with numbers, special characters, keywords, built-ins
- Add sanitization logic to remove invalid characters and prefix with 'Crew' when needed
- Add comprehensive test coverage for class name validation edge cases
- Addresses GitHub PR comment from lucasgomide about class name validity

Fixes include:
- Names starting with numbers: '123project' -> 'Crew123Project'
- Python built-ins: 'True' -> 'TrueCrew', 'False' -> 'FalseCrew'
- Special characters: 'hello@world' -> 'HelloWorld'
- Empty/whitespace: '   ' -> 'DefaultCrew'
- All generated class names pass isidentifier() and keyword checks

Co-Authored-By: João <joao@crewai.com>

* refactor: change class name validation to raise errors instead of generating defaults

- Remove default value generation (Crew prefix/suffix, DefaultCrew fallback)
- Raise ValueError with descriptive messages for invalid class names
- Update tests to expect validation errors instead of default corrections
- Addresses GitHub comment feedback from lucasgomide about strict validation

Co-Authored-By: João <joao@crewai.com>

* fix: add working directory safety checks to prevent test interference

Co-Authored-By: João <joao@crewai.com>

* fix: standardize working directory handling in tests to prevent corruption

Co-Authored-By: João <joao@crewai.com>

* fix: eliminate os.chdir() usage in tests to prevent working directory corruption

- Replace os.chdir() with parent_folder parameter for create_folder_structure tests
- Mock create_folder_structure directly for create_crew tests to avoid directory changes
- All 12 tests now pass locally without working directory corruption
- Should resolve the 103 failing tests in Python 3.12 CI

Co-Authored-By: João <joao@crewai.com>

* fix: remove unused os import to resolve lint failure

- Remove unused 'import os' statement from test_create_crew.py
- All tests still pass locally after removing unused import
- Should resolve F401 lint error in CI

Co-Authored-By: João <joao@crewai.com>

* feat: add folder name validation for Python module names

- Implement validation to ensure folder_name is valid Python identifier
- Check that folder names don't start with digits
- Validate folder names are not Python keywords
- Sanitize invalid characters from folder names
- Raise ValueError with descriptive messages for invalid cases
- Update tests to validate both folder and class name requirements
- Addresses GitHub comment requiring folder names to be valid Python module names

Co-Authored-By: João <joao@crewai.com>

* fix: correct folder name validation logic to match test expectations

- Fix validation regex to catch names starting with invalid characters like '@#/'
- Ensure validation properly raises ValueError for cases expected by tests
- Maintain support for valid cases like 'my.project/' -> 'myproject'
- Address lucasgomide's comment about valid Python module names

Co-Authored-By: João <joao@crewai.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: João <joao@crewai.com>
Co-authored-by: Lucas Gomide <lucaslg200@gmail.com>
2025-06-26 10:11:16 -04:00
Greyson LaLonde
ece13fbda0 refactor: implement PEP 621 dynamic versioning (#3068) 2025-06-26 10:02:26 -04:00
Lorenze Jay
0f861338ef chore: update version to 0.134.0 across project files (#3067)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
2025-06-25 16:06:43 -07:00
Lucas Gomide
4d1aabf620 feat: enhance CrewBase MCP tools support to allow selecting multiple tools per agent (#3065)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
* feat: enhance CrewBase MCP tools support to allow selecting multiple tools per agent

* docs: clarify how to access MCP tools

* build: upgrade crewai-tools
2025-06-25 14:59:55 -04:00
Lucas Gomide
f6dfec61d6 feat: add official way to use MCP Tools within a CrewBase (#3058)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
2025-06-24 15:14:59 -04:00
Lucas Gomide
8b176d0598 feat: improve Crew search while resetting their memories (#3057)
* test: add tests to test get_crews

* feat: improve Crew search while resetting their memories

Some memories couldn't be reset due to their reliance on relative external sources like `PDFKnowledge`. This was caused by the need to run the reset memories command from the `src` directory, which could break when external files weren't accessible from that path.

This commit allows the reset command to be executed from the root of the project — the same location typically used to run a crew — improving compatibility and reducing friction.

* feat: skip cli/templates folder while looking for Crew

* refactor: use console.print instead of print
2025-06-24 11:48:59 -04:00
Lucas Gomide
e9d8a853ea feat: support to initialize a tool from defined Tool attributes (#3023)
* feat: support to initialize a tool from defined Tool attributes

* fix: ensure Agent is able to load a list of Tools dynamically
2025-06-20 10:53:37 -04:00
Vidit Ostwal
463ea2b97f Fixed type annotation in task (#3021)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
* Added Union of List of Task, None, NotSpecified

* Seems like a flaky test

* Fixed run time issue

* Fixed Linting issues

* fix pydantic error

* aesthetic changes

---------

Co-authored-by: Lucas Gomide <lucaslg200@gmail.com>
2025-06-19 14:37:46 -04:00
Lucas Gomide
db1e9e9b9a fix: fix pydantic support to 2.7.x (#3016)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
Pydantic 2.7.x does not support a second parameter in model validators with mode="after"
2025-06-16 16:20:10 -04:00
Lucas Gomide
d92382b6cf fix: SSL error while getting LLM data from GH (#3014)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
When running behind cloud-based security users are struggling to donwload LLM data from Github. Usually the following error is raised

```
SSL certificate verification failed: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /BerriAI/litellm/main/model_prices_and_context_window.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1010)')))
Current CA bundle path: /usr/local/etc///.pem
```

This commit ensures the SSL config is beign provided while requesting data
2025-06-16 11:34:04 -04:00
Lorenze Jay
a5b5c8ab37 Lorenze/console printer nice (#3004)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
* fix: possible fix for Thinking stuck

* feat: add agent logging events for execution tracking

- Introduced AgentLogsStartedEvent and AgentLogsExecutionEvent to enhance logging capabilities during agent execution.
- Updated CrewAgentExecutor to emit these events at the start and during execution, respectively.
- Modified EventListener to handle the new logging events and format output accordingly in the console.
- Enhanced ConsoleFormatter to display agent logs in a structured format, improving visibility of agent actions and outputs.

* drop emoji

* refactor: improve code structure and logging in LiteAgent and ConsoleFormatter

- Refactored imports in lite_agent.py for better readability.
- Enhanced guardrail property initialization in LiteAgent.
- Updated logging functionality to emit AgentLogsExecutionEvent for better tracking.
- Modified ConsoleFormatter to include tool arguments and final output in status updates.
- Improved output formatting for long text in ConsoleFormatter.

* fix tests

---------

Co-authored-by: Eduardo Chiarotti <dudumelgaco@hotmail.com>
2025-06-14 12:21:46 -07:00
Vidit Ostwal
7f12e98de5 Added sanitize role feature in mem0 storage (#2988)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
* Added sanitize role feature in mme0 storage

* Used chroma db functionality
2025-06-12 13:14:34 -04:00
Lorenze Jay
99133104dd Update version to 0.130.0 and dependencies in pyproject.toml and uv.lock (#3002)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
- Bump CrewAI version from 0.126.0 to 0.130.0 in pyproject.toml and uv.lock.
- Update optional dependency 'crewai-tools' version from 0.46.0 to 0.47.1.
- Adjust dependency specifications in CLI templates to reflect the new version.
2025-06-11 17:01:11 -07:00
devin-ai-integration[bot]
970a63c13c Fix issue 2993: Prevent Flow status logs from hiding human input (#2994)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
* Fix issue 2993: Prevent Flow status logs from hiding human input

- Add pause_live_updates() and resume_live_updates() methods to ConsoleFormatter
- Modify _ask_human_input() to pause Flow status updates during human input
- Add comprehensive tests for pause/resume functionality and integration
- Ensure Live session is properly managed during human input prompts
- Fix prevents Flow status logs from overwriting user input prompts

Fixes #2993

Co-Authored-By: João <joao@crewai.com>

* Fix lint: Remove unused pytest import

- Remove unused pytest import from test_console_formatter_pause_resume.py
- Fixes F401 lint error identified in CI

Co-Authored-By: João <joao@crewai.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: João <joao@crewai.com>
2025-06-11 12:08:00 -04:00
devin-ai-integration[bot]
06c991d8c3 Fix telemetry singleton pattern to respect dynamic environment variables (#2946)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
* Fix telemetry singleton pattern to respect dynamic environment variables

- Modified Telemetry.__init__ to prevent re-initialization with _initialized flag
- Updated _safe_telemetry_operation to check _is_telemetry_disabled() dynamically
- Added comprehensive tests for environment variables set after singleton creation
- Fixed singleton contamination in existing tests by adding proper reset
- Resolves issue #2945 where CREWAI_DISABLE_TELEMETRY=true was ignored when set after import

Co-Authored-By: João <joao@crewai.com>

* Implement code review improvements

- Move _initialized flag to __new__ method for better encapsulation
- Add type hints to _safe_telemetry_operation method
- Consolidate telemetry execution checks into _should_execute_telemetry helper
- Add pytest fixtures to reduce test setup redundancy
- Enhanced documentation for singleton behavior

Co-Authored-By: João <joao@crewai.com>

* Fix mypy type-checker errors

- Add explicit bool type annotation to _initialized field
- Fix return value in task_started method to not return _safe_telemetry_operation result
- Simplify initialization logic to set _initialized once in __init__

Co-Authored-By: João <joao@crewai.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: João <joao@crewai.com>
Co-authored-by: Lorenze Jay <63378463+lorenzejay@users.noreply.github.com>
2025-06-10 17:38:40 -07:00
Lucas Gomide
739eb72fd0 LiteAgent w/ Guardrail (#2982)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
* feat: add guardrail support for Agents when using direct kickoff calls

* refactor: expose guardrail func in a proper utils file

* fix: resolve Self import on python 3.10
2025-06-10 13:32:32 -04:00
Lucas Gomide
5c51349a85 Support async tool executions (#2983)
* test: fix structured tool tests

No tests were being executed from this file

* feat: support to run async tool

Some Tool requires async execution. This commit allow us to collect tool result from coroutines

* docs: add docs about asynchronous tool support
2025-06-10 12:17:06 -04:00