Commit Graph

1570 Commits

Author SHA1 Message Date
Lucas Gomide
2ab33ac549 Merge remote-tracking branch 'origin/main' into lg-agent-experiments 2025-07-11 14:58:16 -03:00
Vidit Ostwal
eec1262d4f Fix agent knowledge (#2831)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
* Added add_sources()

* Fixed the agent knowledge querying

* Added test cases

* Fixed linting issue

* Fixed logic

* Seems like a falky test case

* Minor changes

* Added knowledge attriute to the crew documentation

* Flaky test

* fixed spaces

* Flaky Test Case

* Seems like a flaky test case

---------

Co-authored-by: Lucas Gomide <lucaslg200@gmail.com>
2025-07-11 13:52:26 -04:00
Lucas Gomide
4439755b2d feat: add experimental folder for beta features (#3141) 2025-07-11 13:43:29 -04:00
Tony Kipkemboi
c6caa763d7 docs: Add guardrail attribute documentation and examples (#3139)
- Document string-based guardrails in tasks
- Add guardrail examples to YAML configuration
- Fix Python code formatting in PT-BR CLI docs
2025-07-11 13:32:59 -04:00
Lucas Gomide
08fa3797ca Introducing Agent evaluation (#3130)
* feat: add exchanged messages in LLMCallCompletedEvent

* feat: add GoalAlignment metric for Agent evaluation

* feat: add SemanticQuality metric for Agent evaluation

* feat: add Tool Metrics for Agent evaluation

* feat: add Reasoning Metrics for Agent evaluation, still in progress

* feat: add AgentEvaluator class

This class will evaluate Agent' results and report to user

* fix: do not evaluate Agent by default

This is a experimental feature we still need refine it further

* test: add Agent eval tests

* fix: render all feedback per iteration

* style: resolve linter issues

* style: fix mypy issues

* fix: allow messages be empty on LLMCallCompletedEvent
2025-07-11 13:18:03 -04:00
Lucas Gomide
e3b044c044 style: resolve linter issues 2025-07-11 14:09:22 -03:00
Lucas Gomide
ee490a19fb fix: encode string before hashing 2025-07-11 14:09:22 -03:00
Lucas Gomide
de425450d4 style: fix linter issues 2025-07-11 14:09:22 -03:00
Lucas Gomide
3fee619798 test: add tests for Evaluator Experiment 2025-07-11 14:09:22 -03:00
Lucas Gomide
bb2a0f95ba fix: use time-aware to define experiment result 2025-07-11 14:09:22 -03:00
Lucas Gomide
22d46999fe fix: make crew required to run an experiment 2025-07-11 14:09:22 -03:00
Lucas Gomide
9f00760437 refactor: isolate Console print in a dedicated class 2025-07-11 14:09:22 -03:00
Lucas Gomide
ffab51ce2c refactor: remove unused method 2025-07-11 14:09:22 -03:00
Lucas Gomide
007f396f4d chore: split Experimental evaluation classes 2025-07-11 14:09:22 -03:00
Lucas Gomide
cf91084f21 fix: fix track of new test cases 2025-07-11 14:09:22 -03:00
Lucas Gomide
ccea8870d7 fix: reset evaluator for each experiement iteraction 2025-07-11 14:09:22 -03:00
Lucas Gomide
15f1912c2e feat: add Experiment evaluation framework with baseline comparison 2025-07-11 14:09:22 -03:00
Lucas Gomide
064997464e fix: allow messages be empty on LLMCallCompletedEvent 2025-07-11 14:05:25 -03:00
Lucas Gomide
6f0ed6642b style: fix mypy issues 2025-07-11 13:02:34 -03:00
Lucas Gomide
43f339fa84 style: resolve linter issues 2025-07-11 13:02:34 -03:00
Lucas Gomide
5ea221e54e fix: render all feedback per iteration 2025-07-11 13:02:34 -03:00
Lucas Gomide
d4c15ec25f test: add Agent eval tests 2025-07-11 13:02:34 -03:00
Lucas Gomide
37cfbe7389 fix: do not evaluate Agent by default
This is a experimental feature we still need refine it further
2025-07-11 13:02:34 -03:00
Lucas Gomide
6d7c7d940e feat: add AgentEvaluator class
This class will evaluate Agent' results and report to user
2025-07-11 13:02:34 -03:00
Lucas Gomide
80bd23a8a9 feat: add Reasoning Metrics for Agent evaluation, still in progress 2025-07-11 13:02:34 -03:00
Lucas Gomide
50593d1485 feat: add Tool Metrics for Agent evaluation 2025-07-11 13:02:34 -03:00
Lucas Gomide
60084af745 feat: add SemanticQuality metric for Agent evaluation 2025-07-11 13:02:34 -03:00
Lucas Gomide
be4ade8c45 feat: add GoalAlignment metric for Agent evaluation 2025-07-11 13:02:34 -03:00
Lucas Gomide
6a49a24810 feat: add exchanged messages in LLMCallCompletedEvent 2025-07-11 13:02:34 -03:00
Greyson LaLonde
bf8fa3232b Add SQLite FTS5 support to test workflow (#3140)
* Add SQLite FTS5 support to test workflow

* Add explanatory comment for SQLite FTS5 workaround
2025-07-11 12:01:25 -04:00
Heitor Carvalho
a6e60a5d42 fix: use production workos environment id (#3129)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
2025-07-09 17:09:01 -04:00
Lorenze Jay
7b0f3aabd9 chore: update crewAI and dependencies to version 0.141.0 and 0.51.0 (#3128)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
- Bump crewAI version to 0.141.0 in __init__.py for alignment with updated dependencies.
- Update `crewai-tools` dependency version to 0.51.0 in pyproject.toml and related template files.
- Add new testing dependencies: pytest-split and pytest-xdist for improved test execution.
- Ensure compatibility with the latest package versions in uv.lock and template files.
0.141.0
2025-07-09 10:37:06 -07:00
Lucas Gomide
f071966951 docs: add docs about Agent.kickoff usage (#3121)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
Co-authored-by: Tony Kipkemboi <iamtonykipkemboi@gmail.com>
2025-07-08 16:15:40 -04:00
Lucas Gomide
318310bb7a docs: add docs about Agent repository (#3122) 2025-07-08 15:56:08 -04:00
Greyson LaLonde
34a03f882c feat: add crew context tracking for LLM guardrail events (#3111)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
Add crew context tracking using OpenTelemetry baggage for thread-safe propagation. Context is set during kickoff and cleaned up in finally block. Added thread safety tests with mocked agent execution.
2025-07-07 16:33:07 -04:00
Greyson LaLonde
a0fcc0c8d1 Speed up GitHub Actions tests with parallelization (#3107)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
- Add pytest-xdist and pytest-split to dev dependencies for parallel test execution
- Split tests into 8 parallel groups per Python version for better distribution
- Enable CPU-level parallelization with -n auto to maximize resource usage
- Add fail-fast strategy and maxfail=3 to stop early on failures
- Add job name to match branch protection rules
- Reduce test timeout from default to 30s for faster failure detection
- Remove redundant cache configuration
2025-07-03 21:08:00 -04:00
Lorenze Jay
748c25451c Lorenze/new version 0.140.0 (#3106)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
* fix: clean up whitespace and update dependencies

* Removed unnecessary whitespace in multiple files for consistency.
* Updated `crewai-tools` dependency version to `0.49.0` in `pyproject.toml` and related template files.
* Bumped CrewAI version to `0.140.0` in `__init__.py` for alignment with updated dependencies.

* chore: update pyproject.toml to exclude documentation from build targets

* Added exclusions for the `docs` directory in both wheel and sdist build targets to streamline the build process and reduce unnecessary file inclusion.

* chore: update uv.lock for dependency resolution and Python version compatibility

* Incremented revision to 2.
* Updated resolution markers to include support for Python 3.13 and adjusted platform checks for better compatibility.
* Added new wheel URLs for zstandard version 0.23.0 to ensure availability across various platforms.

* chore: pin json-repair dependency version in pyproject.toml and uv.lock

* Updated json-repair dependency from a range to a specific version (0.25.2) for consistency and to avoid potential compatibility issues.
* Adjusted related entries in uv.lock to reflect the pinned version, ensuring alignment across project files.

* chore: pin agentops dependency version in pyproject.toml and uv.lock

* Updated agentops dependency from a range to a specific version (0.3.18) for consistency and to avoid potential compatibility issues.
* Adjusted related entries in uv.lock to reflect the pinned version, ensuring alignment across project files.

* test: enhance cache call assertions in crew tests

* Improved the test for cache hitting between agents by filtering mock calls to ensure they include the expected 'tool' and 'input' keywords.
* Added assertions to verify the number of cache calls and their expected arguments, enhancing the reliability of the test.
* Cleaned up whitespace and improved readability in various test cases for better maintainability.
0.140.0
2025-07-02 15:22:18 -07:00
Heitor Carvalho
a77dcdd419 feat: add multiple provider support (#3089)
* Remove `crewai signup` command, update docs

* Add `Settings.clear()` and clear settings before each login

* Add pyjwt

* Remove print statement from ToolCommand.login()

* Remove auth0 dependency

* Update docs
2025-07-02 16:44:47 -04:00
Greyson LaLonde
68f5bdf0d9 feat: add console logging for LLM guardrail events (#3105)
* feat: add console logging for memory events

* fix: emit guardrail events in correct order and handle exceptions

* fix: remove unreachable elif conditions in guardrail event listener

* fix: resolve mypy type errors in guardrail event handler
2025-07-02 16:19:22 -04:00
Irineu Brito
7f83947020 fix: correct code example language inconsistency in pt-BR docs (#3088)
* fix: correct code example language inconsistency in pt-BR docs

* fix: fix: fully standardize code example language and naming in pt-BR docs

* fix: fix: fully standardize code example language and naming in pt-BR docs fixed variables

* fix: fix: fully standardize code example language and naming in pt-BR docs fixed params

---------

Co-authored-by: Lucas Gomide <lucaslg200@gmail.com>
2025-07-02 12:18:32 -04:00
Lucas Gomide
ceb310bcde docs: add docs about Memory Events (#3104)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
2025-07-02 11:10:45 -04:00
Lucas Gomide
ae57e5723c feat: add console logging for memory system usage (#3103) 2025-07-02 11:00:26 -04:00
Lucas Gomide
ab39753a75 Introduce MemoryEvents to monitor their usage (#3098)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
* feat: emit events about memory usage

* test: add tests about memory events usage

* fixed linter issues

* test: use scoped_handlers while listener Memory events
2025-07-01 22:50:39 -04:00
Tony Kipkemboi
640e1a7bc2 Add docs redirects and development tools (#3096)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
* Add Reo.dev tracking script to documentation

* Comprehensive docs improvements and development tools

- Add comprehensive .cursorrules with CrewAI and Flow development patterns
- Add redirect rules for old doc links without /en/ prefix
- Replace changelog pages with direct GitHub releases links
- Fix installation page directory tree rendering issue
- Fix broken Visual Studio Build Tools link formatting
- Remove obsolete changelog files to reduce maintenance overhead

These changes improve developer experience and ensure all old documentation links continue working.
2025-07-01 14:41:34 -04:00
Lorenze Jay
e544ff8ba3 refactor: streamline collection handling in RAGStorage (#3097)
Replaced the try-except block for collection retrieval with a single call to get_or_create_collection, simplifying the code and improving readability. Added logging to confirm whether the collection was found or created.
2025-07-01 10:14:39 -07:00
Lucas Gomide
49c0144154 feat: improve data training for models up to 7B parameters (#3085)
* feat: improve data training for models up to 7B parameters.

* docs: training considerations for small models to the documentation
2025-07-01 11:47:47 -04:00
Tony Kipkemboi
2ab002a5bf Add Reo.dev tracking script to documentation (#3094)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
2025-07-01 10:29:28 -04:00
Lucas Gomide
b7bf15681e feat: add capability to track LLM calls by task and agent (#3087)
* feat: add capability to track LLM calls by task and agent

This makes it possible to filter or scope LLM events by specific agents or tasks, which can be very useful for debugging or analytics in real-time application

* feat: add docs about LLM tracking by Agents and Tasks

* fix incompatible BaseLLM.call method signature

* feat: support to filter LLM Events from Lite Agent
2025-07-01 09:30:16 -04:00
Tony Kipkemboi
af9c01f5d3 Add Scarf analytics tracking (#3086)
Some checks failed
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
* Add Scarf analytics tracking

* Fix bandit security warning for urlopen

* Fix linting errors

* Refactor telemetry: reuse existing logic and simplify exceptions
2025-06-30 17:48:45 -04:00
Irineu Brito
5a12b51ba2 fix: Correct typo 'depployments' to 'deployments' in documentation 'instalation' (#3081) 2025-06-30 12:19:31 -04:00