Lucas Gomide 08fa3797ca Introducing Agent evaluation (#3130) ...

* feat: add exchanged messages in LLMCallCompletedEvent

* feat: add GoalAlignment metric for Agent evaluation

* feat: add SemanticQuality metric for Agent evaluation

* feat: add Tool Metrics for Agent evaluation

* feat: add Reasoning Metrics for Agent evaluation, still in progress

* feat: add AgentEvaluator class

This class will evaluate Agent' results and report to user

* fix: do not evaluate Agent by default

This is a experimental feature we still need refine it further

* test: add Agent eval tests

* fix: render all feedback per iteration

* style: resolve linter issues

* style: fix mypy issues

* fix: allow messages be empty on LLMCallCompletedEvent

2025-07-11 13:18:03 -04:00

..

agents

chore: add missing __init__.py files (#2719)

2025-04-29 07:35:26 -07:00

cassettes

Introducing Agent evaluation (#3130)

2025-07-11 13:18:03 -04:00

cli

fix: use production workos environment id (#3129)

2025-07-09 17:09:01 -04:00

config

Supporting no-code Guardrail creation (#2636)

2025-04-30 10:47:58 -04:00

evaluation

Introducing Agent evaluation (#3130)

2025-07-11 13:18:03 -04:00

knowledge

pytest improvements to handle flaky test (#2726)

2025-05-01 15:48:29 -04:00

memory

Introduce MemoryEvents to monitor their usage (#3098)

2025-07-01 22:50:39 -04:00

pipeline

remove all references to pipeline and pipeline router (#1661)

2024-12-04 12:39:34 -05:00

security

adding fingerprints (#2332)

2025-03-14 03:00:30 -03:00

storage

chore: add missing __init__.py files (#2719)

2025-04-29 07:35:26 -07:00

telemetry

Fix telemetry singleton pattern to respect dynamic environment variables (#2946)

2025-06-10 17:38:40 -07:00

tools

Support async tool executions (#2983)

2025-06-10 12:17:06 -04:00

utilities

feat: add crew context tracking for LLM guardrail events (#3111)

2025-07-07 16:33:07 -04:00

__init__.py

first stab at early concepts

2023-10-29 19:51:59 -03:00

agent_reasoning_test.py

Add reasoning attribute to Agent class (#2866)

2025-05-20 07:40:40 -07:00

agent_test.py

feat: support to initialize a tool from defined Tool attributes (#3023)

2025-06-20 10:53:37 -04:00

conftest.py

Add inject_date flag to Agent for automatic date injection (#2870)

2025-05-21 12:58:57 -07:00

crew_test.py

Lorenze/new version 0.140.0 (#3106)

2025-07-02 15:22:18 -07:00

custom_llm_test.py

feat: add capability to track LLM calls by task and agent (#3087)

2025-07-01 09:30:16 -04:00

flow_test.py

Support multiple router calls and address issue #2175 (#2231)

2025-02-26 13:42:17 -05:00

imports_test.py

Fix #2547: Add TaskOutput and CrewOutput to public exports

2025-04-09 09:35:05 +00:00

llm_test.py

Introducing Agent evaluation (#3130)

2025-07-11 13:18:03 -04:00

project_test.py

feat: enhance CrewBase MCP tools support to allow selecting multiple tools per agent (#3065)

2025-06-25 14:59:55 -04:00

task_test.py

Add inject_date flag to Agent for automatic date injection (#2870)

2025-05-21 12:58:57 -07:00

test_agent_inject_date.py

Add inject_date flag to Agent for automatic date injection (#2870)

2025-05-21 12:58:57 -07:00

test_crew_thread_safety.py

feat: add crew context tracking for LLM guardrail events (#3111)

2025-07-07 16:33:07 -04:00

test_flow_default_override.py

Stateful flows (#1931)

2025-01-20 13:30:09 -03:00

test_flow_human_input_integration.py

Fix issue 2993: Prevent Flow status logs from hiding human input (#2994)

2025-06-11 12:08:00 -04:00

test_flow_persistence.py

WIP crew events emitter (#2048)

2025-02-19 13:52:47 -08:00

test_hallucination_guardrail.py

Add HallucinationGuardrail no-op implementation with tests (#2869)

2025-05-21 13:47:41 -04:00

test_lite_agent.py

Update test_lite_agent.py (#3040)

2025-06-26 09:55:53 -04:00

test_markdown_task.py

Add markdown attribute to Task class (#2865)

2025-05-19 23:26:03 -07:00

test_multimodal_validation.py

fix: update LLMCallStartedEvent message type to support multimodal content (#2475)

2025-03-26 16:29:15 -03:00

test_task_guardrails.py

Add HallucinationGuardrail no-op implementation with tests (#2869)

2025-05-21 13:47:41 -04:00