Introducing Agent evaluation (#3130)

Browse Source

* feat: add exchanged messages in LLMCallCompletedEvent

* feat: add GoalAlignment metric for Agent evaluation

* feat: add SemanticQuality metric for Agent evaluation

* feat: add Tool Metrics for Agent evaluation

* feat: add Reasoning Metrics for Agent evaluation, still in progress

* feat: add AgentEvaluator class

This class will evaluate Agent' results and report to user

* fix: do not evaluate Agent by default

This is a experimental feature we still need refine it further

* test: add Agent eval tests

* fix: render all feedback per iteration

* style: resolve linter issues

* style: fix mypy issues

* fix: allow messages be empty on LLMCallCompletedEvent

...

This commit is contained in:

Lucas Gomide

2025-07-11 14:18:03 -03:00

committed by GitHub

parent bf8fa3232b

commit 08fa3797ca

26 changed files with 2930 additions and 14 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

430

tests/cassettes/TestAgentEvaluator.test_evaluate_current_iteration.yaml Normal file

File diff suppressed because one or more lines are too long