Fix TaskEvaluation validation errors for missing quality and dict suggestions

Browse Source

Fixes #3915

This commit addresses Pydantic validation errors that occur when the LLM
output doesn't match the expected TaskEvaluation schema:

1. Missing 'quality' field - LLM sometimes omits this field
2. 'suggestions' as list of dicts - LLM returns [{'point': '...', 'priority': 'high'}]
   instead of list[str]
3. 'score' field instead of 'quality' - LLM uses 'score' as alternate field name

Changes:
- Make 'quality' field optional (float | None) with default None
- Make 'suggestions' field optional with default empty list
- Make 'entities' field optional with default empty list
- Add ConfigDict(extra='ignore') to ignore unexpected fields
- Add model_validator to map 'score' to 'quality' when quality is missing
- Add field_validator for 'suggestions' to normalize dict format to list[str]
  - Extracts 'point' value from dicts with 'point' key
  - Handles single dict, single string, list of mixed types, and None
- Add field_validator for 'quality' to coerce int/str to float

The fix is backward compatible - LongTermMemoryItem already accepts
quality=None, and the strict crew evaluation path uses a separate
TaskEvaluationPydanticOutput model that remains unchanged.

Tests:
- Added 16 comprehensive unit tests covering all edge cases
- All existing tests continue to pass
- Tests replicate exact error scenarios from issue #3915

Co-Authored-By: João <joao@crewai.com>

...

This commit is contained in:

Devin AI

2025-11-14 12:42:01 +00:00

parent d7bdac12a2

commit 60dd25aba8

3 changed files with 4570 additions and 4037 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

8260

uv.lock generated

File diff suppressed because it is too large Load Diff