Devin AI 60dd25aba8 Fix TaskEvaluation validation errors for missing quality and dict suggestions ...

Fixes #3915

This commit addresses Pydantic validation errors that occur when the LLM
output doesn't match the expected TaskEvaluation schema:

1. Missing 'quality' field - LLM sometimes omits this field
2. 'suggestions' as list of dicts - LLM returns [{'point': '...', 'priority': 'high'}]
   instead of list[str]
3. 'score' field instead of 'quality' - LLM uses 'score' as alternate field name

Changes:
- Make 'quality' field optional (float | None) with default None
- Make 'suggestions' field optional with default empty list
- Make 'entities' field optional with default empty list
- Add ConfigDict(extra='ignore') to ignore unexpected fields
- Add model_validator to map 'score' to 'quality' when quality is missing
- Add field_validator for 'suggestions' to normalize dict format to list[str]
  - Extracts 'point' value from dicts with 'point' key
  - Handles single dict, single string, list of mixed types, and None
- Add field_validator for 'quality' to coerce int/str to float

The fix is backward compatible - LongTermMemoryItem already accepts
quality=None, and the strict crew evaluation path uses a separate
TaskEvaluationPydanticOutput model that remains unchanged.

Tests:
- Added 16 comprehensive unit tests covering all edge cases
- All existing tests continue to pass
- Tests replicate exact error scenarios from issue #3915

Co-Authored-By: João <joao@crewai.com>

2025-11-14 12:42:01 +00:00

1.2 MiB

Raw Permalink Blame History

View Raw