mirror of
https://github.com/crewAIInc/crewAI.git
synced 2026-08-11 00:42:48 +00:00
328e0e36e9
Task and crew failures were indistinguishable from successes in telemetry,
which is why error_count is zero for every month in the downstream
aggregates rather than merely low.
Three separate defects:
1. Task failures were recorded as successes. TaskFailedEvent routed to
Telemetry.task_ended, which calls close_span() - and close_span
unconditionally sets StatusCode.OK. Every failed task was exported as
OK, so no downstream query could ever count one.
2. Crew failures were not recorded at all, and leaked their span.
on_crew_failed never touched telemetry, so a crew that raised left
_execution_span open: never ended, never exported. The failure was
invisible and the span was lost entirely.
3. Some task failures leaked their span too. on_task_failed only ended the
span when source.agent.crew was present, so a task failing without one
was popped from the span map and never closed.
Changes:
- Add close_span_with_error(), which sets StatusCode.ERROR and optionally
records an error_type attribute.
- Add Telemetry.task_failed() and Telemetry.crew_failed(); crew_failed
clears _execution_span so it cannot be double-closed.
- Wire TaskFailedEvent and CrewKickoffFailedEvent to them, closing spans
unconditionally so neither can leak.
- Add optional error_type to TaskFailedEvent and CrewKickoffFailedEvent,
populated with type(e).__name__ at the four emit sites. Defaults to None,
so existing callers are unaffected.
PII: only the exception *class name* is recorded, never the message, which
routinely contains prompts, model output, and credentials. close_span_with_error
drops any value failing str.isidentifier(), so a message cannot be recorded
even if passed by mistake. Tests assert this against six message-shaped
inputs.
Tests: new tests/telemetry/test_failure_instrumentation.py (16 tests) covering
error status, the success/failure distinction, the PII guard, span-leak
regressions for both task and crew, and event backwards compatibility. The
module sets OTEL_SDK_DISABLED explicitly - the suite runs with the SDK
disabled and the root conftest pops the variable on teardown, so tests that
need real spans must not rely on that leak.
Note: total_duration_ms is a separate, pipeline-side issue. The raw `duration`
column is a Go-style string ("2.026641s"), so toInt64OrZero() yields 0 for
99.99% of rows. That fix belongs in the ClickHouse materialized views, not here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UNumDnNbiyw3pv1WakAe6t