mirror of
https://github.com/crewAIInc/crewAI.git
synced 2026-09-20 01:55:38 +00:00
4e0b2e2b15
* fix(telemetry): record task failures as failures, not as successes close_span() sets StatusCode.OK unconditionally, and TaskFailedEvent was routed to Telemetry.task_ended, which calls it. Every failed task was therefore exported as OK, which is why error_count downstream is not merely low but exactly zero: 240.0M task executions across 13 months in crew_task_executions_daily_target, error_count = 0 in every one of them. The same line had a second defect. The span was only ended when source.agent.crew was present, so a task failing without one was popped from the span map and then never closed - never ended, never exported, invisible rather than mislabelled. task_failed takes no crew (it reads nothing off one), so that condition disappears rather than being widened. Only the exception class name is recorded, never the message, which routinely contains prompts, model output, file paths and credentials. close_span_with_error drops any value failing str.isidentifier(), so a message cannot be recorded even if one is passed by mistake. This is the task half of closed PR #6781, re-cut onto main as that PR asked for. The crew half is deliberately left out: crew_execution_span() returns None unless share_crew=True, so crew._execution_span is None for nearly every user and a crew-failure handler would exit immediately for the default population. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RfV2uMqWRcdfufMvtdCVoN * fix(telemetry): take the exception class for error_type, not a free-form string cursor and CodeRabbit both flagged the sanitization, and they were right: the package already had a stronger convention and this change had not used it. Telemetry._safe_error_type takes the exception *class*, and its docstring says why in as many words - "a single-word message such as 'secret_token' is itself a valid identifier", so filtering a string with isidentifier() is not enough. The ported code predates that helper and reinvented the weaker check. TaskFailedEvent.error_type is now type[BaseException] | None, so pydantic itself rejects a message before any of our code runs, and task_failed routes it through _safe_error_type. The identifier check in close_span_with_error stays as the second gate on the derived name, which is the role _safe_error_type's docstring already describes. Also adds producer-level tests, which CodeRabbit correctly identified as missing: every earlier test constructed TaskFailedEvent directly, so a regression in the two emit sites this change touches in task.py would have passed the whole suite. The sync and async producers are driven through Task._execute_core and Task._aexecute_core with a distinctive exception class, and each patches a different agent method (execute_task vs aexecute_task), which is why they can regress independently. Verified by dropping error_type from both producers: all three new tests fail, and pass again when restored. Removes an unused `import os` left behind when the fixture was rewritten. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RfV2uMqWRcdfufMvtdCVoN * test(telemetry): capture producer failures at the emit boundary, not via the bus The producer tests subscribed a handler to crewai_event_bus and asserted on what it received. That passed this file in isolation and every randomized local run, then failed in CI inside a 621-test shard with zero events captured: FAILED tests/telemetry/test_task_failure_instrumentation.py:: test_sync_producer_puts_the_exception_class_on_the_event assert 0 == 1 + where 0 = len([]) task_failed is an "ending" event, and with an empty scope stack - there is no real kickoff in these tests - dispatch is conditional on event-context state that other tests in the same worker process can leave behind. Subscribing made the assertion depend on the bus choosing to dispatch, which is not what these tests are about: they are about what the producer in task.py constructs. Patching crewai_event_bus.emit records the event unconditionally at the point the producer hands it over, with no dispatch involved. Both producers ignore emit's return value, so returning None is faithful. Containment re-verified after the change: dropping error_type from both producers fails exactly these three tests and nothing else. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RfV2uMqWRcdfufMvtdCVoN * fix(events): keep TaskFailedEvent JSON-serializable with a class-valued error_type error_type holds an exception class, which is not a JSON type, so model_dump(mode="json") raised PydanticSerializationError for the whole event - not just that field. Two real consumers depend on it: the checkpoint listener dumps every event through EventRecord, and the tracing listener JSON-POSTs events to AMP. A single task failure therefore took out checkpointing. field_serializer with when_used="json" returns the class name. The "json" scope is load-bearing: event_listener hands the live class to Telemetry.task_failed, which needs it for _safe_error_type, so python-mode dumps must keep the class. The annotation is a module-level _ExceptionClass alias rather than an inline type[BaseException], because TaskFailedEvent declares a field named `type` which shadows the builtin for the rest of the class body - inline, it raises TypeError at import ("task_failed"[BaseException]) and mypy rejects it as "Variable ... is not valid as a type". Quoting satisfies neither tool: ruff flags UP037 and mypy still resolves it in the class scope. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RfV2uMqWRcdfufMvtdCVoN * fix(events): let a dumped error_type restore, instead of degrading the event The serializer added in the previous commit stopped model_dump(mode="json") from raising, but nothing accepted the class-name string back. _resolve_event (state/event_record.py:32-35) wraps cls.model_validate in a bare except and falls back to BaseEvent, so restoring a checkpoint after a task failure silently dropped the whole event -- including `error`, a plain string that would otherwise have survived. Traded a loud failure for a quiet one. Measured before: dumped error_type='ValueError' and error='boom', restored as BaseEvent with neither attribute. After: restores as TaskFailedEvent with error='boom' and error_type is ValueError. A BeforeValidator resolves a name against real exception classes only -- builtins first, then a walk of BaseException.__subclasses__(). So this does not reopen the hole the class-typed field closes: "secret_token" resolves to nothing, is returned unchanged, and is rejected by the field's own type. Asserted for secret_token, sk_live_1234, dict and os. A name whose class is not imported in this process still degrades, which is deliberate: synthesising a class from an arbitrary string is the injection risk this field exists to avoid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RfV2uMqWRcdfufMvtdCVoN --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Vidit Ostwal <110953813+Vidit-Ostwal@users.noreply.github.com>