mirror of
https://github.com/crewAIInc/crewAI.git
synced 2026-09-22 10:56:50 +00:00
4e0b2e2b15
* fix(telemetry): record task failures as failures, not as successes close_span() sets StatusCode.OK unconditionally, and TaskFailedEvent was routed to Telemetry.task_ended, which calls it. Every failed task was therefore exported as OK, which is why error_count downstream is not merely low but exactly zero: 240.0M task executions across 13 months in crew_task_executions_daily_target, error_count = 0 in every one of them. The same line had a second defect. The span was only ended when source.agent.crew was present, so a task failing without one was popped from the span map and then never closed - never ended, never exported, invisible rather than mislabelled. task_failed takes no crew (it reads nothing off one), so that condition disappears rather than being widened. Only the exception class name is recorded, never the message, which routinely contains prompts, model output, file paths and credentials. close_span_with_error drops any value failing str.isidentifier(), so a message cannot be recorded even if one is passed by mistake. This is the task half of closed PR #6781, re-cut onto main as that PR asked for. The crew half is deliberately left out: crew_execution_span() returns None unless share_crew=True, so crew._execution_span is None for nearly every user and a crew-failure handler would exit immediately for the default population. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RfV2uMqWRcdfufMvtdCVoN * fix(telemetry): take the exception class for error_type, not a free-form string cursor and CodeRabbit both flagged the sanitization, and they were right: the package already had a stronger convention and this change had not used it. Telemetry._safe_error_type takes the exception *class*, and its docstring says why in as many words - "a single-word message such as 'secret_token' is itself a valid identifier", so filtering a string with isidentifier() is not enough. The ported code predates that helper and reinvented the weaker check. TaskFailedEvent.error_type is now type[BaseException] | None, so pydantic itself rejects a message before any of our code runs, and task_failed routes it through _safe_error_type. The identifier check in close_span_with_error stays as the second gate on the derived name, which is the role _safe_error_type's docstring already describes. Also adds producer-level tests, which CodeRabbit correctly identified as missing: every earlier test constructed TaskFailedEvent directly, so a regression in the two emit sites this change touches in task.py would have passed the whole suite. The sync and async producers are driven through Task._execute_core and Task._aexecute_core with a distinctive exception class, and each patches a different agent method (execute_task vs aexecute_task), which is why they can regress independently. Verified by dropping error_type from both producers: all three new tests fail, and pass again when restored. Removes an unused `import os` left behind when the fixture was rewritten. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RfV2uMqWRcdfufMvtdCVoN * test(telemetry): capture producer failures at the emit boundary, not via the bus The producer tests subscribed a handler to crewai_event_bus and asserted on what it received. That passed this file in isolation and every randomized local run, then failed in CI inside a 621-test shard with zero events captured: FAILED tests/telemetry/test_task_failure_instrumentation.py:: test_sync_producer_puts_the_exception_class_on_the_event assert 0 == 1 + where 0 = len([]) task_failed is an "ending" event, and with an empty scope stack - there is no real kickoff in these tests - dispatch is conditional on event-context state that other tests in the same worker process can leave behind. Subscribing made the assertion depend on the bus choosing to dispatch, which is not what these tests are about: they are about what the producer in task.py constructs. Patching crewai_event_bus.emit records the event unconditionally at the point the producer hands it over, with no dispatch involved. Both producers ignore emit's return value, so returning None is faithful. Containment re-verified after the change: dropping error_type from both producers fails exactly these three tests and nothing else. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RfV2uMqWRcdfufMvtdCVoN * fix(events): keep TaskFailedEvent JSON-serializable with a class-valued error_type error_type holds an exception class, which is not a JSON type, so model_dump(mode="json") raised PydanticSerializationError for the whole event - not just that field. Two real consumers depend on it: the checkpoint listener dumps every event through EventRecord, and the tracing listener JSON-POSTs events to AMP. A single task failure therefore took out checkpointing. field_serializer with when_used="json" returns the class name. The "json" scope is load-bearing: event_listener hands the live class to Telemetry.task_failed, which needs it for _safe_error_type, so python-mode dumps must keep the class. The annotation is a module-level _ExceptionClass alias rather than an inline type[BaseException], because TaskFailedEvent declares a field named `type` which shadows the builtin for the rest of the class body - inline, it raises TypeError at import ("task_failed"[BaseException]) and mypy rejects it as "Variable ... is not valid as a type". Quoting satisfies neither tool: ruff flags UP037 and mypy still resolves it in the class scope. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RfV2uMqWRcdfufMvtdCVoN * fix(events): let a dumped error_type restore, instead of degrading the event The serializer added in the previous commit stopped model_dump(mode="json") from raising, but nothing accepted the class-name string back. _resolve_event (state/event_record.py:32-35) wraps cls.model_validate in a bare except and falls back to BaseEvent, so restoring a checkpoint after a task failure silently dropped the whole event -- including `error`, a plain string that would otherwise have survived. Traded a loud failure for a quiet one. Measured before: dumped error_type='ValueError' and error='boom', restored as BaseEvent with neither attribute. After: restores as TaskFailedEvent with error='boom' and error_type is ValueError. A BeforeValidator resolves a name against real exception classes only -- builtins first, then a walk of BaseException.__subclasses__(). So this does not reopen the hole the class-typed field closes: "secret_token" resolves to nothing, is returned unchanged, and is rejected by the field's own type. Asserted for secret_token, sk_live_1234, dict and os. A name whose class is not imported in this process still degrades, which is deliberate: synthesising a class from an arbitrary string is the injection risk this field exists to avoid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RfV2uMqWRcdfufMvtdCVoN --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Vidit Ostwal <110953813+Vidit-Ostwal@users.noreply.github.com>
87 lines
9.3 KiB
Plaintext
87 lines
9.3 KiB
Plaintext
---
|
|
title: Telemetry
|
|
description: Understanding the telemetry data collected by CrewAI and how it contributes to the enhancement of the library.
|
|
icon: signal-stream
|
|
mode: "wide"
|
|
---
|
|
|
|
## Telemetry
|
|
|
|
<Note>
|
|
By default, we collect no data that would be considered personal information under GDPR and other privacy regulations.
|
|
We do collect Tool's names and Agent's roles, so be advised not to include any personal information in the tool's names or the Agent's roles.
|
|
Because no personal information is collected, it's not necessary to worry about data residency.
|
|
When `share_crew` is enabled, additional data is collected which may contain personal information if included by the user.
|
|
Users should exercise caution when enabling this feature to ensure compliance with privacy regulations.
|
|
</Note>
|
|
|
|
CrewAI utilizes anonymous telemetry to gather usage statistics with the primary goal of enhancing the library.
|
|
Our focus is on improving and developing the features, integrations, and tools most utilized by our users.
|
|
|
|
It's pivotal to understand that by default, **NO personal data is collected** concerning prompts, task descriptions, agents' backstories or goals,
|
|
usage of tools, API calls, responses, any data processed by the agents, or secrets and environment variables.
|
|
When the `share_crew` feature is enabled, detailed data including task descriptions, agents' backstories or goals, and other specific attributes are collected
|
|
to provide deeper insights. This expanded data collection may include personal information if users have incorporated it into their crews or tasks.
|
|
Users should carefully consider the content of their crews and tasks before enabling `share_crew`.
|
|
Users can disable telemetry by setting the environment variable `CREWAI_DISABLE_TELEMETRY` to `true` or by setting `OTEL_SDK_DISABLED` to `true` (note that the latter disables all OpenTelemetry instrumentation globally).
|
|
|
|
### Examples:
|
|
```python
|
|
# Disable CrewAI telemetry only
|
|
os.environ['CREWAI_DISABLE_TELEMETRY'] = 'true'
|
|
|
|
# Disable all OpenTelemetry (including CrewAI)
|
|
os.environ['OTEL_SDK_DISABLED'] = 'true'
|
|
```
|
|
|
|
### Isolation from your own OpenTelemetry setup
|
|
|
|
CrewAI's telemetry runs on its own private `TracerProvider` and never registers
|
|
itself as the global one. This keeps the two directions separate:
|
|
|
|
- Spans from other instrumented libraries in your process — web frameworks,
|
|
database clients, HTTP clients — are never sent to CrewAI.
|
|
- CrewAI's telemetry spans are never sent to your observability backend, so they
|
|
will not appear in Langfuse, Braintrust, Phoenix, or any other collector you
|
|
configure.
|
|
|
|
Observability integrations are unaffected: they instrument CrewAI through their
|
|
own tracer provider, which is independent of the one described here.
|
|
|
|
### Data Explanation:
|
|
| Defaulted | Data | Reason and Specifics |
|
|
|:----------|:------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------|
|
|
| Yes | CrewAI and Python Version | Tracks software versions. Example: CrewAI v1.2.3, Python 3.8.10. No personal data. |
|
|
| Yes | Crew Metadata | Includes: randomly generated key and ID, process type (e.g., 'sequential', 'parallel'), boolean flag for memory usage (true/false), a boolean flag for whether any inputs were passed to the run (true/false — never the input keys or values, which are only collected when `share_crew` is enabled), count of tasks, count of agents. All non-personal. |
|
|
| Yes | Agent Data | Includes: randomly generated key and ID, role name (should not include personal info), boolean settings (verbose, delegation enabled, code execution allowed), max iterations, max RPM, max retry limit, LLM info (see LLM Attributes), list of tool names (should not include personal info). No personal data. |
|
|
| Yes | Task Metadata | Includes: randomly generated key and ID, boolean execution settings (async_execution, human_input), associated agent's role and key, list of tool names. All non-personal. |
|
|
| Yes | Tool Usage Statistics | Includes: tool name (should not include personal info), number of usage attempts (integer), LLM attributes used. No personal data. |
|
|
| Yes | Test Execution Data | Includes: crew's randomly generated key and ID, number of iterations, model name used, quality score (float), execution time (in seconds). All non-personal. |
|
|
| Yes | Task Lifecycle Data | Includes: creation and execution start/end times, crew and task identifiers, and whether the task succeeded or failed. When a task fails, the **class name** of the exception is recorded (for example `TimeoutError`) so failures can be counted and diagnosed — never the error message, which can contain prompts, model output, file paths or credentials. Stored as spans with timestamps. No personal data. |
|
|
| Yes | LLM Attributes | Includes: name, model_name, model, top_k, temperature, and class name of the LLM. All technical, non-personal data. |
|
|
| Yes | Project Creation using crewAI CLI | Includes: that a new project was scaffolded by `crewai create`, which kind it was (`crew`, `json_crew` or `flow`), and the project ID minted for that new project and written into its own `pyproject.toml`. That is the new project's own ID, recorded separately from the `project_id` of the directory the command was run from — the two can differ. No project name, no file contents, no code. No personal data. |
|
|
| Yes | Crew Deployment attempt using crewAI CLI | Includes: The fact a deploy is being made and crew id, whether it's trying to pull logs, and whether the deploy was started from a CLI command or from the run TUI. No project or crew contents. No personal data. |
|
|
| Yes | Execution Environment | Includes: which AI coding assistant is running the process, if any (one of a fixed list such as `claude_code`, `codex`, `cursor`, or `unknown`), where the process runs (one of a fixed list such as `ci`, `container`, `serverless`, `interactive`), and the `project_id` from your `pyproject.toml` when one is configured. Detection reads only whether known environment variables are set, never their values. No personal data. |
|
|
| Yes | Flow Lifecycle Signals | Includes: that a flow started, whether it completed or failed, whether one of its methods failed, whether it paused for human input or feedback, whether the start was a resumed run, whether a conversation turn failed, how long the flow ran, and whether the flow is one CrewAI runs internally or one you wrote. The flow name is recorded, as it already is for flow creation and execution. When a flow or one of its methods fails, the **class name** of the exception is recorded (for example `TimeoutError`) so that failures can be diagnosed — never the error message, which can contain prompts, model output, file paths or credentials. Method names and flow state are never recorded. No personal data. |
|
|
| Yes | Trace Sharing Signal | Includes: that a batch of traces was successfully shared with CrewAI AMP, and whether it was shared anonymously (before you have an account) or linked to your account. Like every span, it also carries the Execution Environment attributes described above (`project_id` when configured, the coding assistant, and the runtime). This row describes sharing telemetry only — not the trace contents or access granted by shared trace links. Trace contents, inputs, and outputs are never recorded on this signal. Before sharing traces, review secrets, personal data, and AMP redaction and retention settings. |
|
|
| No | Agent's Expanded Data | Includes: goal description, backstory text, i18n prompt file identifier. Users should ensure no personal info is included in text fields. |
|
|
| No | Detailed Task Information | Includes: task description, expected output description, context references. Users should ensure no personal info is included in these fields. |
|
|
| No | Environment Information | Includes: platform, release, system, version, and CPU count. Example: 'Windows 10', 'x86_64'. No personal data. |
|
|
| No | Crew and Task Inputs and Outputs | Includes: input parameters and output results as non-identifiable data. Users should ensure no personal info is included. |
|
|
| No | Comprehensive Crew Execution Data | Includes: detailed logs of crew operations, all agents and tasks data, final output. All non-personal and technical in nature. |
|
|
|
|
<Note>
|
|
"No" in the "Defaulted" column indicates that this data is only collected when `share_crew` is set to `true`.
|
|
</Note>
|
|
|
|
### Opt-In Further Telemetry Sharing
|
|
|
|
Users can choose to share their complete telemetry data by enabling the `share_crew` attribute to `True` in their crew configurations.
|
|
Enabling `share_crew` results in the collection of detailed crew and task execution data, including `goal`, `backstory`, `context`, and `output` of tasks.
|
|
This enables a deeper insight into usage patterns.
|
|
|
|
<Warning>
|
|
If you enable `share_crew`, the collected data may include personal information if it has been incorporated into crew configurations, task descriptions, or outputs.
|
|
Users should carefully review their data and ensure compliance with GDPR and other applicable privacy regulations before enabling this feature.
|
|
</Warning>
|