diff --git a/docs/edge/ar/telemetry.mdx b/docs/edge/ar/telemetry.mdx index 56eb62e673..894fac8d93 100644 --- a/docs/edge/ar/telemetry.mdx +++ b/docs/edge/ar/telemetry.mdx @@ -61,7 +61,7 @@ os.environ['OTEL_SDK_DISABLED'] = 'true' | نعم | سمات LLM | تشمل: الاسم، model_name، model، top_k، temperature، واسم فئة LLM. كلها بيانات تقنية غير شخصية. | | نعم | محاولة نشر الطاقم باستخدام CLI الخاص بـ CrewAI | تشمل: حقيقة إجراء النشر ومعرّف الطاقم، وما إذا كان يحاول سحب السجلات، لا بيانات أخرى. | | نعم | بيئة التنفيذ | تشمل: مساعد البرمجة بالذكاء الاصطناعي الذي يشغّل العملية إن وُجد (واحد من قائمة ثابتة مثل `claude_code` أو `codex` أو `cursor` أو `unknown`)، ومكان تشغيل العملية (واحد من قائمة ثابتة مثل `ci` أو `container` أو `serverless` أو `interactive`)، و`project_id` من ملف `pyproject.toml` عند ضبطه. يتحقق الاكتشاف فقط مما إذا كانت متغيرات البيئة المعروفة مضبوطة، ولا يقرأ قيمها أبدًا. لا بيانات شخصية. | -| نعم | إشارات دورة حياة التدفق | تشمل: ما إذا كان التدفق قد اكتمل أو فشل، وما إذا فشلت إحدى دواله، وما إذا توقّف مؤقتًا لطلب إدخال أو ملاحظات بشرية، وما إذا فشلت دورة محادثة. أعداد فقط - لا تُسجَّل أبدًا أسماء التدفقات أو الدوال أو رسائل الأخطاء أو حالة التدفق. لا بيانات شخصية. | +| نعم | إشارات دورة حياة التدفق | تشمل: ما إذا كان التدفق قد اكتمل أو فشل، وما إذا فشلت إحدى دواله، وما إذا توقّف مؤقتًا لطلب إدخال أو ملاحظات بشرية، وما إذا فشلت دورة محادثة، ومدة تشغيل التدفق. يُسجَّل اسم التدفق، كما هو الحال بالفعل عند إنشاء التدفق وتنفيذه. لا تُسجَّل أبدًا أسماء الدوال أو رسائل الأخطاء أو حالة التدفق. لا بيانات شخصية. | | لا | بيانات الوكيل الموسّعة | تشمل: وصف الهدف، نص الخلفية، معرّف ملف موجهات i18n. يجب على المستخدمين التأكد من عدم تضمين معلومات شخصية في حقول النص. | | لا | معلومات المهمة التفصيلية | تشمل: وصف المهمة، وصف المخرجات المتوقعة، مراجع السياق. يجب على المستخدمين التأكد من عدم تضمين معلومات شخصية في هذه الحقول. | | لا | معلومات البيئة | تشمل: المنصة، الإصدار، النظام، الإصدار، وعدد وحدات المعالجة المركزية. مثال: 'Windows 10'، 'x86_64'. لا بيانات شخصية. | diff --git a/docs/edge/en/telemetry.mdx b/docs/edge/en/telemetry.mdx index 63d3361fce..be3ec9b9b6 100644 --- a/docs/edge/en/telemetry.mdx +++ b/docs/edge/en/telemetry.mdx @@ -61,7 +61,7 @@ own tracer provider, which is independent of the one described here. | Yes | LLM Attributes | Includes: name, model_name, model, top_k, temperature, and class name of the LLM. All technical, non-personal data. | | Yes | Crew Deployment attempt using crewAI CLI | Includes: The fact a deploy is being made and crew id, and if it's trying to pull logs, no other data. | | Yes | Execution Environment | Includes: which AI coding assistant is running the process, if any (one of a fixed list such as `claude_code`, `codex`, `cursor`, or `unknown`), where the process runs (one of a fixed list such as `ci`, `container`, `serverless`, `interactive`), and the `project_id` from your `pyproject.toml` when one is configured. Detection reads only whether known environment variables are set, never their values. No personal data. | -| Yes | Flow Lifecycle Signals | Includes: whether a flow completed or failed, whether one of its methods failed, whether it paused for human input or feedback, and whether a conversation turn failed. Counts only - flow names, method names, error messages and flow state are never recorded. No personal data. | +| Yes | Flow Lifecycle Signals | Includes: whether a flow completed or failed, whether one of its methods failed, whether it paused for human input or feedback, whether a conversation turn failed, and how long the flow ran. The flow name is recorded, as it already is for flow creation and execution. Method names, error messages and flow state are never recorded. No personal data. | | No | Agent's Expanded Data | Includes: goal description, backstory text, i18n prompt file identifier. Users should ensure no personal info is included in text fields. | | No | Detailed Task Information | Includes: task description, expected output description, context references. Users should ensure no personal info is included in these fields. | | No | Environment Information | Includes: platform, release, system, version, and CPU count. Example: 'Windows 10', 'x86_64'. No personal data. | diff --git a/docs/edge/ko/telemetry.mdx b/docs/edge/ko/telemetry.mdx index 49e0b50dce..d66a58fa48 100644 --- a/docs/edge/ko/telemetry.mdx +++ b/docs/edge/ko/telemetry.mdx @@ -59,7 +59,7 @@ provider로 등록하지 않습니다. 이를 통해 양방향이 분리됩니 | 예 | LLM 속성 | LLM의 이름, model_name, 모델, top_k, temperature 및 클래스명이 포함됩니다. 모두 기술적이고 비개인 정보입니다. | | 예 | crewAI CLI를 통한 Crew 배포 시도 | 배포가 시도되고 있고 crew id가 포함되며, 로그를 가져오려고 하는 경우에만 해당. 다른 데이터 없음. | | 예 | 실행 환경 | 포함: 프로세스를 실행 중인 AI 코딩 어시스턴트(있는 경우, `claude_code`, `codex`, `cursor`, `unknown` 등 고정 목록 중 하나), 프로세스가 실행되는 위치(`ci`, `container`, `serverless`, `interactive` 등 고정 목록 중 하나), 그리고 `pyproject.toml`에 설정된 경우 `project_id`. 감지는 알려진 환경 변수의 설정 여부만 확인하며 값은 읽지 않음. 개인 데이터 없음. | -| 예 | Flow 라이프사이클 신호 | 포함 항목: flow의 완료 또는 실패 여부, 해당 메서드의 실패 여부, 사람의 입력이나 피드백을 위해 일시 중지되었는지 여부, 대화 턴의 실패 여부. 횟수만 수집하며 flow 이름, 메서드 이름, 오류 메시지, flow 상태는 절대 기록하지 않습니다. 개인 정보 없음. | +| 예 | Flow 라이프사이클 신호 | 포함 항목: flow의 완료 또는 실패 여부, 해당 메서드의 실패 여부, 사람의 입력이나 피드백을 위해 일시 중지되었는지 여부, 대화 턴의 실패 여부, flow 실행 시간. flow 이름은 기록되며, 이는 flow 생성 및 실행에서 이미 그러합니다. 메서드 이름, 오류 메시지, flow 상태는 절대 기록하지 않습니다. 개인 정보 없음. | | 아니오 | 에이전트 확장 데이터 | 목표 설명, 배경 이야기 텍스트, i18n 프롬프트 파일 식별자가 포함됩니다. 사용자들은 텍스트 필드에 개인 정보가 포함되지 않도록 해야 합니다. | | 아니오 | 상세 작업 정보 | 작업 설명, 예상 출력 설명, 컨텍스트 참조가 포함됩니다. 사용자들은 이러한 필드에 개인 정보가 포함되지 않도록 해야 합니다. | | 아니오 | 환경 정보 | 플랫폼, 릴리즈, 시스템, 버전, CPU 개수가 포함됩니다. 예: 'Windows 10', 'x86_64'. 개인 정보 없음. | diff --git a/docs/edge/pt-BR/telemetry.mdx b/docs/edge/pt-BR/telemetry.mdx index e50c1b206d..2b3d5ca0bc 100644 --- a/docs/edge/pt-BR/telemetry.mdx +++ b/docs/edge/pt-BR/telemetry.mdx @@ -61,7 +61,7 @@ por meio do próprio tracer provider, que é independente do descrito aqui. | Sim | Atributos do LLM | Inclui: nome, model_name, model, top_k, temperatura e nome da classe do LLM. Todos técnicos, sem dados pessoais. | | Sim | Tentativa de Deploy do Crew pelo CLI do crewAI | Inclui: O fato de um deploy estar sendo realizado e o crew id, e se está tentando buscar logs, sem mais dados. | | Sim | Ambiente de Execução | Inclui: qual assistente de código com IA está executando o processo, se houver (um de uma lista fixa como `claude_code`, `codex`, `cursor` ou `unknown`), onde o processo é executado (um de uma lista fixa como `ci`, `container`, `serverless`, `interactive`) e o `project_id` do seu `pyproject.toml` quando houver um configurado. A detecção lê apenas se variáveis de ambiente conhecidas estão definidas, nunca seus valores. Sem dados pessoais. | -| Sim | Sinais de Ciclo de Vida do Flow | Inclui: se um flow foi concluído ou falhou, se um de seus métodos falhou, se ele pausou para entrada ou feedback humano e se um turno de conversa falhou. Apenas contagens - nomes de flows, nomes de métodos, mensagens de erro e estado do flow nunca são registrados. Sem dados pessoais. | +| Sim | Sinais de Ciclo de Vida do Flow | Inclui: se um flow foi concluído ou falhou, se um de seus métodos falhou, se ele pausou para entrada ou feedback humano, se um turno de conversa falhou e por quanto tempo o flow executou. O nome do flow é registrado, como já ocorre na criação e execução do flow. Nomes de métodos, mensagens de erro e estado do flow nunca são registrados. Sem dados pessoais. | | Não | Dados Expandidos do Agente | Inclui: descrição do objetivo, texto da história, identificador de arquivo i18n prompt. Usuários devem garantir que não haja info pessoal nesses campos de texto. | | Não | Informações Detalhadas da Tarefa | Inclui: descrição da tarefa, descrição do resultado esperado, referências de contexto. Usuários devem garantir que não haja info pessoal nessas áreas. | | Não | Informações de Ambiente | Inclui: plataforma, release, sistema, versão e quantidade de CPUs. Exemplo: 'Windows 10', 'x86_64'. Sem dados pessoais. | diff --git a/lib/crewai/src/crewai/events/event_listener.py b/lib/crewai/src/crewai/events/event_listener.py index 4e7fb91543..c5b011947c 100644 --- a/lib/crewai/src/crewai/events/event_listener.py +++ b/lib/crewai/src/crewai/events/event_listener.py @@ -1,6 +1,7 @@ from __future__ import annotations from io import StringIO +import time from typing import TYPE_CHECKING, Any from pydantic import Field, PrivateAttr @@ -307,11 +308,27 @@ class EventListener(BaseEventListener): def on_flow_created(_: Any, event: FlowCreatedEvent) -> None: self._telemetry.flow_creation_span(event.flow_name) + def _report_flow_duration(source: Any, flow_name: str, outcome: str) -> None: + """Emit the elapsed time for a flow that reached a terminal state. + + A flow can finish without this listener having seen it start - a + conversational turn re-emits completion for a restored run - so a + missing stamp means "no duration to report", not an error. + """ + started_at = getattr(source, "_telemetry_started_at", None) + if started_at is None: + return + source._telemetry_started_at = None + self._telemetry.flow_completed_span( + flow_name, (time.monotonic() - started_at) * 1000, outcome + ) + @crewai_event_bus.on(FlowStartedEvent) def on_flow_started(source: Any, event: FlowStartedEvent) -> None: self._telemetry.flow_execution_span( event.flow_name, list(source._methods.keys()) ) + source._telemetry_started_at = time.monotonic() if not getattr(source, "suppress_flow_events", False): self.formatter.handle_flow_created(event.flow_name, str(source.flow_id)) self.formatter.handle_flow_started(event.flow_name, str(source.flow_id)) @@ -319,6 +336,7 @@ class EventListener(BaseEventListener): @crewai_event_bus.on(FlowFinishedEvent) def on_flow_finished(source: Any, event: FlowFinishedEvent) -> None: self._telemetry.feature_usage_span("flow:completed") + _report_flow_duration(source, event.flow_name, "completed") if not getattr(source, "suppress_flow_events", False): self.formatter.handle_flow_status( @@ -329,6 +347,7 @@ class EventListener(BaseEventListener): @crewai_event_bus.on(FlowFailedEvent) def on_flow_failed(source: Any, event: FlowFailedEvent) -> None: self._telemetry.feature_usage_span("flow:failed") + _report_flow_duration(source, event.flow_name, "failed") if not getattr(source, "suppress_flow_events", False): self.formatter.handle_flow_status( diff --git a/lib/crewai/src/crewai/flow/runtime/__init__.py b/lib/crewai/src/crewai/flow/runtime/__init__.py index 46278f4fe1..23bba18fb5 100644 --- a/lib/crewai/src/crewai/flow/runtime/__init__.py +++ b/lib/crewai/src/crewai/flow/runtime/__init__.py @@ -719,6 +719,10 @@ class Flow(BaseModel, Generic[T], metaclass=FlowMeta): _method_call_counts: dict[FlowMethodName, int] = PrivateAttr(default_factory=dict) _is_execution_resuming: bool = PrivateAttr(default=False) _restored_from_checkpoint: bool = PrivateAttr(default=False) + # Monotonic stamp set by the telemetry listener at flow start, so the + # duration span emitted at the end does not need to hold a span open for + # the life of the run. + _telemetry_started_at: float | None = PrivateAttr(default=None) _event_futures: list[Future[None]] = PrivateAttr(default_factory=list) _pending_feedback_context: PendingFeedbackContext | None = PrivateAttr(default=None) _human_feedback_method_outputs: dict[str, Any] = PrivateAttr(default_factory=dict) diff --git a/lib/crewai/src/crewai/telemetry/telemetry.py b/lib/crewai/src/crewai/telemetry/telemetry.py index ea2052acbc..bcc27e55ae 100644 --- a/lib/crewai/src/crewai/telemetry/telemetry.py +++ b/lib/crewai/src/crewai/telemetry/telemetry.py @@ -1013,6 +1013,39 @@ class Telemetry: self._safe_telemetry_operation(_operation) + def flow_completed_span( + self, flow_name: str, duration_ms: float, outcome: str + ) -> None: + """Records how long a flow ran and how it ended. + + A separate span from ``Flow Execution`` rather than that span held open + to completion: ``Flow Execution`` is emitted and closed at start, and + the daily aggregate counts it, so holding it would drop every run that + is killed or crashes from the execution count entirely. + + The elapsed time is recorded as an explicit ``duration_ms`` attribute + rather than left to the span's own duration, which the ingestion + pipeline stores as a suffixed string ("0.0000184s") that downstream + aggregation cannot parse. + + Args: + flow_name: Name of the flow that finished. + duration_ms: Wall-clock milliseconds from flow start, measured on a + monotonic clock. + outcome: Either ``"completed"`` or ``"failed"``. + """ + + def _operation() -> None: + tracer = self.provider.get_tracer(TRACER_NAME) + span = tracer.start_span("Flow Completed") + self._add_attribute(span, "crewai_version", version("crewai")) + self._add_attribute(span, "flow_name", flow_name) + self._add_attribute(span, "duration_ms", duration_ms) + self._add_attribute(span, "outcome", outcome) + close_span(span) + + self._safe_telemetry_operation(_operation) + def env_context_span(self, tool: str) -> None: """Records the coding tool environment context.""" diff --git a/lib/crewai/tests/telemetry/test_flow_telemetry.py b/lib/crewai/tests/telemetry/test_flow_telemetry.py index 72c0e30bde..99b24bdaf9 100644 --- a/lib/crewai/tests/telemetry/test_flow_telemetry.py +++ b/lib/crewai/tests/telemetry/test_flow_telemetry.py @@ -13,6 +13,7 @@ the input and conversation-failure events had no listener at all. from __future__ import annotations import contextlib +import time import pytest @@ -41,6 +42,24 @@ def _reregister_listener() -> None: listener_module.event_listener.setup_listeners(crewai_event_bus) +@pytest.fixture +def durations(monkeypatch: pytest.MonkeyPatch) -> list[tuple[str, float, str]]: + """Record every (flow_name, duration_ms, outcome) the listener reports.""" + from crewai.events import event_listener as listener_module + + _reregister_listener() + + recorded: list[tuple[str, float, str]] = [] + monkeypatch.setattr( + listener_module.event_listener._telemetry, + "flow_completed_span", + lambda flow_name, duration_ms, outcome: recorded.append( + (flow_name, duration_ms, outcome) + ), + ) + return recorded + + @pytest.fixture def features(monkeypatch: pytest.MonkeyPatch) -> list[str]: """Record every feature the listener reports for a real flow run. @@ -199,3 +218,88 @@ def test_no_user_authored_strings_are_recorded(features: list[str]) -> None: assert "my_secret_method_name" not in feature assert "secret error detail" not in feature assert "SecretNamedFlow" not in feature + + +def test_completed_flow_reports_a_real_duration( + durations: list[tuple[str, float, str]], +) -> None: + """Elapsed time must be measured, not merely present.""" + + class SlowFlow(Flow): + @start() + def go(self) -> str: + time.sleep(0.05) + return "ok" + + SlowFlow().kickoff() + + assert len(durations) == 1 + flow_name, duration_ms, outcome = durations[0] + assert flow_name == "SlowFlow" + assert outcome == "completed" + assert duration_ms >= 50 + + +def test_failed_flow_reports_its_duration_and_outcome( + durations: list[tuple[str, float, str]], +) -> None: + class SlowBoomFlow(Flow): + @start() + def go(self) -> str: + time.sleep(0.05) + raise RuntimeError("boom") + + with pytest.raises(RuntimeError, match="boom"): + SlowBoomFlow().kickoff() + + assert len(durations) == 1 + flow_name, duration_ms, outcome = durations[0] + assert flow_name == "SlowBoomFlow" + assert outcome == "failed" + assert duration_ms >= 50 + + +def test_no_duration_is_reported_without_a_recorded_start( + durations: list[tuple[str, float, str]], +) -> None: + """A completion with no observed start reports nothing, and does not raise. + + A conversational turn can re-emit completion for a restored run, so the + stamp is genuinely absent rather than impossible. + """ + from crewai.events.event_bus import crewai_event_bus + from crewai.events.types.flow_events import FlowFinishedEvent + + class NeverStartedFlow(Flow): + @start() + def go(self) -> str: + return "ok" + + flow = NeverStartedFlow() + crewai_event_bus.emit( + flow, + FlowFinishedEvent(flow_name="NeverStartedFlow", result="ok", state={}), + ) + + assert durations == [] + + +def test_duration_is_reported_once_per_run( + durations: list[tuple[str, float, str]], +) -> None: + """The stamp is cleared on use, so a repeated completion cannot double-count.""" + from crewai.events.event_bus import crewai_event_bus + from crewai.events.types.flow_events import FlowFinishedEvent + + class OkFlow(Flow): + @start() + def go(self) -> str: + return "ok" + + flow = OkFlow() + flow.kickoff() + crewai_event_bus.emit( + flow, FlowFinishedEvent(flow_name="OkFlow", result="ok", state={}) + ) + + assert len(durations) == 1