feat(flow): tag flow origin and report resumed runs

Two gaps found while testing the pause/resume path end to end.

Resumed runs were invisible. There is no resume event: a restored run re-enters
through kickoff(), so it looked identical to a fresh start. flow:resumed is
derived from _is_execution_resuming at flow start, which makes
flow:paused - flow:resumed the abandonment rate.

Flow counts are dominated by CrewAI's own AgentExecutor, which is itself a Flow
and runs once per agent execution - it is the top flow in the warehouse by a
wide margin. Nothing distinguished it from a user's flows except guessing at the
name. Both Flow Execution and Flow Completed now carry origin: "internal" when
the flow class is defined under crewai.*, "user" otherwise. Tagging only the new
span would have left the existing daily count unsplittable.

Both span methods take origin with a default, so their signatures stay
backward compatible.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
This commit is contained in:
Joao Moura
2026-08-11 15:42:49 -07:00
parent fbe1debd96
commit bda1bd4026
8 changed files with 154 additions and 11 deletions

View File

@@ -61,7 +61,7 @@ os.environ['OTEL_SDK_DISABLED'] = 'true'
| نعم | سمات LLM | تشمل: الاسم، model_name، model، top_k، temperature، واسم فئة LLM. كلها بيانات تقنية غير شخصية. |
| نعم | محاولة نشر الطاقم باستخدام CLI الخاص بـ CrewAI | تشمل: حقيقة إجراء النشر ومعرّف الطاقم، وما إذا كان يحاول سحب السجلات، لا بيانات أخرى. |
| نعم | بيئة التنفيذ | تشمل: مساعد البرمجة بالذكاء الاصطناعي الذي يشغّل العملية إن وُجد (واحد من قائمة ثابتة مثل `claude_code` أو `codex` أو `cursor` أو `unknown`)، ومكان تشغيل العملية (واحد من قائمة ثابتة مثل `ci` أو `container` أو `serverless` أو `interactive`)، و`project_id` من ملف `pyproject.toml` عند ضبطه. يتحقق الاكتشاف فقط مما إذا كانت متغيرات البيئة المعروفة مضبوطة، ولا يقرأ قيمها أبدًا. لا بيانات شخصية. |
| نعم | إشارات دورة حياة التدفق | تشمل: ما إذا كان التدفق قد اكتمل أو فشل، وما إذا فشلت إحدى دواله، وما إذا توقّف مؤقتًا لطلب إدخال أو ملاحظات بشرية، وما إذا فشلت دورة محادثة، ومدة تشغيل التدفق. يُسجَّل اسم التدفق، كما هو الحال بالفعل عند إنشاء التدفق وتنفيذه. لا تُسجَّل أبدًا أسماء الدوال أو رسائل الأخطاء أو حالة التدفق. لا بيانات شخصية. |
| نعم | إشارات دورة حياة التدفق | تشمل: ما إذا كان التدفق قد اكتمل أو فشل، وما إذا فشلت إحدى دواله، وما إذا توقّف مؤقتًا لطلب إدخال أو ملاحظات بشرية، وما إذا استُؤنف بعد ذلك، وما إذا فشلت دورة محادثة، ومدة تشغيل التدفق، وما إذا كان التدفق من التدفقات التي يشغّلها CrewAI داخليًا أم من كتابتك. يُسجَّل اسم التدفق، كما هو الحال بالفعل عند إنشاء التدفق وتنفيذه. لا تُسجَّل أبدًا أسماء الدوال أو رسائل الأخطاء أو حالة التدفق. لا بيانات شخصية. |
| لا | بيانات الوكيل الموسّعة | تشمل: وصف الهدف، نص الخلفية، معرّف ملف موجهات i18n. يجب على المستخدمين التأكد من عدم تضمين معلومات شخصية في حقول النص. |
| لا | معلومات المهمة التفصيلية | تشمل: وصف المهمة، وصف المخرجات المتوقعة، مراجع السياق. يجب على المستخدمين التأكد من عدم تضمين معلومات شخصية في هذه الحقول. |
| لا | معلومات البيئة | تشمل: المنصة، الإصدار، النظام، الإصدار، وعدد وحدات المعالجة المركزية. مثال: 'Windows 10'، 'x86_64'. لا بيانات شخصية. |

View File

@@ -61,7 +61,7 @@ own tracer provider, which is independent of the one described here.
| Yes | LLM Attributes | Includes: name, model_name, model, top_k, temperature, and class name of the LLM. All technical, non-personal data. |
| Yes | Crew Deployment attempt using crewAI CLI | Includes: The fact a deploy is being made and crew id, and if it's trying to pull logs, no other data. |
| Yes | Execution Environment | Includes: which AI coding assistant is running the process, if any (one of a fixed list such as `claude_code`, `codex`, `cursor`, or `unknown`), where the process runs (one of a fixed list such as `ci`, `container`, `serverless`, `interactive`), and the `project_id` from your `pyproject.toml` when one is configured. Detection reads only whether known environment variables are set, never their values. No personal data. |
| Yes | Flow Lifecycle Signals | Includes: whether a flow completed or failed, whether one of its methods failed, whether it paused for human input or feedback, whether a conversation turn failed, and how long the flow ran. The flow name is recorded, as it already is for flow creation and execution. Method names, error messages and flow state are never recorded. No personal data. |
| Yes | Flow Lifecycle Signals | Includes: whether a flow completed or failed, whether one of its methods failed, whether it paused for human input or feedback, whether it was resumed afterwards, whether a conversation turn failed, how long the flow ran, and whether the flow is one CrewAI runs internally or one you wrote. The flow name is recorded, as it already is for flow creation and execution. Method names, error messages and flow state are never recorded. No personal data. |
| No | Agent's Expanded Data | Includes: goal description, backstory text, i18n prompt file identifier. Users should ensure no personal info is included in text fields. |
| No | Detailed Task Information | Includes: task description, expected output description, context references. Users should ensure no personal info is included in these fields. |
| No | Environment Information | Includes: platform, release, system, version, and CPU count. Example: 'Windows 10', 'x86_64'. No personal data. |

View File

@@ -59,7 +59,7 @@ provider로 등록하지 않습니다. 이를 통해 양방향이 분리됩니
| 예 | LLM 속성 | LLM의 이름, model_name, 모델, top_k, temperature 및 클래스명이 포함됩니다. 모두 기술적이고 비개인 정보입니다. |
| 예 | crewAI CLI를 통한 Crew 배포 시도 | 배포가 시도되고 있고 crew id가 포함되며, 로그를 가져오려고 하는 경우에만 해당. 다른 데이터 없음. |
| 예 | 실행 환경 | 포함: 프로세스를 실행 중인 AI 코딩 어시스턴트(있는 경우, `claude_code`, `codex`, `cursor`, `unknown` 등 고정 목록 중 하나), 프로세스가 실행되는 위치(`ci`, `container`, `serverless`, `interactive` 등 고정 목록 중 하나), 그리고 `pyproject.toml`에 설정된 경우 `project_id`. 감지는 알려진 환경 변수의 설정 여부만 확인하며 값은 읽지 않음. 개인 데이터 없음. |
| 예 | Flow 라이프사이클 신호 | 포함 항목: flow의 완료 또는 실패 여부, 해당 메서드의 실패 여부, 사람의 입력이나 피드백을 위해 일시 중지되었는지 여부, 대화 턴의 실패 여부, flow 실행 시간. flow 이름은 기록되며, 이는 flow 생성 및 실행에서 이미 그러합니다. 메서드 이름, 오류 메시지, flow 상태는 절대 기록하지 않습니다. 개인 정보 없음. |
| 예 | Flow 라이프사이클 신호 | 포함 항목: flow의 완료 또는 실패 여부, 해당 메서드의 실패 여부, 사람의 입력이나 피드백을 위해 일시 중지되었는지 여부, 이후 재개되었는지 여부, 대화 턴의 실패 여부, flow 실행 시간, 그리고 해당 flow가 CrewAI가 내부적으로 실행하는 것인지 사용자가 작성한 것인지 여부. flow 이름은 기록되며, 이는 flow 생성 및 실행에서 이미 그러합니다. 메서드 이름, 오류 메시지, flow 상태는 절대 기록하지 않습니다. 개인 정보 없음. |
| 아니오 | 에이전트 확장 데이터 | 목표 설명, 배경 이야기 텍스트, i18n 프롬프트 파일 식별자가 포함됩니다. 사용자들은 텍스트 필드에 개인 정보가 포함되지 않도록 해야 합니다. |
| 아니오 | 상세 작업 정보 | 작업 설명, 예상 출력 설명, 컨텍스트 참조가 포함됩니다. 사용자들은 이러한 필드에 개인 정보가 포함되지 않도록 해야 합니다. |
| 아니오 | 환경 정보 | 플랫폼, 릴리즈, 시스템, 버전, CPU 개수가 포함됩니다. 예: 'Windows 10', 'x86_64'. 개인 정보 없음. |

View File

@@ -61,7 +61,7 @@ por meio do próprio tracer provider, que é independente do descrito aqui.
| Sim | Atributos do LLM | Inclui: nome, model_name, model, top_k, temperatura e nome da classe do LLM. Todos técnicos, sem dados pessoais. |
| Sim | Tentativa de Deploy do Crew pelo CLI do crewAI | Inclui: O fato de um deploy estar sendo realizado e o crew id, e se está tentando buscar logs, sem mais dados. |
| Sim | Ambiente de Execução | Inclui: qual assistente de código com IA está executando o processo, se houver (um de uma lista fixa como `claude_code`, `codex`, `cursor` ou `unknown`), onde o processo é executado (um de uma lista fixa como `ci`, `container`, `serverless`, `interactive`) e o `project_id` do seu `pyproject.toml` quando houver um configurado. A detecção lê apenas se variáveis de ambiente conhecidas estão definidas, nunca seus valores. Sem dados pessoais. |
| Sim | Sinais de Ciclo de Vida do Flow | Inclui: se um flow foi concluído ou falhou, se um de seus métodos falhou, se ele pausou para entrada ou feedback humano, se um turno de conversa falhou e por quanto tempo o flow executou. O nome do flow é registrado, como já ocorre na criação e execução do flow. Nomes de métodos, mensagens de erro e estado do flow nunca são registrados. Sem dados pessoais. |
| Sim | Sinais de Ciclo de Vida do Flow | Inclui: se um flow foi concluído ou falhou, se um de seus métodos falhou, se ele pausou para entrada ou feedback humano, se foi retomado em seguida, se um turno de conversa falhou, por quanto tempo o flow executou e se o flow é um que o CrewAI executa internamente ou um que você escreveu. O nome do flow é registrado, como já ocorre na criação e execução do flow. Nomes de métodos, mensagens de erro e estado do flow nunca são registrados. Sem dados pessoais. |
| Não | Dados Expandidos do Agente | Inclui: descrição do objetivo, texto da história, identificador de arquivo i18n prompt. Usuários devem garantir que não haja info pessoal nesses campos de texto. |
| Não | Informações Detalhadas da Tarefa | Inclui: descrição da tarefa, descrição do resultado esperado, referências de contexto. Usuários devem garantir que não haja info pessoal nessas áreas. |
| Não | Informações de Ambiente | Inclui: plataforma, release, sistema, versão e quantidade de CPUs. Exemplo: 'Windows 10', 'x86_64'. Sem dados pessoais. |

View File

@@ -308,6 +308,17 @@ class EventListener(BaseEventListener):
def on_flow_created(_: Any, event: FlowCreatedEvent) -> None:
self._telemetry.flow_creation_span(event.flow_name)
def _flow_origin(source: Any) -> str:
"""Separate CrewAI's own flows from the caller's.
The agent executor is itself a Flow and runs once per agent
execution, so it dominates flow counts and would otherwise be
indistinguishable from flows a user wrote.
"""
return (
"internal" if type(source).__module__.startswith("crewai.") else "user"
)
def _report_flow_duration(source: Any, flow_name: str, outcome: str) -> None:
"""Emit the elapsed time for a flow that reached a terminal state.
@@ -320,15 +331,23 @@ class EventListener(BaseEventListener):
return
source._telemetry_started_at = None
self._telemetry.flow_completed_span(
flow_name, (time.monotonic() - started_at) * 1000, outcome
flow_name,
(time.monotonic() - started_at) * 1000,
outcome,
_flow_origin(source),
)
@crewai_event_bus.on(FlowStartedEvent)
def on_flow_started(source: Any, event: FlowStartedEvent) -> None:
self._telemetry.flow_execution_span(
event.flow_name, list(source._methods.keys())
event.flow_name, list(source._methods.keys()), _flow_origin(source)
)
source._telemetry_started_at = time.monotonic()
if getattr(source, "_is_execution_resuming", False):
# No resume event exists, so a run restored from a pause is only
# visible here. Without it, paused flows can be counted but
# abandoned ones cannot be told apart from resumed ones.
self._telemetry.feature_usage_span("flow:resumed")
if not getattr(source, "suppress_flow_events", False):
self.formatter.handle_flow_created(event.flow_name, str(source.flow_id))
self.formatter.handle_flow_started(event.flow_name, str(source.flow_id))

View File

@@ -991,12 +991,18 @@ class Telemetry:
self._safe_telemetry_operation(_operation)
def flow_execution_span(self, flow_name: str, node_names: list[str]) -> None:
def flow_execution_span(
self, flow_name: str, node_names: list[str], origin: str = "user"
) -> None:
"""Records the execution of a flow.
Args:
flow_name: Name of the flow being executed.
node_names: List of nodes being executed in the flow.
origin: ``"internal"`` for flows CrewAI itself runs (the agent
executor), ``"user"`` for flows the caller authored. Without it
the agent executor, which runs once per agent execution, is
indistinguishable from a user's own flows in the daily counts.
"""
def _operation() -> None:
@@ -1009,12 +1015,13 @@ class Telemetry:
)
self._add_attribute(span, "flow_name", flow_name)
self._add_attribute(span, "node_names", json.dumps(node_names))
self._add_attribute(span, "origin", origin)
close_span(span)
self._safe_telemetry_operation(_operation)
def flow_completed_span(
self, flow_name: str, duration_ms: float, outcome: str
self, flow_name: str, duration_ms: float, outcome: str, origin: str = "user"
) -> None:
"""Records how long a flow ran and how it ended.
@@ -1033,6 +1040,8 @@ class Telemetry:
duration_ms: Wall-clock milliseconds from flow start, measured on a
monotonic clock.
outcome: Either ``"completed"`` or ``"failed"``.
origin: ``"internal"`` for flows CrewAI itself runs (the agent
executor), ``"user"`` for flows the caller authored.
"""
def _operation() -> None:
@@ -1042,6 +1051,7 @@ class Telemetry:
self._add_attribute(span, "flow_name", flow_name)
self._add_attribute(span, "duration_ms", duration_ms)
self._add_attribute(span, "outcome", outcome)
self._add_attribute(span, "origin", origin)
close_span(span)
self._safe_telemetry_operation(_operation)

View File

@@ -42,6 +42,24 @@ def _reregister_listener() -> None:
listener_module.event_listener.setup_listeners(crewai_event_bus)
@pytest.fixture
def flow_spans(monkeypatch: pytest.MonkeyPatch) -> list[tuple[str, str]]:
"""Record (flow_name, origin) for every Flow Execution span."""
from crewai.events import event_listener as listener_module
_reregister_listener()
recorded: list[tuple[str, str]] = []
monkeypatch.setattr(
listener_module.event_listener._telemetry,
"flow_execution_span",
lambda flow_name, node_names, origin="user": recorded.append(
(flow_name, origin)
),
)
return recorded
@pytest.fixture
def durations(monkeypatch: pytest.MonkeyPatch) -> list[tuple[str, float, str]]:
"""Record every (flow_name, duration_ms, outcome) the listener reports."""
@@ -53,7 +71,7 @@ def durations(monkeypatch: pytest.MonkeyPatch) -> list[tuple[str, float, str]]:
monkeypatch.setattr(
listener_module.event_listener._telemetry,
"flow_completed_span",
lambda flow_name, duration_ms, outcome: recorded.append(
lambda flow_name, duration_ms, outcome, origin="user": recorded.append(
(flow_name, duration_ms, outcome)
),
)
@@ -127,7 +145,7 @@ def test_a_failed_flow_is_still_counted_as_an_execution(
monkeypatch.setattr(
listener_module.event_listener._telemetry,
"flow_execution_span",
lambda flow_name, node_names: started.append(flow_name),
lambda flow_name, node_names, origin="user": started.append(flow_name),
)
class BoomFlow(Flow):
@@ -303,3 +321,97 @@ def test_duration_is_reported_once_per_run(
)
assert len(durations) == 1
def test_user_authored_flows_are_tagged_as_user(flow_spans) -> None:
class MyOwnFlow(Flow):
@start()
def go(self) -> str:
return "ok"
MyOwnFlow().kickoff()
assert ("MyOwnFlow", "user") in flow_spans
def test_crewais_own_agent_executor_is_tagged_internal(flow_spans) -> None:
"""The agent executor is a Flow and runs once per agent execution.
Without an origin tag it is indistinguishable from a user's flows in the
daily counts, and it dominates them.
"""
from crewai import Agent, Crew, Task
from crewai.llms.base_llm import BaseLLM
class StubLLM(BaseLLM):
def __init__(self) -> None:
super().__init__(model="stub-model")
def call(self, messages, **kwargs) -> str:
return "Final Answer: done"
def supports_function_calling(self) -> bool:
return False
def supports_stop_words(self) -> bool:
return False
def get_context_window_size(self) -> int:
return 8192
agent = Agent(role="R", goal="G", backstory="B", llm=StubLLM())
task = Task(description="Do it", expected_output="A result", agent=agent)
Crew(agents=[agent], tasks=[task]).kickoff()
origins = {name: origin for name, origin in flow_spans}
assert origins.get("AgentExecutor") == "internal"
def test_resumed_flow_is_reported(tmp_path, features: list[str]) -> None:
"""A restored run is only visible here - there is no resume event.
Without it, a paused flow that was abandoned cannot be told apart from one
the user came back to.
"""
from pydantic import BaseModel
from crewai.events.event_bus import crewai_event_bus
from crewai.events.types.flow_events import FlowPausedEvent
from crewai.flow.persistence.sqlite import SQLiteFlowPersistence
persistence = SQLiteFlowPersistence(str(tmp_path / "flows.db"))
class State(BaseModel):
id: str = "resume-test-1"
class AsyncProvider:
def request_feedback(self, context: PendingFeedbackContext, flow: Flow) -> str:
raise HumanFeedbackPending(context=context)
class ReviewFlow(Flow[State]):
@start()
@human_feedback(message="Review:", provider=AsyncProvider())
def draft(self) -> str:
return "draft"
@listen(draft)
def finish(self, result) -> str:
return f"final: {result.feedback}"
paused: dict[str, str] = {}
@crewai_event_bus.on(FlowPausedEvent)
def _capture(source, event) -> None:
paused["flow_id"] = event.flow_id
with contextlib.suppress(BaseException):
ReviewFlow(persistence=persistence).kickoff()
assert "flow:paused" in features
assert "flow:resumed" not in features
flow = ReviewFlow.from_pending(paused["flow_id"], persistence)
flow.resume("looks good")
assert "flow:resumed" in features
assert "flow:completed" in features

View File

@@ -505,7 +505,9 @@ def test_flow_emits_start_event(reset_event_listener_singleton):
flow.kickoff()
assert event_received.wait(timeout=5), "Timeout waiting for flow started event"
mock_telemetry.flow_execution_span.assert_called_once_with("TestFlow", ["begin"])
mock_telemetry.flow_execution_span.assert_called_once_with(
"TestFlow", ["begin"], "user"
)
assert len(received_events) == 1
assert received_events[0].flow_name == "TestFlow"
assert received_events[0].type == "flow_started"