From b10c4ffcdc8e63ba6fff1b7ea5b8f80653ab7d20 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Moura?= Date: Mon, 3 Aug 2026 19:25:06 -0300 Subject: [PATCH] feat: add project_id to link OSS usage to an enterprise account (#6791) * feat: add project_id to link OSS usage to an enterprise account Adds a stable per-project identifier so a project's OSS traces and runs can be attributed to an account after signup. There was no such identifier before: [tool.crewai] held only `type`, the deploy UUID was printed to the console but never persisted, Settings.org_uuid is global rather than per-project, and trace batches carried only crew_fingerprint/crew_name. The id lives in the project's pyproject.toml, so it is committed with the repository and stays stable across machines, teammates, CI, and containers - unlike a machine- or user-derived identifier, which is unstable in exactly the containerized production environments that matter most. crewai-core: - get_project_id(): read-only lookup of [tool.crewai].project_id. Safe for library code; never creates or modifies anything. - get_or_create_project_id(): mints a uuid4 and persists it, returning (id, created) so callers can tell the user. Best-effort - returns (None, False) for a missing, malformed, or read-only pyproject.toml rather than raising. - Insertion edits the raw TOML text instead of round-tripping through a writer, so comments, key order, and formatting elsewhere survive. The key is placed at the end of the [tool.crewai] table, before the next table header, so it cannot land in a neighbouring section. - LoginPayload and TraceExecutionContext gain optional project_id. Sent on two paths: - Traces: project_id is added to execution_context, which is sent on both the ephemeral and authenticated paths, so a project's traces remain attributable before and after the user creates an account. - Login: `crewai login` already sends the pseudonymous user_identifier on an authenticated request; adding project_id means one request carries account + user + project, which is the link itself. Minting is restricted to CLI commands the user explicitly invoked - `crewai create` for new projects and `crewai run` to backfill existing ones - and is announced when it happens. Library code only ever reads. Silently rewriting a user's pyproject.toml during Crew.kickoff() would be surprising. Privacy: project_id is a random uuid4 in a file the user commits. It is visible in a diff, contains nothing personal, and identifies a project rather than a person - so this needs none of the notice changes that attaching a user identifier to all telemetry would require. Tests: 18 new tests covering minting, stability, table placement, comment and formatting preservation, five pyproject layouts, the neighbouring-table regression, and graceful handling of missing/malformed/read-only files. Verified end-to-end that both create paths mint distinct ids, that the trace payload carries project_id on both the ephemeral and authenticated paths, and that the login payload carries user_identifier and project_id together. Follow-ups, deliberately not included: adding project_id to telemetry spans, and backend persistence of the (account, user_identifier, project_id) triple. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01UNumDnNbiyw3pv1WakAe6t * refactor: drop the console announcement when minting project_id Minting now happens silently. With no message to print, the (id, created) tuple had no consumer, so simplify the API rather than keep the flag around for a hypothetical caller: - get_or_create_project_id() returns `str | None` instead of `tuple[str | None, bool]`. - Remove crewai_cli.utils.ensure_project_id, which existed only to print the message and discard the flag. The four call sites (crewai create crew, crewai create flow, crewai run, and tool-repository login) now call get_or_create_project_id directly. - Update tests for the simplified signature; still 18 tests covering minting, stability, table placement, formatting preservation, five pyproject layouts, and missing/malformed/read-only handling. Behaviour is otherwise unchanged: minting stays restricted to CLI commands the user invoked, library code still only reads via get_project_id, and a missing or read-only pyproject.toml still returns None rather than raising. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01UNumDnNbiyw3pv1WakAe6t * fix: harden project_id minting against TOML corruption; address review Several reviewers found ways the raw-text edit could produce invalid TOML. Each is now fixed and covered by a test that fails without the fix. Duplicate project_id key (Cursor bugbot, Copilot x2): - get_project_id() reports a blank or non-string value as "absent", so a file containing `project_id = ""` took the insert path and gained a second project_id line - a duplicate key, and therefore invalid TOML that no tomli-based tool could read afterwards. - _insert_project_id is now _set_project_id: it replaces an existing assignment inside [tool.crewai] instead of appending unconditionally. Table header with a trailing comment (CodeRabbit major, Cursor bugbot): - `[tool.crewai] # config` is valid TOML but failed exact string equality, so the fallback appended a second [tool.crewai] header - a redefined table, also invalid TOML, and silent because get_project_id swallows the resulting decode error. - Added _is_table_header(), which tolerates a trailing comment and does not match similar names such as [tool.crewai-extra]. Writing into malformed TOML (Cursor bugbot, Copilot): - get_or_create_project_id relied on get_project_id, which cannot distinguish "no id" from "unparsable file", so it appended to files it could not parse. - The locked path now parses explicitly and bails on a decode error, and re-parses the updated content before writing, so this feature can never be the reason a project's pyproject.toml stops parsing. Concurrency and atomicity (CodeRabbit major): - Two CLI processes could both see no id, mint different uuids, and clobber each other, leaving a caller holding an id that is not on disk. Minting now takes the existing crewai_core cross-process lock, re-reads under it, and returns the id that persists. - Writes go through a temp file in the same directory plus os.replace, so an interruption cannot truncate pyproject.toml. File mode is copied across, and the temp file is removed on failure. - os.replace only needs a writable directory, which would have let an atomic write silently overwrite a file the user marked read-only; writability is now checked explicitly so that case still returns None. Line endings (CodeRabbit): - Path.read_text/write_text normalized CRLF to LF, so minting would rewrite a CRLF-committed file entirely. Read and write now use newline="" and the inserted line ending is derived from the existing content. Default create path skipped minting (Cursor bugbot): - `crewai create crew` defaults to create_json_crew; only the --classic and flow paths minted, so most new projects had no id until a later command. Wired into create_json_crew as well. Verified all three paths now mint distinct ids. Do not mint during login (CodeRabbit major): - ToolCommand.login ran get_or_create_project_id, which is outside the sanctioned minting commands and is invoked by `crewai tools create` from a freshly scaffolded directory before the project is persisted. It now uses the read-only get_project_id. Verified login leaves pyproject.toml untouched. Not applied: Copilot asked for a console message when an id is written, in create_crew and create_flow. Minting was made deliberately silent in the previous commit, so the (id, created) tuple and the announcement are both gone by design. Tests: 32 in test_project_id.py, up from 18. New cases cover blank and non-string existing ids, three commented-header forms, similar table names, malformed input, CRLF and LF preservation, concurrent minting convergence, file-mode preservation, and temp-file cleanup. Confirmed the header and duplicate-key tests fail when the fixes are reverted. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01UNumDnNbiyw3pv1WakAe6t * fix: never create [tool.crewai], treat whitespace ids as absent, harden test `crewai run` could rewrite unrelated projects (Cursor bugbot, high): - get_or_create_project_id ran before the cwd was established as a CrewAI project, and _set_project_id appended a [tool.crewai] table when none existed. Any directory with a pyproject.toml could therefore gain one - including on `crewai run --definition`, which may otherwise succeed. - _set_project_id no longer creates the table; it returns None when [tool.crewai] is absent, so a key is only ever added to a table the project already declares. The templates all ship the table, so no create path needs the old fallback. - The minting call in run_crew moved after the --definition early return, so an explicit-flow run does not touch the cwd at all. - Presence is checked, not truthiness: an empty [tool.crewai] is still a CrewAI marker, and get_crewai_project_config returns {} both for that and for an absent table. - Verified an unrelated project's pyproject.toml is byte-identical after a mint attempt. Whitespace-only project_id accepted as valid (CodeRabbit): - `project_id = " "` is truthy, so it was returned as an identity and would have propagated into login payloads and tracing context. It also meant the '" "' parameter of the replacement test asserted nothing. - Added _usable_project_id, which strips before deciding, used by both get_project_id and the locked mint path. Concurrency test could hang CI (CodeRabbit, major): - Neither the barrier nor the joins had timeouts, so a thread dying early or blocking on the lock would hang the job rather than fail it. The result count was also unchecked, so a dead thread still passed. - Added timeouts, an explicit liveness assertion, a result-count assertion, a lock around the shared result list, and corrected the docstring: this covers the read-modify-write race with threads, not the cross-process backend. Tests: 35, up from 32. New coverage for the absent-table refusal and three whitespace forms; the blank-id replacement case now asserts a real uuid replaced the blank value. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01UNumDnNbiyw3pv1WakAe6t * chore(deps): force gitpython 3.1.57+ for GHSA-p538-c434-8v24 and GHSA-3f7w-8rr8-f37f Unrelated to project_id; bundled here only because it blocks this PR's vulnerability scan. Two advisories were published for gitpython 3.1.55 after main last passed the scan: - GHSA-p538-c434-8v24: arbitrary file truncation via `git rev-list --output` argument injection. Fixed in 3.1.56. - GHSA-3f7w-8rr8-f37f: unguarded git option forwarding in IndexFile.checkout() and TagReference. Fixed in 3.1.57. - Bump the override floor to gitpython>=3.1.57 and declare the same floor in crewai-tools, so consumers installing the published package are covered and not only this repo's lock. - 3.1.57 was published 2026-07-26, past gitpython's exclude-newer-package cutoff of 2026-07-24, so that cutoff moves to 2026-07-27. Without it the floor is unresolvable. pip-audit against the updated lock reports no known vulnerabilities. Verified gitpython 3.1.57 resolves and that crewai_tools and crewai_cli.git still import. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01UNumDnNbiyw3pv1WakAe6t --------- Co-authored-by: Claude Opus 5 (1M context) --- lib/cli/src/crewai_cli/create_crew.py | 3 + lib/cli/src/crewai_cli/create_flow.py | 3 + lib/cli/src/crewai_cli/create_json_crew.py | 4 + lib/cli/src/crewai_cli/run_crew.py | 10 + lib/cli/src/crewai_cli/tools/main.py | 7 +- lib/cli/src/crewai_cli/utils.py | 4 + lib/crewai-core/src/crewai_core/plus_api.py | 18 +- lib/crewai-core/src/crewai_core/project.py | 284 +++++++++++++++ lib/crewai-tools/pyproject.toml | 4 +- .../listeners/tracing/trace_batch_manager.py | 5 + lib/crewai/tests/telemetry/test_project_id.py | 342 ++++++++++++++++++ pyproject.toml | 7 +- uv.lock | 12 +- 13 files changed, 691 insertions(+), 12 deletions(-) create mode 100644 lib/crewai/tests/telemetry/test_project_id.py diff --git a/lib/cli/src/crewai_cli/create_crew.py b/lib/cli/src/crewai_cli/create_crew.py index 549fc023e..334c398f2 100644 --- a/lib/cli/src/crewai_cli/create_crew.py +++ b/lib/cli/src/crewai_cli/create_crew.py @@ -14,6 +14,7 @@ from crewai_cli.provider import ( ) from crewai_cli.utils import ( copy_template, + get_or_create_project_id, is_dmn_mode_enabled, load_env_vars, write_env_file, @@ -320,6 +321,8 @@ def create_crew( copy_template(src_file, dst_file, name, class_name, folder_name) if not parent_folder: + # Minted at creation so the project has a stable identity from run one. + get_or_create_project_id(folder_path / "pyproject.toml") initialize_if_git_available(folder_path) click.secho(f"Crew {name} created successfully!", fg="green", bold=True) diff --git a/lib/cli/src/crewai_cli/create_flow.py b/lib/cli/src/crewai_cli/create_flow.py index 7921c2847..8345c95f4 100644 --- a/lib/cli/src/crewai_cli/create_flow.py +++ b/lib/cli/src/crewai_cli/create_flow.py @@ -5,6 +5,7 @@ import click from crewai_core.telemetry import Telemetry from crewai_cli.git import initialize_if_git_available +from crewai_cli.utils import get_or_create_project_id from crewai_cli.version import get_crewai_tools_dependency @@ -31,6 +32,8 @@ def create_flow(name: str, *, declarative: bool = False) -> None: else: _create_python_flow(name, class_name, folder_name, project_root) + # Minted at creation so the project has a stable identity from run one. + get_or_create_project_id(project_root / "pyproject.toml") initialize_if_git_available(project_root) click.secho(f"Flow {name} created successfully!", fg="green", bold=True) diff --git a/lib/cli/src/crewai_cli/create_json_crew.py b/lib/cli/src/crewai_cli/create_json_crew.py index 84426900e..d3aa74102 100644 --- a/lib/cli/src/crewai_cli/create_json_crew.py +++ b/lib/cli/src/crewai_cli/create_json_crew.py @@ -18,6 +18,7 @@ from crewai_cli.model_catalog import get_provider_models from crewai_cli.tui_picker import pick_many, pick_one from crewai_cli.utils import ( enable_prompt_line_editing, + get_or_create_project_id, is_dmn_mode_enabled, load_env_vars, render_template, @@ -968,6 +969,9 @@ def create_json_crew( for model in models: _setup_env(folder_path, model) + # Minted at creation so the project has a stable identity from run one. + # This is the default `crewai create crew` path, not just --classic. + get_or_create_project_id(folder_path / "pyproject.toml") initialize_if_git_available(folder_path) click.echo() diff --git a/lib/cli/src/crewai_cli/run_crew.py b/lib/cli/src/crewai_cli/run_crew.py index c4116e1b6..05dfe0bd1 100644 --- a/lib/cli/src/crewai_cli/run_crew.py +++ b/lib/cli/src/crewai_cli/run_crew.py @@ -20,6 +20,7 @@ from crewai_cli.input_prompt import ( ) from crewai_cli.utils import ( build_env_with_all_tool_credentials, + get_or_create_project_id, is_dmn_mode_enabled, ) from crewai_cli.version import get_crewai_tools_dependency, get_crewai_version @@ -627,6 +628,15 @@ def run_crew( return pyproject_data = read_toml() + + # Backfills projects created before project_id existed. Only here, in a + # command the user explicitly invoked - never from the SDK during kickoff. + # Placed after the --definition early return so an explicit-flow run does + # not touch the cwd; get_or_create_project_id itself refuses to act unless + # [tool.crewai] is already present, so an unrelated project is never + # rewritten. + get_or_create_project_id() + if json_crew_definition := configured_project_json_crew(pyproject_data): # Declarative (JSON) crews resolve inputs the same way flows do: --inputs # layers over the crew's declared defaults, missing {placeholder}s are diff --git a/lib/cli/src/crewai_cli/tools/main.py b/lib/cli/src/crewai_cli/tools/main.py index 9917f097a..7f7fcf51b 100644 --- a/lib/cli/src/crewai_cli/tools/main.py +++ b/lib/cli/src/crewai_cli/tools/main.py @@ -17,6 +17,7 @@ from crewai_cli.constants import DEFAULT_CREWAI_ENTERPRISE_URL from crewai_cli.utils import ( build_env_with_tool_repository_credentials, get_project_description, + get_project_id, get_project_name, get_project_version, read_toml, @@ -228,8 +229,12 @@ class ToolCommand(BaseCommand, PlusAPIMixin): def login(self) -> None: get_user_id = _require_get_user_id() + # Read-only: login is not one of the sanctioned minting commands, and + # `crewai tools create` calls it from inside a freshly scaffolded + # directory before the tool project is persisted. login_response = self.plus_api_client.login_to_tool_repository( - user_identifier=get_user_id() + user_identifier=get_user_id(), + project_id=get_project_id(), ) if login_response.status_code != 200: diff --git a/lib/cli/src/crewai_cli/utils.py b/lib/cli/src/crewai_cli/utils.py index e20bcfea1..71206c8a4 100644 --- a/lib/cli/src/crewai_cli/utils.py +++ b/lib/cli/src/crewai_cli/utils.py @@ -9,7 +9,9 @@ from typing import Any import click from crewai_core.project import ( + get_or_create_project_id as get_or_create_project_id, get_project_description as get_project_description, + get_project_id as get_project_id, get_project_name as get_project_name, get_project_version as get_project_version, parse_toml as parse_toml, @@ -30,7 +32,9 @@ __all__ = [ "copy_template", "enable_prompt_line_editing", "fetch_and_json_env_file", + "get_or_create_project_id", "get_project_description", + "get_project_id", "get_project_name", "get_project_version", "is_dmn_mode_enabled", diff --git a/lib/crewai-core/src/crewai_core/plus_api.py b/lib/crewai-core/src/crewai_core/plus_api.py index 4ed5e640d..c1acbad4e 100644 --- a/lib/crewai-core/src/crewai_core/plus_api.py +++ b/lib/crewai-core/src/crewai_core/plus_api.py @@ -69,7 +69,7 @@ class _WithUserIdentifier(TypedDict): class LoginPayload(_WithUserIdentifier): - pass + project_id: NotRequired[str] class TraceExecutionContext(TypedDict): @@ -78,6 +78,7 @@ class TraceExecutionContext(TypedDict): flow_name: str | None crewai_version: str privacy_level: str + project_id: NotRequired[str | None] class TraceExecutionMetadata(TypedDict): @@ -229,11 +230,24 @@ class PlusAPI: return client.request(method, url, files=files, **request_kwargs) def login_to_tool_repository( - self, user_identifier: str | None = None + self, user_identifier: str | None = None, project_id: str | None = None ) -> httpx.Response: + """Log in to the tool repository. + + This request is authenticated, so sending user_identifier and project_id + alongside it links the account to the local pseudonymous user id and to + the project the command was run from - letting prior anonymous usage of + that project be attributed after signup. + + Args: + user_identifier: Local pseudonymous user id. + project_id: ``[tool.crewai].project_id`` of the current project. + """ payload: LoginPayload = {} if user_identifier: payload["user_identifier"] = user_identifier + if project_id: + payload["project_id"] = project_id return self._make_request("POST", f"{self.TOOLS_RESOURCE}/login", json=payload) def get_tool(self, handle: str) -> httpx.Response: diff --git a/lib/crewai-core/src/crewai_core/project.py b/lib/crewai-core/src/crewai_core/project.py index 9c7e6a33e..80b4596f4 100644 --- a/lib/crewai-core/src/crewai_core/project.py +++ b/lib/crewai-core/src/crewai_core/project.py @@ -3,13 +3,19 @@ from __future__ import annotations from functools import reduce +import os from pathlib import Path, PureWindowsPath +import shutil import sys +import tempfile from typing import Any +import uuid from rich.console import Console import tomli +from crewai_core.lock_store import lock as store_lock + if sys.version_info >= (3, 11): import tomllib @@ -221,3 +227,281 @@ def get_project_description( return _get_project_attribute( pyproject_path, ["project", "description"], require=require ) + + +_PROJECT_ID_KEY = "project_id" + + +def get_project_id(pyproject_path: str | Path = "pyproject.toml") -> str | None: + """Return ``[tool.crewai].project_id`` if the project has one. + + Read-only and safe to call from library code: it never creates or modifies + anything. Use this everywhere except the CLI commands that are allowed to + mint an id (see :func:`get_or_create_project_id`). + + Args: + pyproject_path: Path to the project's ``pyproject.toml``. + + Returns: + The project id, or None when the file is missing, unreadable, or has + no id configured. + """ + try: + pyproject_data = read_toml(pyproject_path) + except (OSError, tomli.TOMLDecodeError): + return None + + return _usable_project_id(get_crewai_project_config(pyproject_data)) + + +def _has_crewai_table(pyproject_data: dict[str, Any]) -> bool: + """True if ``[tool.crewai]`` exists, even when empty. + + Distinguishes "declared but empty" from "absent", which + :func:`get_crewai_project_config` cannot: it returns ``{}`` for both. + """ + tool_config = pyproject_data.get("tool") + return isinstance(tool_config, dict) and isinstance(tool_config.get("crewai"), dict) + + +def _usable_project_id(crewai_config: dict[str, Any]) -> str | None: + """Return the configured id if it is usable as an identifier. + + Whitespace-only values are treated as absent: they are truthy in Python but + are not an identity, and would otherwise propagate into login payloads and + tracing context. + """ + project_id = crewai_config.get(_PROJECT_ID_KEY) + if not isinstance(project_id, str): + return None + stripped = project_id.strip() + return stripped or None + + +def get_or_create_project_id( + pyproject_path: str | Path = "pyproject.toml", +) -> str | None: + """Return the project's id, minting and persisting one if absent. + + Writes ``project_id`` into the ``[tool.crewai]`` table so it is committed + with the repository. That makes it stable across machines, teammates, CI, + and containers - unlike a machine- or user-derived identifier. + + Only CLI commands the user explicitly invoked should call this. Library + code must use :func:`get_project_id` instead; silently rewriting a user's + ``pyproject.toml`` during ``Crew.kickoff()`` would be surprising. + + Args: + pyproject_path: Path to the project's ``pyproject.toml``. + + Returns: + The project id, or None when ``pyproject.toml`` is missing, malformed, + or not writable. Best-effort - never raises. + """ + path = Path(pyproject_path) + if not path.is_file(): + return None + + # Cross-process lock: two CLI invocations could otherwise both see no id, + # mint different uuids, and clobber each other - leaving one caller holding + # an id that is not the one on disk. + try: + with store_lock(_project_id_lock_name(path)): + return _get_or_create_project_id_locked(path) + except Exception: + # Lock backend unavailable; a torn write is worse than no id. + return get_project_id(path) + + +def _project_id_lock_name(path: Path) -> str: + """Return a stable lock name for a project's ``pyproject.toml``.""" + return f"file:{os.path.realpath(path)}" + + +def _get_or_create_project_id_locked(path: Path) -> str | None: + """Read-modify-write the project id while holding the lock. + + Re-reads under the lock so a concurrent minter's id is returned rather than + overwritten. + """ + try: + content = _read_preserving_newlines(path) + except OSError: + return None + + # Parse here rather than relying on get_project_id, which reports malformed + # files and absent ids identically. Appending to a file we cannot parse + # would corrupt it further, so bail instead. + try: + pyproject_data = parse_toml(content) + except (tomli.TOMLDecodeError, ValueError): + return None + + crewai_config = get_crewai_project_config(pyproject_data) + existing = _usable_project_id(crewai_config) + if existing: + return existing + + # Only ever add a key to an existing [tool.crewai] table. Creating the table + # would rewrite the pyproject.toml of any directory that merely happens to + # have one, which `crewai run` could otherwise do before it has established + # that the cwd is a CrewAI project at all. + # + # Presence, not truthiness: an empty `[tool.crewai]` table is still a CrewAI + # marker, and get_crewai_project_config returns {} for both cases. + if not _has_crewai_table(pyproject_data): + return None + + project_id = str(uuid.uuid4()) + updated = _set_project_id(content, project_id) + if updated is None: + return None + + # Verify before writing: never leave a project with unparsable TOML because + # of this feature. + try: + parse_toml(updated) + except (tomli.TOMLDecodeError, ValueError): + return None + + # Checked explicitly: os.replace only needs a writable *directory*, so an + # atomic write would happily overwrite a file the user marked read-only. + if not os.access(path, os.W_OK): + return None + + try: + _write_atomically(path, updated) + except OSError: + # Read-only checkout, permissions, container FS - not worth failing over. + return None + + return project_id + + +def _read_preserving_newlines(path: Path) -> str: + """Read text without translating line endings. + + ``Path.read_text`` normalizes CRLF to LF, so a later write would silently + convert a CRLF-committed file to LF and show up as a whole-file diff. + """ + with path.open("r", encoding="utf-8", newline="") as handle: + return handle.read() + + +def _write_atomically(path: Path, content: str) -> None: + """Replace ``path`` with ``content`` via a temp file in the same directory. + + An interrupted or concurrent write must never leave a truncated + ``pyproject.toml`` behind. + """ + directory = path.parent + handle = tempfile.NamedTemporaryFile( + "w", + encoding="utf-8", + newline="", + dir=directory, + prefix=f".{path.name}.", + suffix=".tmp", + delete=False, + ) + tmp_path = Path(handle.name) + try: + with handle: + handle.write(content) + handle.flush() + os.fsync(handle.fileno()) + shutil.copymode(path, tmp_path) + os.replace(tmp_path, path) + except BaseException: + tmp_path.unlink(missing_ok=True) + raise + + +def _is_table_header(line: str, table: str) -> bool: + """True if ``line`` opens ``table``, tolerating a trailing inline comment. + + ``[tool.crewai] # config`` is valid TOML. Comparing the stripped line to + the header verbatim would miss it, and the caller would then append a second + ``[tool.crewai]`` header - a duplicate table definition, which is invalid + TOML. + """ + stripped = line.strip() + if not stripped.startswith("["): + return False + closing = stripped.find("]") + if closing == -1: + return False + if stripped[: closing + 1] != table: + return False + remainder = stripped[closing + 1 :].strip() + return remainder == "" or remainder.startswith("#") + + +def _is_any_table_header(line: str) -> bool: + """True if ``line`` opens any TOML table or array-of-tables.""" + return line.lstrip().startswith("[") + + +def _project_id_key_index(lines: list[str], start: int, end: int) -> int | None: + """Return the index of an existing ``project_id`` assignment in a range.""" + for index in range(start, end): + candidate = lines[index].strip() + if not candidate or candidate.startswith("#"): + continue + key, separator, _ = candidate.partition("=") + if separator and key.strip().strip("\"'") == _PROJECT_ID_KEY: + return index + return None + + +def _set_project_id(content: str, project_id: str) -> str | None: + """Set ``project_id`` in the ``[tool.crewai]`` table of TOML source text. + + Replaces an existing ``project_id`` assignment rather than adding a second + one: a blank or non-string value reads as "absent", and appending in that + case would produce a duplicate key and therefore invalid TOML. + + Edits the raw text rather than round-tripping through a TOML writer so + formatting, ordering, and comments in the rest of the file are preserved. + + Args: + content: Full contents of a ``pyproject.toml``. + project_id: The id to set. + + Returns: + Updated file contents, or None if the edit could not be made safely. + """ + lines = content.splitlines(keepends=True) + newline = "\r\n" if "\r\n" in content else "\n" + entry = f'{_PROJECT_ID_KEY} = "{project_id}"{newline}' + + for index, line in enumerate(lines): + if not _is_table_header(line, "[tool.crewai]"): + continue + + # Bound the table: everything up to the next table header. + table_end = len(lines) + for offset in range(index + 1, len(lines)): + if _is_any_table_header(lines[offset]): + table_end = offset + break + + existing = _project_id_key_index(lines, index + 1, table_end) + if existing is not None: + lines[existing] = entry + return "".join(lines) + + # Step back over trailing blank lines so the key stays in the table. + insert_at = table_end + while insert_at > index + 1 and not lines[insert_at - 1].strip(): + insert_at -= 1 + + if insert_at > 0 and not lines[insert_at - 1].endswith(("\n", "\r")): + lines[insert_at - 1] += newline + + lines.insert(insert_at, entry) + return "".join(lines) + + # No [tool.crewai] table. Never create one: that would let this feature + # rewrite the pyproject.toml of a directory that is not a CrewAI project. + return None diff --git a/lib/crewai-tools/pyproject.toml b/lib/crewai-tools/pyproject.toml index 71ca4bd23..319b1a561 100644 --- a/lib/crewai-tools/pyproject.toml +++ b/lib/crewai-tools/pyproject.toml @@ -107,7 +107,9 @@ stagehand = [ "stagehand>=0.4.1", ] github = [ - "gitpython>=3.1.55,<4", + # <3.1.57 has GHSA-p538-c434-8v24 (arbitrary file truncation) and + # GHSA-3f7w-8rr8-f37f (unguarded git option forwarding). + "gitpython>=3.1.57,<4", "PyGithub==1.59.1", ] rag = [ diff --git a/lib/crewai/src/crewai/events/listeners/tracing/trace_batch_manager.py b/lib/crewai/src/crewai/events/listeners/tracing/trace_batch_manager.py index 72bb2452e..6f3abd399 100644 --- a/lib/crewai/src/crewai/events/listeners/tracing/trace_batch_manager.py +++ b/lib/crewai/src/crewai/events/listeners/tracing/trace_batch_manager.py @@ -14,6 +14,7 @@ from crewai_core.plus_api import ( TraceExecutionMetadata, TraceFinalizePayload, ) +from crewai_core.project import get_project_id from crewai_core.settings import Settings from rich.console import Console from rich.panel import Panel @@ -145,6 +146,10 @@ class TraceBatchManager: "flow_name": execution_metadata.get("flow_name", None), "crewai_version": self.current_batch.version, "privacy_level": user_context.get("privacy_level", "standard"), + # Read-only: never mints an id. Sent on both the ephemeral and + # authenticated paths, so a project's traces stay attributable + # to it before and after the user creates an account. + "project_id": get_project_id(), } execution_metadata_payload: TraceExecutionMetadata = { "expected_duration_estimate": execution_metadata.get( diff --git a/lib/crewai/tests/telemetry/test_project_id.py b/lib/crewai/tests/telemetry/test_project_id.py new file mode 100644 index 000000000..e24aa725c --- /dev/null +++ b/lib/crewai/tests/telemetry/test_project_id.py @@ -0,0 +1,342 @@ +"""Tests for the project_id used to link OSS usage to an enterprise account.""" + +import uuid + +import pytest + +from crewai_core.project import ( + get_or_create_project_id, + get_project_id, + parse_toml, +) + + +CREW_PYPROJECT = """\ +[project] +name = "my_crew" +version = "0.1.0" +dependencies = ["crewai"] + +[tool.crewai] +type = "crew" + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" +""" + + +@pytest.fixture +def pyproject(tmp_path): + path = tmp_path / "pyproject.toml" + path.write_text(CREW_PYPROJECT) + return path + + +def test_returns_none_when_no_id_configured(pyproject): + assert get_project_id(pyproject) is None + + +def test_mints_and_persists_an_id(pyproject): + project_id = get_or_create_project_id(pyproject) + + assert uuid.UUID(project_id) + assert get_project_id(pyproject) == project_id + + +def test_id_is_stable_across_calls(pyproject): + first = get_or_create_project_id(pyproject) + second = get_or_create_project_id(pyproject) + + assert first == second, "must not mint a second id" + assert uuid.UUID(first) + + +def test_id_lands_in_the_tool_crewai_table(pyproject): + project_id = get_or_create_project_id(pyproject) + + data = parse_toml(pyproject.read_text()) + assert data["tool"]["crewai"]["project_id"] == project_id + assert data["tool"]["crewai"]["type"] == "crew", "existing keys must survive" + + +def test_other_tables_are_preserved(pyproject): + get_or_create_project_id(pyproject) + + data = parse_toml(pyproject.read_text()) + assert data["project"]["name"] == "my_crew" + assert data["project"]["dependencies"] == ["crewai"] + assert data["build-system"]["build-backend"] == "hatchling.build" + + +def test_comments_and_formatting_are_preserved(tmp_path): + """Raw-text editing rather than a TOML round-trip, so comments survive.""" + path = tmp_path / "pyproject.toml" + path.write_text( + '# top comment\n[project]\nname = "x" # inline comment\n\n[tool.crewai]\ntype = "flow"\n' + ) + + get_or_create_project_id(path) + + content = path.read_text() + assert "# top comment" in content + assert "# inline comment" in content + + +@pytest.mark.parametrize( + ("source", "label"), + [ + ('[project]\nname = "x"\n\n[tool.crewai]\ntype = "crew"\n', "table then EOF"), + ('[tool.crewai]\ntype = "crew"', "no trailing newline"), + ('[project]\nname = "x"\n[tool.crewai]\n[other]\na = 1\n', "empty table"), + ( + '[tool.crewai]\ntype = "crew"\n\n\n[build-system]\nrequires = []\n', + "blank lines before next table", + ), + ], +) +def test_produces_valid_toml_for_varied_layouts(tmp_path, source, label): + path = tmp_path / "pyproject.toml" + path.write_text(source) + + project_id = get_or_create_project_id(path) + + assert project_id is not None, label + data = parse_toml(path.read_text()) + assert data["tool"]["crewai"]["project_id"] == project_id, label + + +def test_id_does_not_leak_into_a_neighbouring_table(tmp_path): + """The key must never land under [build-system].""" + path = tmp_path / "pyproject.toml" + path.write_text( + '[tool.crewai]\ntype = "crew"\n\n[build-system]\nrequires = ["hatchling"]\n' + ) + + get_or_create_project_id(path) + + data = parse_toml(path.read_text()) + assert "project_id" in data["tool"]["crewai"] + assert "project_id" not in data["build-system"] + + +def test_absent_tool_crewai_table_is_never_created(tmp_path): + """Refuse to mint rather than rewrite a non-CrewAI project's pyproject.toml. + + `crewai run` in any directory that merely happens to have a pyproject.toml + must not gain a [tool.crewai] table as a side effect. + """ + path = tmp_path / "pyproject.toml" + original = '[project]\nname = "unrelated"\n' + path.write_text(original) + + assert get_or_create_project_id(path) is None + assert path.read_text() == original, "unrelated project was modified" + + +def test_missing_file_is_not_an_error(tmp_path): + assert get_or_create_project_id(tmp_path / "nope.toml") is None + + +def test_malformed_toml_is_not_an_error(tmp_path): + path = tmp_path / "pyproject.toml" + path.write_text("this is not [valid toml") + + assert get_project_id(path) is None + + +def test_read_only_file_is_not_an_error(pyproject): + """A read-only checkout must not break the command that called this.""" + pyproject.chmod(0o444) + try: + project_id = get_or_create_project_id(pyproject) + finally: + pyproject.chmod(0o644) + + assert project_id is None + + +def test_get_project_id_never_creates_anything(pyproject): + """Library code calls the read-only variant; it must not mutate the file.""" + before = pyproject.read_text() + + assert get_project_id(pyproject) is None + + assert pyproject.read_text() == before + + +@pytest.mark.parametrize("blank", ['""', "' '", '"\\t"']) +def test_blank_or_whitespace_id_is_treated_as_absent(tmp_path, blank): + """Whitespace is truthy in Python but is not an identity. + + Accepting it would propagate a useless value into login payloads and + tracing context. + """ + path = tmp_path / "pyproject.toml" + path.write_text(f'[tool.crewai]\ntype = "crew"\nproject_id = {blank}\n') + + assert get_project_id(path) is None + + +def test_malformed_toml_is_never_written_to(tmp_path): + """Appending to a file we cannot parse would corrupt it further.""" + path = tmp_path / "pyproject.toml" + original = 'this is not [valid toml\nproject_id = "x' + path.write_text(original) + + assert get_or_create_project_id(path) is None + assert path.read_text() == original, "malformed file must be left untouched" + + +@pytest.mark.parametrize("blank", ['""', "''", '" "', '"\\t\\t"']) +def test_blank_existing_id_is_replaced_not_duplicated(tmp_path, blank): + """A blank id reads as absent; appending would make a duplicate key.""" + path = tmp_path / "pyproject.toml" + path.write_text(f'[tool.crewai]\ntype = "crew"\nproject_id = {blank}\n') + + project_id = get_or_create_project_id(path) + + content = path.read_text() + assert content.count("project_id") == 1, f"duplicate key: {content!r}" + data = parse_toml(content) # would raise on a duplicate key + assert data["tool"]["crewai"]["project_id"] == project_id + assert data["tool"]["crewai"]["type"] == "crew" + assert uuid.UUID(project_id), "must mint a real id, not keep the blank one" + + +def test_non_string_existing_id_is_replaced(tmp_path): + path = tmp_path / "pyproject.toml" + path.write_text("[tool.crewai]\nproject_id = 42\n") + + project_id = get_or_create_project_id(path) + + data = parse_toml(path.read_text()) + assert data["tool"]["crewai"]["project_id"] == project_id + assert isinstance(project_id, str) + + +@pytest.mark.parametrize( + "header", + [ + "[tool.crewai] # crewai config", + "[tool.crewai]# no space", + "[tool.crewai]\t# tab then comment", + ], +) +def test_table_header_with_trailing_comment_is_found(tmp_path, header): + """A commented header is valid TOML; missing it appends a duplicate table.""" + path = tmp_path / "pyproject.toml" + path.write_text(f'{header}\ntype = "crew"\n') + + project_id = get_or_create_project_id(path) + + content = path.read_text() + assert content.count("[tool.crewai]") == 1, f"duplicate table: {content!r}" + data = parse_toml(content) # would raise on a redefined table + assert data["tool"]["crewai"]["project_id"] == project_id + assert data["tool"]["crewai"]["type"] == "crew" + + +def test_similar_table_names_are_not_matched(tmp_path): + """[tool.crewai-extra] must not be mistaken for [tool.crewai].""" + path = tmp_path / "pyproject.toml" + path.write_text('[tool.crewai-extra]\nfoo = 1\n\n[tool.crewai]\ntype = "crew"\n') + + project_id = get_or_create_project_id(path) + + data = parse_toml(path.read_text()) + assert data["tool"]["crewai"]["project_id"] == project_id + assert "project_id" not in data["tool"]["crewai-extra"] + + +def test_crlf_line_endings_are_preserved(tmp_path): + """read_text/write_text would silently rewrite the whole file as LF.""" + path = tmp_path / "pyproject.toml" + path.write_bytes(b'[project]\r\nname = "x"\r\n\r\n[tool.crewai]\r\ntype = "crew"\r\n') + + project_id = get_or_create_project_id(path) + + raw = path.read_bytes() + assert b"\r\n" in raw + assert raw.count(b"\n") == raw.count(b"\r\n"), "mixed line endings introduced" + assert parse_toml(raw.decode())["tool"]["crewai"]["project_id"] == project_id + + +def test_lf_file_stays_lf(tmp_path): + path = tmp_path / "pyproject.toml" + path.write_bytes(b'[tool.crewai]\ntype = "crew"\n') + + get_or_create_project_id(path) + + assert b"\r\n" not in path.read_bytes() + + +def test_concurrent_minting_converges_on_one_id(tmp_path): + """Concurrent minters must all return the id that ends up on disk. + + Uses threads in one process, so it covers the read-modify-write race rather + than the cross-process lock backend itself. + """ + import threading + + workers = 8 + path = tmp_path / "pyproject.toml" + path.write_text(CREW_PYPROJECT) + + returned: list[str | None] = [] + results_lock = threading.Lock() + # Timed out rather than unbounded: a thread dying before the barrier, or + # blocking on the lock, would otherwise hang CI instead of failing. + start = threading.Barrier(workers, timeout=30) + + def mint() -> None: + start.wait() + project_id = get_or_create_project_id(path) + with results_lock: + returned.append(project_id) + + threads = [threading.Thread(target=mint) for _ in range(workers)] + for thread in threads: + thread.start() + for thread in threads: + thread.join(timeout=30) + + assert not [t for t in threads if t.is_alive()], "thread did not finish in time" + assert len(returned) == workers, f"only {len(returned)}/{workers} threads returned" + + persisted = parse_toml(path.read_text())["tool"]["crewai"]["project_id"] + assert set(returned) == {persisted}, ( + f"callers disagreed with disk: returned={set(returned)} persisted={persisted}" + ) + + +def test_file_mode_is_preserved(tmp_path): + """The atomic replace must not widen permissions on pyproject.toml.""" + path = tmp_path / "pyproject.toml" + path.write_text(CREW_PYPROJECT) + path.chmod(0o600) + + get_or_create_project_id(path) + + assert path.stat().st_mode & 0o777 == 0o600 + + +def test_no_temp_files_left_behind(tmp_path): + path = tmp_path / "pyproject.toml" + path.write_text(CREW_PYPROJECT) + + get_or_create_project_id(path) + + assert [p.name for p in tmp_path.iterdir()] == ["pyproject.toml"] + + +def test_ids_are_unique_across_projects(tmp_path): + ids = set() + for name in ("a", "b", "c"): + path = tmp_path / name / "pyproject.toml" + path.parent.mkdir() + path.write_text(CREW_PYPROJECT) + project_id = get_or_create_project_id(path) + ids.add(project_id) + + assert len(ids) == 3 diff --git a/pyproject.toml b/pyproject.toml index 032f9ccc2..e6b5b6b0e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -172,7 +172,7 @@ info = "Commits must follow Conventional Commits 1.0.0." [tool.uv] exclude-newer = "3 days" # These security fixes are newer than the global supply-chain cutoff. -exclude-newer-package = { pypdf = "2026-06-24T00:00:00Z", msgpack = "2026-06-20T00:00:00Z", pydantic-settings = "2026-06-20T00:00:00Z", langsmith = "2026-06-20T00:00:00Z", gitpython = "2026-07-24T00:00:00Z" } +exclude-newer-package = { pypdf = "2026-06-24T00:00:00Z", msgpack = "2026-06-20T00:00:00Z", pydantic-settings = "2026-06-20T00:00:00Z", langsmith = "2026-06-20T00:00:00Z", gitpython = "2026-07-27T00:00:00Z" } # composio-core pins rich<14 but textual requires rich>=14. # onnxruntime 1.24+ dropped Python 3.10 wheels; cap it so qdrant[fastembed] resolves on 3.10. @@ -190,6 +190,9 @@ exclude-newer-package = { pypdf = "2026-06-24T00:00:00Z", msgpack = "2026-06-20T # gitpython <3.1.51 has GHSA-2f96-g7mh-g2hx, GHSA-v396-v7q4-x2qj, and GHSA-956x-8gvw-wg5v. # gitpython <=3.1.51 has GHSA-rwj8-pgh3-r573; fixed in 3.1.52. # gitpython 3.1.52 has GHSA-3rp5-jjmw-4wv2, GHSA-fjr4-x663-mwxc, GHSA-6p8h-3wgx-97gf, and GHSA-r9mr-m37c-5fr3; force 3.1.55+. +# gitpython <3.1.56 has GHSA-p538-c434-8v24 (arbitrary file truncation via `git rev-list --output` argument +# injection) and <3.1.57 has GHSA-3f7w-8rr8-f37f (unguarded git option forwarding in IndexFile.checkout and +# TagReference); force 3.1.57+. Its exclude-newer-package cutoff is bumped to 2026-07-27 to admit that release. # pyasn1 <0.6.4 has GHSA-8ppf-4f7h-5ppj and GHSA-hm4w-wwcw-mr6r; force 0.6.4+. # urllib3 <2.7.0 has GHSA-qccp-gfcp-xxvc (ProxyManager cross-origin redirect leaks Authorization/Cookie) and GHSA-mf9v-mfxr-j63j (streaming decompression-bomb bypass); force 2.7.0+. # langsmith <0.8.18 has GHSA-3644-q5cj-c5c7 (public prompt manifest deserialization, SSRF/secret disclosure) @@ -224,7 +227,7 @@ override-dependencies = [ "pypdf>=6.14.2,<7", "uv>=0.11.15,<1", "python-multipart>=0.0.27,<1", - "gitpython>=3.1.55,<4", + "gitpython>=3.1.57,<4", "pyasn1>=0.6.4", "langsmith>=0.8.18,<1", "authlib>=1.6.12", diff --git a/uv.lock b/uv.lock index 25124c3b4..f1160b8a4 100644 --- a/uv.lock +++ b/uv.lock @@ -19,7 +19,7 @@ exclude-newer-span = "P3D" [options.exclude-newer-package] msgpack = "2026-06-20T00:00:00Z" langsmith = "2026-06-20T00:00:00Z" -gitpython = "2026-07-24T00:00:00Z" +gitpython = "2026-07-27T00:00:00Z" pypdf = "2026-06-24T00:00:00Z" pydantic-settings = "2026-06-20T00:00:00Z" @@ -37,7 +37,7 @@ overrides = [ { name = "authlib", specifier = ">=1.6.12" }, { name = "cryptography", specifier = ">=46.0.7" }, { name = "docling-core", extras = ["chunking"], specifier = ">=2.74.1" }, - { name = "gitpython", specifier = ">=3.1.55,<4" }, + { name = "gitpython", specifier = ">=3.1.57,<4" }, { name = "langchain-core", specifier = ">=1.3.3,<2" }, { name = "langchain-text-splitters", specifier = ">=1.1.2,<2" }, { name = "langsmith", specifier = ">=0.8.18,<1" }, @@ -1758,7 +1758,7 @@ requires-dist = [ { name = "e2b-code-interpreter", marker = "extra == 'e2b'", specifier = "~=2.6.0" }, { name = "exa-py", marker = "extra == 'exa-py'", specifier = ">=1.8.7" }, { name = "firecrawl-py", marker = "extra == 'firecrawl-py'", specifier = ">=1.8.0" }, - { name = "gitpython", marker = "extra == 'github'", specifier = ">=3.1.55,<4" }, + { name = "gitpython", marker = "extra == 'github'", specifier = ">=3.1.57,<4" }, { name = "hyperbrowser", marker = "extra == 'hyperbrowser'", specifier = ">=0.18.0" }, { name = "langchain-apify", marker = "extra == 'apify'", specifier = ">=0.1.2,<1.0.0" }, { name = "linkup-sdk", marker = "extra == 'linkup-sdk'", specifier = ">=0.2.2" }, @@ -2771,14 +2771,14 @@ wheels = [ [[package]] name = "gitpython" -version = "3.1.55" +version = "3.1.57" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "gitdb" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/b2/ab/ba0d29f2fa2277ed6256b2ac09003494045355f3a10bf32f351761287870/gitpython-3.1.55.tar.gz", hash = "sha256:781e3b1624dad81b24e9524bf0297b69786a0706db2cbceec1e2b05c38e5152f", size = 225071, upload-time = "2026-07-23T02:52:43.246Z" } +sdist = { url = "https://files.pythonhosted.org/packages/ba/0d/132ed135c871b6bf91adf16a0e43797cd535b81d4973b5d09291c54fc5ee/gitpython-3.1.57.tar.gz", hash = "sha256:c493ec57c0ef6b19743798b6a5af859c71814b524e7e6f97baa2f8e658961488", size = 225898, upload-time = "2026-07-26T07:33:26.351Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/20/6a/d3b8208d2f8aac66abe8ccc1c23fa2c89464ec42cc71a601e95d05902428/gitpython-3.1.55-py3-none-any.whl", hash = "sha256:7c9ec1e69c158c081632ab35c41471e302c96db2ae42165036a5d2403378812e", size = 216590, upload-time = "2026-07-23T02:52:41.932Z" }, + { url = "https://files.pythonhosted.org/packages/41/6e/2139de986d9c7c3ac86f1f8be43858ce90bdfe2f7175e6c80c650ba15242/gitpython-3.1.57-py3-none-any.whl", hash = "sha256:4ccf7d73c10f5c9e76043fbb2675ac5a1b3ff5b41e648f56bcbed5f63792ecaf", size = 217151, upload-time = "2026-07-26T07:33:24.838Z" }, ] [[package]]