6 Commits

Author	SHA1	Message	Date
João Moura	b10c4ffcdc	feat: add project_id to link OSS usage to an enterprise account (#6791) ... * feat: add project_id to link OSS usage to an enterprise account Adds a stable per-project identifier so a project's OSS traces and runs can be attributed to an account after signup. There was no such identifier before: [tool.crewai] held only `type`, the deploy UUID was printed to the console but never persisted, Settings.org_uuid is global rather than per-project, and trace batches carried only crew_fingerprint/crew_name. The id lives in the project's pyproject.toml, so it is committed with the repository and stays stable across machines, teammates, CI, and containers - unlike a machine- or user-derived identifier, which is unstable in exactly the containerized production environments that matter most. crewai-core: - get_project_id(): read-only lookup of [tool.crewai].project_id. Safe for library code; never creates or modifies anything. - get_or_create_project_id(): mints a uuid4 and persists it, returning (id, created) so callers can tell the user. Best-effort - returns (None, False) for a missing, malformed, or read-only pyproject.toml rather than raising. - Insertion edits the raw TOML text instead of round-tripping through a writer, so comments, key order, and formatting elsewhere survive. The key is placed at the end of the [tool.crewai] table, before the next table header, so it cannot land in a neighbouring section. - LoginPayload and TraceExecutionContext gain optional project_id. Sent on two paths: - Traces: project_id is added to execution_context, which is sent on both the ephemeral and authenticated paths, so a project's traces remain attributable before and after the user creates an account. - Login: `crewai login` already sends the pseudonymous user_identifier on an authenticated request; adding project_id means one request carries account + user + project, which is the link itself. Minting is restricted to CLI commands the user explicitly invoked - `crewai create` for new projects and `crewai run` to backfill existing ones - and is announced when it happens. Library code only ever reads. Silently rewriting a user's pyproject.toml during Crew.kickoff() would be surprising. Privacy: project_id is a random uuid4 in a file the user commits. It is visible in a diff, contains nothing personal, and identifies a project rather than a person - so this needs none of the notice changes that attaching a user identifier to all telemetry would require. Tests: 18 new tests covering minting, stability, table placement, comment and formatting preservation, five pyproject layouts, the neighbouring-table regression, and graceful handling of missing/malformed/read-only files. Verified end-to-end that both create paths mint distinct ids, that the trace payload carries project_id on both the ephemeral and authenticated paths, and that the login payload carries user_identifier and project_id together. Follow-ups, deliberately not included: adding project_id to telemetry spans, and backend persistence of the (account, user_identifier, project_id) triple. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UNumDnNbiyw3pv1WakAe6t * refactor: drop the console announcement when minting project_id Minting now happens silently. With no message to print, the (id, created) tuple had no consumer, so simplify the API rather than keep the flag around for a hypothetical caller: - get_or_create_project_id() returns `str | None` instead of `tuple[str | None, bool]`. - Remove crewai_cli.utils.ensure_project_id, which existed only to print the message and discard the flag. The four call sites (crewai create crew, crewai create flow, crewai run, and tool-repository login) now call get_or_create_project_id directly. - Update tests for the simplified signature; still 18 tests covering minting, stability, table placement, formatting preservation, five pyproject layouts, and missing/malformed/read-only handling. Behaviour is otherwise unchanged: minting stays restricted to CLI commands the user invoked, library code still only reads via get_project_id, and a missing or read-only pyproject.toml still returns None rather than raising. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UNumDnNbiyw3pv1WakAe6t * fix: harden project_id minting against TOML corruption; address review Several reviewers found ways the raw-text edit could produce invalid TOML. Each is now fixed and covered by a test that fails without the fix. Duplicate project_id key (Cursor bugbot, Copilot x2): - get_project_id() reports a blank or non-string value as "absent", so a file containing `project_id = ""` took the insert path and gained a second project_id line - a duplicate key, and therefore invalid TOML that no tomli-based tool could read afterwards. - _insert_project_id is now _set_project_id: it replaces an existing assignment inside [tool.crewai] instead of appending unconditionally. Table header with a trailing comment (CodeRabbit major, Cursor bugbot): - `[tool.crewai] # config` is valid TOML but failed exact string equality, so the fallback appended a second [tool.crewai] header - a redefined table, also invalid TOML, and silent because get_project_id swallows the resulting decode error. - Added _is_table_header(), which tolerates a trailing comment and does not match similar names such as [tool.crewai-extra]. Writing into malformed TOML (Cursor bugbot, Copilot): - get_or_create_project_id relied on get_project_id, which cannot distinguish "no id" from "unparsable file", so it appended to files it could not parse. - The locked path now parses explicitly and bails on a decode error, and re-parses the updated content before writing, so this feature can never be the reason a project's pyproject.toml stops parsing. Concurrency and atomicity (CodeRabbit major): - Two CLI processes could both see no id, mint different uuids, and clobber each other, leaving a caller holding an id that is not on disk. Minting now takes the existing crewai_core cross-process lock, re-reads under it, and returns the id that persists. - Writes go through a temp file in the same directory plus os.replace, so an interruption cannot truncate pyproject.toml. File mode is copied across, and the temp file is removed on failure. - os.replace only needs a writable directory, which would have let an atomic write silently overwrite a file the user marked read-only; writability is now checked explicitly so that case still returns None. Line endings (CodeRabbit): - Path.read_text/write_text normalized CRLF to LF, so minting would rewrite a CRLF-committed file entirely. Read and write now use newline="" and the inserted line ending is derived from the existing content. Default create path skipped minting (Cursor bugbot): - `crewai create crew` defaults to create_json_crew; only the --classic and flow paths minted, so most new projects had no id until a later command. Wired into create_json_crew as well. Verified all three paths now mint distinct ids. Do not mint during login (CodeRabbit major): - ToolCommand.login ran get_or_create_project_id, which is outside the sanctioned minting commands and is invoked by `crewai tools create` from a freshly scaffolded directory before the project is persisted. It now uses the read-only get_project_id. Verified login leaves pyproject.toml untouched. Not applied: Copilot asked for a console message when an id is written, in create_crew and create_flow. Minting was made deliberately silent in the previous commit, so the (id, created) tuple and the announcement are both gone by design. Tests: 32 in test_project_id.py, up from 18. New cases cover blank and non-string existing ids, three commented-header forms, similar table names, malformed input, CRLF and LF preservation, concurrent minting convergence, file-mode preservation, and temp-file cleanup. Confirmed the header and duplicate-key tests fail when the fixes are reverted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UNumDnNbiyw3pv1WakAe6t * fix: never create [tool.crewai], treat whitespace ids as absent, harden test `crewai run` could rewrite unrelated projects (Cursor bugbot, high): - get_or_create_project_id ran before the cwd was established as a CrewAI project, and _set_project_id appended a [tool.crewai] table when none existed. Any directory with a pyproject.toml could therefore gain one - including on `crewai run --definition`, which may otherwise succeed. - _set_project_id no longer creates the table; it returns None when [tool.crewai] is absent, so a key is only ever added to a table the project already declares. The templates all ship the table, so no create path needs the old fallback. - The minting call in run_crew moved after the --definition early return, so an explicit-flow run does not touch the cwd at all. - Presence is checked, not truthiness: an empty [tool.crewai] is still a CrewAI marker, and get_crewai_project_config returns {} both for that and for an absent table. - Verified an unrelated project's pyproject.toml is byte-identical after a mint attempt. Whitespace-only project_id accepted as valid (CodeRabbit): - `project_id = " "` is truthy, so it was returned as an identity and would have propagated into login payloads and tracing context. It also meant the '" "' parameter of the replacement test asserted nothing. - Added _usable_project_id, which strips before deciding, used by both get_project_id and the locked mint path. Concurrency test could hang CI (CodeRabbit, major): - Neither the barrier nor the joins had timeouts, so a thread dying early or blocking on the lock would hang the job rather than fail it. The result count was also unchecked, so a dead thread still passed. - Added timeouts, an explicit liveness assertion, a result-count assertion, a lock around the shared result list, and corrected the docstring: this covers the read-modify-write race with threads, not the cross-process backend. Tests: 35, up from 32. New coverage for the absent-table refusal and three whitespace forms; the blank-id replacement case now asserts a real uuid replaced the blank value. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UNumDnNbiyw3pv1WakAe6t * chore(deps): force gitpython 3.1.57+ for GHSA-p538-c434-8v24 and GHSA-3f7w-8rr8-f37f Unrelated to project_id; bundled here only because it blocks this PR's vulnerability scan. Two advisories were published for gitpython 3.1.55 after main last passed the scan: - GHSA-p538-c434-8v24: arbitrary file truncation via `git rev-list --output` argument injection. Fixed in 3.1.56. - GHSA-3f7w-8rr8-f37f: unguarded git option forwarding in IndexFile.checkout() and TagReference. Fixed in 3.1.57. - Bump the override floor to gitpython>=3.1.57 and declare the same floor in crewai-tools, so consumers installing the published package are covered and not only this repo's lock. - 3.1.57 was published 2026-07-26, past gitpython's exclude-newer-package cutoff of 2026-07-24, so that cutoff moves to 2026-07-27. Without it the floor is unresolvable. pip-audit against the updated lock reports no known vulnerabilities. Verified gitpython 3.1.57 resolves and that crewai_tools and crewai_cli.git still import. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UNumDnNbiyw3pv1WakAe6t --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 15:25:06 -07:00
Vinicius Brasil	a149a30bc0	Fix JSON crew template rendering (#6359) ... JSON crews were not using existing CLI templates.	2026-06-26 13:48:48 -07:00
João Moura	f364a7d988	Fix JSON crew version pin (#6342) ... * Fix JSON crew version pin * Use bounded CrewAI dependency range	2026-06-26 05:19:14 -03:00
João Moura	ebbc0998ef	Implement DMN mode support in crew creation and execution (#6194) ... * Implement DMN mode support in crew creation and execution - Added `is_dmn_mode_enabled` utility to check for enterprise non-interactive mode based on the `CREWAI_DMN` environment variable. - Updated `create` function in `cli.py` to enforce required parameters when DMN mode is active, raising appropriate usage errors. - Enhanced `create_crew` and `create_json_crew` functions to skip provider prompts and handle folder existence checks in DMN mode. - Introduced non-interactive defaults for agent and task creation in DMN mode, ensuring seamless project setup without user input. - Modified `run_crew` to bypass TUI and handle runtime inputs directly when in DMN mode, improving execution flow for JSON-defined crews. - Added tests to validate DMN mode behavior, ensuring correct handling of required inputs and non-interactive defaults. * Implement DMN mode support in crew creation and execution - Introduced `is_dmn_mode_enabled()` utility to check for non-interactive mode based on the `CREWAI_DMN` environment variable. - Updated `create` function to enforce required parameters when DMN mode is active, raising appropriate usage errors. - Modified `create_crew` and `create_json_crew` functions to skip provider prompts and utilize non-interactive defaults in DMN mode. - Enhanced `run_crew` to bypass TUI and handle runtime inputs directly in DMN mode, ensuring smooth execution without user interaction. - Added tests to validate DMN mode behavior, including requirements for type and name, and ensuring proper handling of existing folders and missing inputs.	2026-06-16 19:48:31 -03:00
João Moura	bb477f8a91	JSON first crews (#6131) ... * feat(cli): introduce JSON crew project support and TUI enhancements - Added support for creating and running JSON-defined crew projects, allowing users to scaffold projects with a new `create_json_crew.py` file. - Implemented a full-screen Textual TUI for crew execution in `crew_run_tui.py`, enhancing user interaction with a two-column layout. - Updated `run_crew.py` to prioritize JSON crew projects and added daemon mode for running without TUI. - Introduced interactive pickers in `tui_picker.py` for improved CLI prompts. - Enhanced validation for JSON crew files in `validate.py` to ensure proper structure and agent definitions. - Updated `.gitignore` to exclude demo and crewai directories. * feat: update LLM model references to gpt-5.4-mini - Changed default LLM model from gpt-4o-mini to gpt-5.4-mini across various files, including CLI options, JSON crew configurations, and agent definitions. - Enhanced benchmark and human feedback functionalities to utilize the new model. - Improved user interface elements in the TUI for better interaction and feedback during execution. - Added support for new skills directory in JSON crew project creation. * feat(benchmark): add crew-level benchmarking functionality - Introduced a new `benchmark` command in the CLI for crew-level benchmarking, allowing users to specify agents, models, and timeout settings. - Implemented `CrewBenchmarkCase` to handle crew-level benchmark cases with inputs and criteria. - Enhanced the benchmark runner to support progress tracking and detailed reporting of results for multiple models. - Added tests for loading crew benchmark cases and validating their structure. - Updated existing benchmark functions to accommodate the new crew-level execution model. * feat(cli): enhance JSON crew project functionality and TUI improvements - Added optional agent-level guardrails and advanced options in JSON crew configurations to improve output validation and flexibility. - Updated the TUI to better handle plan step statuses, including visual indicators for task completion and failure. - Introduced methods for parsing and managing step observation events, ensuring accurate updates to task statuses during execution. - Enhanced validation for JSON crew projects, ensuring proper structure and error handling for agent and task definitions. - Added comprehensive tests for new features and validation logic, ensuring robustness in JSON crew project handling. * refactor(cli): streamline JSON crew project handling and improve validation - Refactored JSON crew project loading and validation logic to enhance clarity and maintainability. - Introduced utility functions for finding JSON crew files, improving code reuse across modules. - Removed deprecated benchmark functionality and associated tests to simplify the codebase. - Updated CLI commands to utilize the new JSON project structure, ensuring compatibility with recent changes. - Enhanced test coverage for JSON crew project features, ensuring robust validation and error handling. * feat(cli): enhance activity log navigation and focus management - Added functionality to focus on the activity log when navigating through log entries. - Implemented refresh logic for the log panel to ensure updates are displayed correctly during navigation. - Improved keyboard navigation for log entries, allowing users to expand and scroll through logs seamlessly. - Added tests to verify the correct behavior of log navigation and focus management in the TUI. * feat(cli): enhance JSON crew project interaction and input handling - Introduced a new function to enable prompt line editing for better user experience during input prompts. - Updated the JSON crew project wizards to show interpolation hints for dynamic values, improving user guidance. - Enhanced the handling of missing input placeholders by prompting users for required values during crew setup. - Refactored the crew run logic to ensure proper loading and preparation of JSON-defined crews, including runtime input management. - Added tests to verify the correct behavior of new input handling features and JSON crew project interactions. * feat(cli): improve crew project input prompts and event handling - Enhanced the `_prompt_text` function to allow for configurable spacing before prompts, improving user experience during input collection. - Updated the wizards for agent and task creation to utilize the new prompt configuration, ensuring a more compact and streamlined interaction. - Introduced new plan step lifecycle events (`PlanStepStartedEvent`, `PlanStepCompletedEvent`) to better track the execution status of plan steps. - Refactored the step executor to emit these events during the execution of tasks, improving observability and debugging capabilities. - Added tests to verify the correct behavior of new prompt handling and event emissions during crew project execution. * fix: refine json-first crew interactions * fix: prioritize common json crew tools * fix: make json crew more tools expandable * fix: show json crew tools by category * feat(memory): update default embedder to OpenAI text-embedding-3-large and enhance memory compatibility - Changed the default embedding model for Memory to OpenAI text-embedding-3-large, which uses 3072-dimensional vectors. - Added warnings regarding compatibility issues with existing local memory stores created with 1536-dimensional embeddings. - Updated documentation to reflect the new default embedder and its configuration options. - Enhanced the CLI and codebase to support the new embedding model across various components, ensuring a seamless transition for users. * fix: address PR review feedback for JSON-first crews Review blockers: - Forward trained_agents_file to JSON crews: crewai run -f now exports CREWAI_TRAINED_AGENTS_FILE for the in-process JSON crew path - Wizard agent picker: Esc/cancel now reprompts instead of silently assigning the first agent - JSON tool resolution hard-fails: unknown tool names, missing custom tool files, and invalid custom tool modules raise JSONProjectError with actionable messages instead of warn-and-continue - Embedding dimension mismatch: LanceDB and Qdrant Edge storages raise EmbeddingDimensionMismatchError with reset/pin guidance instead of silently zero-filling vectors or returning empty search results - Custom tool code execution documented in loader docstring and the scaffolded project README CI fixes: - ruff format across lib/ - All 133 PR-introduced mypy errors fixed (llm.py lazy-litellm and cli.py lazy command shims now use TYPE_CHECKING imports; textual is_mounted misuse fixed; pick_many overloads; misc annotations) Bot review comments: - Empty except blocks now have explanatory comments or debug logging - Removed unused _C_BG/_C_PANEL/_C_BORDER globals and redundant import re; tests use a single import style for create_json_crew Tests: trained-agents propagation, wizard cancel, tool resolution failures, and dimension mismatch guidance. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: address second round of PR review comments Cursor Bugbot: - Wizard agent slugs: strip to [a-z0-9_] and fall back to agent_<n> so symbol-only roles can't produce an empty agents/.jsonc filename - Wizard task names: dedupe against prior task names and fall back to task_<n> for symbol-only descriptions CodeRabbit: - Agent.message(): import Task explicitly at runtime instead of relying on the namespace injection done by crewai/__init__ - Async executor: move the native-tools-unsupported fallback from _ainvoke_loop_react (self-recursion) to _ainvoke_loop_native_tools, mirroring the sync implementation - StepExecutor downgrade: keep the in-step conversation and append the text-tooling instructions instead of rebuilding messages, so completed native tool calls are not re-executed - crewai-files: extension-based MIME lookup now runs before byte sniffing so csv/xml types are not degraded to text/plain - Memory storages: validate every record in a save() batch against a consistent embedding dimension (LanceDB previously checked only the first record); added mixed-batch tests - _print_post_tui_summary now typed against CrewRunApp - Docs: Azure OpenAI default embedder change called out in the memory migration warning and provider table Code quality bots: - Removed unused _C_YELLOW/_C_CYAN (crew_run_tui) and _GREEN (tui_picker) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(cli): accordion tool picker in JSON crew wizard The flat tool list had grown to ~90 rows. The picker now shows: - Common tools always visible at the top - Every other category as a single expandable row with tool and selection counts (e.g. "Search & Research (27 tools, 2 selected)") - Expanding a category collapses the previously expanded one - Selections persist across expand/collapse via new preselected support in pick_many; cursor follows the toggled category row tui_picker gains preselected + initial_cursor options on pick_many, and Esc in multi-select now confirms the current selection instead of discarding it (required so collapsing can't silently drop choices). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(cli): remove --daemon flag from crewai run The flag only affected JSON crew projects — classic and flow projects ignored it entirely, which made the behavior inconsistent. Removed the option, the daemon code path (_run_json_crew_daemon), and its helper (_load_json_crew_with_inputs). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: update run command tests after --daemon removal lib/crewai/tests/cli/test_run_crew.py still asserted the old run_crew(trained_agents_file=..., daemon=False) call signature. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cli): exit codes, mid-run quit, async statuses, hyphen placeholders Addresses the latest Bugbot review round: - Failed JSON crew runs now exit non-zero (SystemExit(1)) so scripts and CI don't treat failures as success, mirroring the classic path - Quitting the TUI mid-run now ends the process (os._exit(130)); kickoff runs in a thread worker that cannot be force-cancelled, so letting the CLI return would leave LLM/tool work burning tokens in the background - Sidebar task statuses are now async-safe: completion/failure events resolve the task's own row via identity instead of assuming the most recently started task, and starting a task no longer blanket-marks earlier active rows as done - The runtime-input prompt regex now accepts hyphenated placeholder names ({my-topic}), matching kickoff's interpolation pattern Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: validation safety, custom tool sandboxing, TUI log integrity, memory error surfacing - Deploy validation no longer executes project code: validation mode checks tool declarations structurally (well-formed entries, custom tool file exists) without importing or instantiating anything. custom:<name> resolution only happens on the actual run path. - custom:<name> is constrained to [A-Za-z_][A-Za-z0-9_]* and the resolved path must stay inside the project's tools/ directory, so custom:../foo or absolute-path names cannot execute code outside it. Tool paths resolve relative to the crew project root, not cwd. - TUI task logs are built from per-task state captured at task start (idx, description, agent, start time); an out-of-order completion takes its output from the event and no longer steals or resets the current task's streamed steps/output. - EmbeddingDimensionMismatchError now inherits ValueError instead of RuntimeError so background saves surface it through MemorySaveFailedEvent instead of silently dropping the save; the shutdown catch in _background_encode_batch is narrowed to the "cannot schedule new futures" case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cli): declared project type wins over crew.json presence A flow project that also contains a crew.json(c) file now runs and validates as the flow it declares in pyproject.toml instead of being hijacked by the JSON crew path. Both crewai run (_has_json_crew) and deploy validation (_is_json_crew) check tool.crewai.type; a missing or unreadable pyproject still means a bare JSON crew project. Also documents why StepObservationFailedEvent intentionally marks the plan step "done": the event signals an observer failure, not a step failure, and the executor continues past it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cli): type the declared_type locals so mypy stays clean Comparing an Any-typed .get() chain returns Any, which tripped no-any-return on the previous commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-06-14 04:19:48 -03:00
Greyson LaLonde	93e786d263	refactor: extract CLI into standalone crewai-cli package	2026-05-06 20:46:46 +08:00