* feat(llm): llm_overlay maps models too, not only agent roles
A key `model:<provider/model>` maps every LLM built from that model string
inside the block, and `model:*` every LLM built from any model string. That
reaches what a role cannot: a flow step's own `LLM(...).call()`. The key is
read in `LLM.__new__` (and so `create_llm` and an agent's `llm="..."`), and
on an agent's declared llm built outside the block; a role key still wins for
that agent. The caller's settings follow the mapped model by the rule a
declared llm's do. `MODEL_KEY_PREFIX` is public so another package can
feature-detect the form.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(telemetry): allow-list cli_usage:eval_models attributes
The span `crewai eval --models` records carries whether the caller was
logged in, the models compared (provider/model names) and how many, in both
telemetry classes. Nothing about the run, its output or the organization.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(cli): crewai eval --models compares models on the deployment
`crewai eval --models "a,b"` (and `--deployment UUID` when AMP cannot tell
the deployment from the project id) asks AMP for a models evaluation: the
project id, the models from one comma-separated list (stripped, repeats
dropped, provider/model required, 1-5) and the project's eval.jsonc. It
needs the login, prints and opens the report link, says where the
comparison is while it runs, then prints one row per model (goal, tasks,
agents, tools, cost, time; the deployed one marked, the best per column
starred) and the top three suggestions. Exit 0 when it finished, 1 when it
failed or could not start. Everything from the wire is printed as Text.
The comparison is counted as `cli_usage:eval_models` once AMP accepts it.
Mode 1 now sends the project id with `create_evaluation` too, so every
evaluation is filed under the project it is about.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(cli): crewai deploy create and push send the project id
Every create (git or ZIP) and every push (redeploy by name or uuid, or a
ZIP update) carries `[tool.crewai].project_id` when the project has one, so
AMP knows which project a deployment runs and `crewai eval --models` can find
it. The id is read, never minted: a deploy does not rewrite pyproject.toml.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(llm): a model key maps an LLM whose declared SDK is not installed
Nothing is built on the declared model, so a missing SDK for it is no
reason to fail the mapped build; its credentials cannot be matched to the
new provider, so none are carried.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(cli): crewai eval --models leaves its criteria behind as eval.jsonc
When the comparison's answer carries `eval_config` and the project has no
eval.jsonc, it is written through `write_eval_config` and the same one line
Mode 1 prints says so. An existing file is never overwritten.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(llm): a model key compares providers by provider, not by class
`_same_route_provider` now follows `_same_provider`'s rule case for case: a
LiteLLM-routed `LLM(...)` is on the same provider as a native route when its
provider names that native class, so its key and endpoint follow a model
key to another model of the same provider. Another provider still gets none.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* chore(llm): say why mark_mapped may skip an object, and one import form in its tests
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(cli): usage stats name a compared model only when it is a public one
`cli_usage:eval_models` carried the `--models` strings as typed, and those
can identify a customer: a fine-tune id, an Azure deployment name, a
self-hosted model or host. A model is now sent by name only when it is an
exact entry of crewAI's model catalog (`LLM_CONTEXT_WINDOW_SIZES`), else as
`<provider>/other`, and a provider crewAI does not route as `other/other`.
An `ft:` id is never sent.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(cli): a refused comparison's test no longer keeps a None return
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(llms): re-importing OpenAI's completion module restores its package attribute
`test_openai_completion_module_is_imported` deletes the module from
sys.modules and lets `LLM(...)` import it again, which binds the new module
on `crewai.llms.providers.openai` as well. monkeypatch restored only the
sys.modules entry, so every later test on that worker saw two module
objects for one name: `patch("crewai.llms.providers.openai.completion.X")`
patched the attribute's module while `from ... import X` read sys.modules'.
On Python 3.10 that made tests/llms/azure/test_azure_responses.py fail
whenever it shared a worker with test_openai.py and test_crew_loader.py,
which the new tests' even split now did. The attribute is restored too.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(llm): model keys strip only a model's own prefix; a role builds from the declaration
Two review findings on the model-for-model overlay:
- `overlay_model_for_model` took the text after the first slash as another
name for the model, so an aggregator's route (`openrouter/openai/gpt-4o`)
matched a `model:openai/gpt-4o` or `model:gpt-4o` key meant for the
native model. A prefix is now stripped only when it is a native provider's
own and what remains has no slash; an instance on an aggregator is compared
as `<provider>/<its model id>`.
- With a role key and a model key both active, the model key mapped the
declared llm first and the role's model was built like that mapped
instance, losing the caller's key and endpoint. The overlay now remembers
what the caller declared for every llm it built, and a role key is built
from that declaration.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(llm): model keys read a provider's other names; a LiteLLM choice survives a missing SDK
- The router's provider aliases are now one table, `crewai.llm.PROVIDER_ALIASES`,
and `llm_overlay` compares model names through it: `google/x` and
`gemini/x` are one model whether the llm is built in the block or was
built before it (a prebuilt instance records `gemini`). An exact key still
wins, and an aggregator's route is still compared whole.
- When the declared model's SDK is missing, a caller's explicit
`is_litellm=True` is kept for the mapped model instead of dropping to a
native route.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(deploy): a git create sends the project id once, at the top level
AMP reads project_id at the top level on every deploy route, so the git
create no longer repeats it inside `deploy`. The redeploy docstring records
why its optional body is safe on an AMP that predates project ids.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(flow): report flow outcome and human-in-the-loop signals
A flow reported only that it started. FlowFinishedEvent, FlowFailedEvent,
MethodExecutionFailedEvent, MethodExecutionPausedEvent and FlowPausedEvent all
reached the console formatter and stopped there, and FlowInputRequestedEvent,
FlowInputReceivedEvent and ConversationTurnFailedEvent had no listener at all -
so success rate, failure rate and every HITL pause were unmeasurable.
Adds flow:completed, flow:failed, flow:method_failed, flow:paused,
flow:hitl_paused, flow:input_requested, flow:input_received and
flow:conversation_turn_failed as feature-usage spans, which the existing
feature-usage aggregation already reads.
Deliberately does not hold the Flow Execution span open to measure duration:
flow_executions_daily_target counts those spans at start, so a run that never
finishes would disappear from the count entirely. Duration needs its own span.
Counts only - flow names, method names, error text and flow state are never
recorded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
* feat(flow): record how long a flow ran
Adds a Flow Completed span carrying flow_name, duration_ms and outcome,
emitted when a flow finishes or fails. Elapsed time comes from a monotonic
stamp taken at flow start and cleared on use.
Kept separate from the Flow Execution span rather than holding that one open:
it is emitted and closed at start and the daily aggregate counts it, so
holding it would drop every run that is killed or crashes from the execution
count. A killed run now simply has no Flow Completed row, and the count is
unaffected.
Elapsed time is an explicit duration_ms attribute rather than the span's own
duration, which the ingestion pipeline stores as a suffixed string
("0.0000184s") that downstream aggregation parses to zero.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
* feat(flow): tag flow origin and report resumed runs
Two gaps found while testing the pause/resume path end to end.
Resumed runs were invisible. There is no resume event: a restored run re-enters
through kickoff(), so it looked identical to a fresh start. flow:resumed is
derived from _is_execution_resuming at flow start, which makes
flow:paused - flow:resumed the abandonment rate.
Flow counts are dominated by CrewAI's own AgentExecutor, which is itself a Flow
and runs once per agent execution - it is the top flow in the warehouse by a
wide margin. Nothing distinguished it from a user's flows except guessing at the
name. Both Flow Execution and Flow Completed now carry origin: "internal" when
the flow class is defined under crewai.*, "user" otherwise. Tagging only the new
span would have left the existing daily count unsplittable.
Both span methods take origin with a default, so their signatures stay
backward compatible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
* fix(flow): scope outcome and resume signals to user flows
Two findings from review, both confirmed against the code.
Outcome features counted CrewAI's own flows. The agent executor, memory
encoding and memory recall are all Flows and all set suppress_flow_events;
they run far more often than anything a user wrote, so flow:completed,
flow:failed and flow:method_failed were mostly bookkeeping. Those three are now
emitted only for flows the caller wrote. Internal outcomes are still recorded
on the Flow Completed span, which carries origin.
flow:resumed counted checkpoint restores. _is_execution_resuming is set both by
from_pending (a human pause) and by a checkpoint restore that never paused for
anyone, so resumes could exceed pauses and the abandonment rate was unusable.
Keyed off _pending_feedback_context instead, which only from_pending sets.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
* fix(flow): declare internal flows instead of inferring them
Three findings from review, all confirmed against the code.
Gating on suppress_flow_events was wrong. That flag asks for console quiet and
is a public field, so a caller who set it on their own flow silently lost
flow:completed, flow:failed and flow:method_failed.
Deciding origin from the defining module was also wrong. Flow.from_declaration()
returns a Flow typed in crewai.flow.flow, so a caller's declarative flow was
reported as one of CrewAI's own - the inversion this split exists to prevent.
Both had the same root cause: the discriminator was inferred. Flow now declares
is_crewai_internal, set on the agent executor and the memory encoding/recall
flows, and one helper serves both origin and the outcome gate.
A failed conversational session was reported as completed. Its session closes
with FlowFinishedEvent whatever happened, so a failed turn produced
flow:conversation_turn_failed and flow:completed together. The turn failure is
now recorded on the flow and read back when the session finishes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
* refactor(flow): report flow lifecycle as spans, not feature usage
Flow start, completion, pause and method failure are lifecycle facts, and the
lifecycle is reported as spans everywhere else. Reporting them through
feature usage put them in a table that aggregates on the feature string alone -
it cannot carry origin, duration or outcome, so those signals could never be
split between a user's flows and the ones CrewAI runs for itself.
Adds Flow Paused and Flow Method Failed spans, and a resumed marker on Flow
Execution so a run restored from a pause is not counted as a second fresh
start. Removes the duplicate feature rows for completed, failed, method_failed,
paused and resumed - every one of those facts is now on a span, with more
attached to it than the feature row ever carried.
Feature usage keeps only genuine adoption signals: flow:hitl_paused,
flow:input_requested, flow:input_received and flow:conversation_turn_failed.
Also clears the conversational turn-failure flag on every terminal path. A turn
that failed without deferred finalization ends via FlowFailedEvent, and the flag
left set there marked the next run on that instance as failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
* test(flow): update the flow_execution_span caller for the resumed argument
Adding the resumed marker changed a signature that tests/utilities/test_events.py
asserts on exactly, and that assertion was not re-run before pushing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
* test(flow): make the checkpoint-restore guard actually guard
The test asserted that flow:resumed was absent from feature usage, but that
signal moved onto the Flow Execution span. The assertion could no longer fail,
so a regression that mis-tagged checkpoint restores as resumes would have gone
unnoticed.
Now asserts the resumed attribute, and waits for the handlers: the manual emit
dispatches asynchronously, so the previous shape also read its result before the
listener had run.
Confirmed it discriminates - keying resumed off _is_execution_resuming again
fails it with [('RestoredFlow', True)] == [('RestoredFlow', False)].
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
* fix(telemetry): record the resumed marker as a string
Verified end to end against the live collector and ClickHouse: the pipeline
encodes a boolean attribute as the presence of a vBool key, so false arrives as
the key simply being absent. That is invisible in the schema and easy to read
wrongly - crew_memory is extracted as "the attribute exists" and consequently
reports 1 for 99.8% of crews against a field that defaults to False.
A string leaves nothing to infer. Confirmed in the warehouse: the emitted span
reads resumed = "false".
Adds direct coverage for the attributes each flow span records, including both
resumed values, and resets the Telemetry singleton in the helper so more than
one span method can be exercised per session.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
* feat: bump versions to 1.15.15
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: update pyproject.toml to specify wheel targets
Added a new section to the pyproject.toml file to include only specific files in the wheel build, enhancing the packaging process. Updated tests to verify the inclusion of these targets.
* feat: add memory save event handling to activity log
Implemented event handlers for MemorySaveStartedEvent, MemorySaveCompletedEvent, and MemorySaveFailedEvent in the crew_run_tui module. This allows the application to log memory save operations, capturing their status and details in the activity log. Added corresponding tests to verify the correct logging behavior for successful and failed memory saves.
* feat: enhance memory save event handling in activity log
Added functionality to suppress nested memory save events and updated the handling of MemorySaveStartedEvent, MemorySaveCompletedEvent, and MemorySaveFailedEvent to improve logging accuracy. Introduced new tests to verify the correct behavior of memory save events, including scenarios for nested events and completion updates for timed-out entries.
* Fix memory save activity log handling
* Normalize alpha package versions
* Update scaffolded crew dependency
* feat: add button to copy setup instructions for CrewAI coding agents
Introduced a button in the documentation that allows users to easily copy setup instructions for CrewAI coding agents. The instructions include installation steps, environment setup, and best practices for using the CrewAI CLI. This enhancement aims to streamline the onboarding process for new users.
* Improve missing CrewAI install guidance
* fix: address pr review feedback
* fix: avoid mismatched memory save rows
* fix: wait for queued memory save events
* fix: avoid matching memory saves on missing ids
* chore: normalize prerelease version to 1.14.8a1
João Moura