5 Commits

Include renames

Author	SHA1	Message	Date
João MouraandClaude Opus 5.5	1b9bbccbed	feat(cli): crewai eval --models, and llm_overlay swaps models as well as roles (#7811) ... * feat(llm): llm_overlay maps models too, not only agent roles A key `model:<provider/model>` maps every LLM built from that model string inside the block, and `model:*` every LLM built from any model string. That reaches what a role cannot: a flow step's own `LLM(...).call()`. The key is read in `LLM.__new__` (and so `create_llm` and an agent's `llm="..."`), and on an agent's declared llm built outside the block; a role key still wins for that agent. The caller's settings follow the mapped model by the rule a declared llm's do. `MODEL_KEY_PREFIX` is public so another package can feature-detect the form. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(telemetry): allow-list cli_usage:eval_models attributes The span `crewai eval --models` records carries whether the caller was logged in, the models compared (provider/model names) and how many, in both telemetry classes. Nothing about the run, its output or the organization. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(cli): crewai eval --models compares models on the deployment `crewai eval --models "a,b"` (and `--deployment UUID` when AMP cannot tell the deployment from the project id) asks AMP for a models evaluation: the project id, the models from one comma-separated list (stripped, repeats dropped, provider/model required, 1-5) and the project's eval.jsonc. It needs the login, prints and opens the report link, says where the comparison is while it runs, then prints one row per model (goal, tasks, agents, tools, cost, time; the deployed one marked, the best per column starred) and the top three suggestions. Exit 0 when it finished, 1 when it failed or could not start. Everything from the wire is printed as Text. The comparison is counted as `cli_usage:eval_models` once AMP accepts it. Mode 1 now sends the project id with `create_evaluation` too, so every evaluation is filed under the project it is about. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(cli): crewai deploy create and push send the project id Every create (git or ZIP) and every push (redeploy by name or uuid, or a ZIP update) carries `[tool.crewai].project_id` when the project has one, so AMP knows which project a deployment runs and `crewai eval --models` can find it. The id is read, never minted: a deploy does not rewrite pyproject.toml. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(llm): a model key maps an LLM whose declared SDK is not installed Nothing is built on the declared model, so a missing SDK for it is no reason to fail the mapped build; its credentials cannot be matched to the new provider, so none are carried. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(cli): crewai eval --models leaves its criteria behind as eval.jsonc When the comparison's answer carries `eval_config` and the project has no eval.jsonc, it is written through `write_eval_config` and the same one line Mode 1 prints says so. An existing file is never overwritten. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(llm): a model key compares providers by provider, not by class `_same_route_provider` now follows `_same_provider`'s rule case for case: a LiteLLM-routed `LLM(...)` is on the same provider as a native route when its provider names that native class, so its key and endpoint follow a model key to another model of the same provider. Another provider still gets none. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * chore(llm): say why mark_mapped may skip an object, and one import form in its tests Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(cli): usage stats name a compared model only when it is a public one `cli_usage:eval_models` carried the `--models` strings as typed, and those can identify a customer: a fine-tune id, an Azure deployment name, a self-hosted model or host. A model is now sent by name only when it is an exact entry of crewAI's model catalog (`LLM_CONTEXT_WINDOW_SIZES`), else as `<provider>/other`, and a provider crewAI does not route as `other/other`. An `ft:` id is never sent. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(cli): a refused comparison's test no longer keeps a None return Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(llms): re-importing OpenAI's completion module restores its package attribute `test_openai_completion_module_is_imported` deletes the module from sys.modules and lets `LLM(...)` import it again, which binds the new module on `crewai.llms.providers.openai` as well. monkeypatch restored only the sys.modules entry, so every later test on that worker saw two module objects for one name: `patch("crewai.llms.providers.openai.completion.X")` patched the attribute's module while `from ... import X` read sys.modules'. On Python 3.10 that made tests/llms/azure/test_azure_responses.py fail whenever it shared a worker with test_openai.py and test_crew_loader.py, which the new tests' even split now did. The attribute is restored too. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(llm): model keys strip only a model's own prefix; a role builds from the declaration Two review findings on the model-for-model overlay: - `overlay_model_for_model` took the text after the first slash as another name for the model, so an aggregator's route (`openrouter/openai/gpt-4o`) matched a `model:openai/gpt-4o` or `model:gpt-4o` key meant for the native model. A prefix is now stripped only when it is a native provider's own and what remains has no slash; an instance on an aggregator is compared as `<provider>/<its model id>`. - With a role key and a model key both active, the model key mapped the declared llm first and the role's model was built like that mapped instance, losing the caller's key and endpoint. The overlay now remembers what the caller declared for every llm it built, and a role key is built from that declaration. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(llm): model keys read a provider's other names; a LiteLLM choice survives a missing SDK - The router's provider aliases are now one table, `crewai.llm.PROVIDER_ALIASES`, and `llm_overlay` compares model names through it: `google/x` and `gemini/x` are one model whether the llm is built in the block or was built before it (a prebuilt instance records `gemini`). An exact key still wins, and an aggregator's route is still compared whole. - When the declared model's SDK is missing, a caller's explicit `is_litellm=True` is kept for the mapped model instead of dropping to a native route. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(deploy): a git create sends the project id once, at the top level AMP reads project_id at the top level on every deploy route, so the git create no longer repeats it inside `deploy`. The redeploy docstring records why its optional body is safe on an AMP that predates project ids. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-30 10:53:42 -07:00
João MouraandClaude Fable 5.1	0a3b8912b3	feat(cli): crewai eval evaluates the last traced run through AMP (#7649) ... * feat(cli): crewai eval evaluates the last traced run through AMP `crewai eval` reads `.crewai/last_run.json` — the record crewAI writes when a traced run's spans reach Wharf — and asks AMP to evaluate that run: POST /crewai_plus/api/v1/tracing/evaluations with the execution id, sending the saved `crewai login` when there is one and nothing otherwise. AMP answers with an evaluation id and a URL; the command prints the URL, opens it, waits for the verdict and prints it (goal gate and the four grades), exit 1 only when the evaluation itself failed. `--run EXECUTION_ID` evaluates another run. With no traced run recorded it offers to turn tracing on for the project (`CREWAI_TRACING_ENABLED=true` in .env, set_key so nothing else in the file moves) and run the crew now with `crewai run`; without a terminal, or declined, it prints the three steps instead. A run that leaves no record behind is explained, never guessed at. AMP's refusals are printed in its own words: a run that needs an account (401 account_required), a refused credential (then `crewai login`), a run AMP does not hold (404), rate limiting (429 with Retry-After), and any other status with AMP's message. The two AMP calls live on the CLI's PlusAPI subclass, so no crewai-core release is needed. Who may evaluate what is AMP's decision, not the command's: an anonymous run once without an account, then it needs one; a run traced while logged in for that organization's members; a deployment execution for members who may see its traces. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(cli): crewai eval — guard the project, survive AMP blips, name the right subject, read the whole record Review findings, each reproduced before the fix: - The run-it-now offer wrote CREWAI_TRACING_ENABLED into ./.env before checking the directory is a crewAI project, then run_crew() died on a missing pyproject.toml with a traceback. Now: no pyproject.toml → one sentence, exit 1, nothing written. - httpx errors (AMP unreachable, a timeout) surfaced as tracebacks. Now the start says "Could not reach AMP to start the evaluation: …"; while waiting, an unreachable AMP or a 5xx is retried up to POLL_RETRIES consecutive times, then reported with the URL — the evaluation keeps running server-side either way. - The POST now carries a 120 s timeout (AMP reads the run's spans inside it), the poll 30 s. - A 200 whose body has no known status (a non-dict, no status, a status outside queued/running/done/failed) polled forever. Now it stops with the status it saw and the URL. - A 404 without a JSON message read "AMP holds no run <evaluation id>" while polling. _refused takes "run <id>" / "evaluation <id>" and says "AMP answered 404 for <subject>" — AMP's own message still wins. - The record's amp_base_url was ignored; the CLI now evaluates the run at the AMP it was traced to. --run keeps the configured AMP. - The post-run explanation names the third cause: a crewai older than the version that records the last run. Tests for each, plus the previously untested paths: Ctrl-C exits 130, a 2xx without an id, a refusal mid-poll, DMN opens no browser, --run skips the offer. 51 passed in lib/cli/tests (eval + plus_api). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(cli): crewai eval — the credential goes only to the configured AMP; an explicit yes before running the crew; a done answer needs a well-formed verdict Review findings (CodeRabbit, the PR gate): - The record's amp_base_url was handed to PlusAPI beside the saved login, so a modified .crewai/last_run.json could send the token to any origin. The client is now built from the configured AMP only (CREWAI_PLUS_URL, the saved settings, app.crewai.com); the project's .env is loaded first, as `crewai run` loads it, so the configured AMP is the one the run was traced to. A record naming another address gets a one-line note and no credential. - The offer to turn tracing on and run the crew defaults to no and says the .env change stays; Enter no longer spends a crew run. - A `done` answer whose verdict is missing or malformed (no gate, grades not an object, a grade not an int or null) is a protocol error, exit 1, instead of an INCONCLUSIVE line with exit 0 or an AttributeError. Tests for each: a foreign origin in the record with a saved token, the .env-loaded same-AMP case, seven malformed verdicts, the prompt's text and default. 59 passed (eval + plus_api); ruff and mypy clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(cli): crewai eval — Enter accepts the offer to run the crew (y/n, Y default; the prompt names both effects) João's call (2026-09-20): the confirm is y/n with Y as the default. The prompt still says tracing stays on in .env and that the crew runs now. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-21 18:34:04 +00:00
Vinicius Brasil	2b90117e88	Add repository agents to flow definitions (#6437) ... Inline agent and crew actions can now use repository-backed agents without duplicating role, goal, and backstory in each definition. Examples: * `agent.with.from_repository: support_specialist` * `crew.with.agents.researcher.from_repository: researcher` `PlusAPI.get_agent` now uses the shared synchronous request path so project loaders can fetch repository agents without nested event loops.	2026-07-02 14:28:01 -07:00
João Moura	53c2284484	Support ZIP deployment fallback and JSON crew project env runs (#6166) ... * Update crewAI CLI with various enhancements and fixes - Updated `create_json_crew.py` to require `crewai[tools]>=1.14.7`. - Enhanced `git.py` with improved repository initialization, including automatic initial commit creation and exclusion patterns for initial commits. - Modified `install_crew.py` to allow error handling during installation with an optional `raise_on_error` parameter. - Expanded `plus_api.py` to include methods for creating and updating crews from ZIP files. - Introduced a new `archive.py` for creating deployable ZIP archives of CrewAI projects, ensuring local artifacts are excluded. - Updated `run_crew.py` to manage JSON crew dependencies and run crews in the project's environment. - Enhanced deployment logic in `main.py` to handle ZIP uploads and improve user feedback during deployment processes. - Added tests for new functionalities and ensured existing tests reflect recent changes in behavior and requirements. * fix(cli): address deploy zip review feedback * fix(cli): sync missing lockfile before deploy * fix(cli): preserve remote deploy on git setup warnings * test(cli): use single deploy main import style * fix(cli): skip project install for json crew sync * fix(cli): load json runner from source checkout * fix(cli): skip json crew sync when locked * fix(cli): address deploy zip review feedback * fix(cli): pass env on zip redeploy * fix(cli): harden json run and zip fallback * fix(cli): validate before deploy lock install * fix(cli): respect poetry lock for json runs * fix(cli): align json zip wrapper detection * fix(deps): bump starlette audit floor * fix(cli): avoid auth retry for deploy exits * fix(cli): update json zip script entrypoints	2026-06-15 18:46:54 -03:00
Greyson LaLonde	93e786d263	refactor: extract CLI into standalone crewai-cli package	2026-05-06 20:46:46 +08:00