7 Commits

Include renames

Author	SHA1	Message	Date
João MouraandClaude Opus 5.5	00e2ff98e9	feat(cli): crewai eval prints a markdown brief when run by an agent (#7907) ... * feat(cli): crewai eval prints a markdown brief when run by an agent When stdout is not a terminal (or CREWAI_DMN is on), `crewai eval` and `crewai eval --models` print the poll response's `brief_markdown` after the verdict and the report link — written straight to stdout, never through Rich markup — or "Brief (markdown): <brief_url>" when only the link came. An older AMP sends neither and nothing extra is printed. In a terminal nothing changes. Exit codes unchanged. `crewai eval --help` and the `--models` help say so, and the AGENTS.md the crew, flow and declarative-flow templates scaffold gains an "Evaluating" section telling a coding agent to run `crewai eval` after a change and act on the brief. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs(cli): an agent re-runs before it re-evaluates; each command's exit rule; the brief may be a link Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(cli): keep the eval guidance out of tool projects; the brief is conditional in --help `crewai create tool` copies the shared AGENTS.md too, but a tool project has no `crewai run` and no traced run for `crewai eval`. The evaluation blocks are marked in the template and dropped for a tool (`evaluation=False`); crew, flow and JSON-crew projects keep them, markers stripped. --help now says the brief prints when AMP provides it, else its link, else nothing more. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-07 10:28:34 -07:00
João MouraandClaude Opus 5.5	1b9bbccbed	feat(cli): crewai eval --models, and llm_overlay swaps models as well as roles (#7811) ... * feat(llm): llm_overlay maps models too, not only agent roles A key `model:<provider/model>` maps every LLM built from that model string inside the block, and `model:*` every LLM built from any model string. That reaches what a role cannot: a flow step's own `LLM(...).call()`. The key is read in `LLM.__new__` (and so `create_llm` and an agent's `llm="..."`), and on an agent's declared llm built outside the block; a role key still wins for that agent. The caller's settings follow the mapped model by the rule a declared llm's do. `MODEL_KEY_PREFIX` is public so another package can feature-detect the form. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(telemetry): allow-list cli_usage:eval_models attributes The span `crewai eval --models` records carries whether the caller was logged in, the models compared (provider/model names) and how many, in both telemetry classes. Nothing about the run, its output or the organization. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(cli): crewai eval --models compares models on the deployment `crewai eval --models "a,b"` (and `--deployment UUID` when AMP cannot tell the deployment from the project id) asks AMP for a models evaluation: the project id, the models from one comma-separated list (stripped, repeats dropped, provider/model required, 1-5) and the project's eval.jsonc. It needs the login, prints and opens the report link, says where the comparison is while it runs, then prints one row per model (goal, tasks, agents, tools, cost, time; the deployed one marked, the best per column starred) and the top three suggestions. Exit 0 when it finished, 1 when it failed or could not start. Everything from the wire is printed as Text. The comparison is counted as `cli_usage:eval_models` once AMP accepts it. Mode 1 now sends the project id with `create_evaluation` too, so every evaluation is filed under the project it is about. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(cli): crewai deploy create and push send the project id Every create (git or ZIP) and every push (redeploy by name or uuid, or a ZIP update) carries `[tool.crewai].project_id` when the project has one, so AMP knows which project a deployment runs and `crewai eval --models` can find it. The id is read, never minted: a deploy does not rewrite pyproject.toml. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(llm): a model key maps an LLM whose declared SDK is not installed Nothing is built on the declared model, so a missing SDK for it is no reason to fail the mapped build; its credentials cannot be matched to the new provider, so none are carried. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(cli): crewai eval --models leaves its criteria behind as eval.jsonc When the comparison's answer carries `eval_config` and the project has no eval.jsonc, it is written through `write_eval_config` and the same one line Mode 1 prints says so. An existing file is never overwritten. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(llm): a model key compares providers by provider, not by class `_same_route_provider` now follows `_same_provider`'s rule case for case: a LiteLLM-routed `LLM(...)` is on the same provider as a native route when its provider names that native class, so its key and endpoint follow a model key to another model of the same provider. Another provider still gets none. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * chore(llm): say why mark_mapped may skip an object, and one import form in its tests Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(cli): usage stats name a compared model only when it is a public one `cli_usage:eval_models` carried the `--models` strings as typed, and those can identify a customer: a fine-tune id, an Azure deployment name, a self-hosted model or host. A model is now sent by name only when it is an exact entry of crewAI's model catalog (`LLM_CONTEXT_WINDOW_SIZES`), else as `<provider>/other`, and a provider crewAI does not route as `other/other`. An `ft:` id is never sent. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(cli): a refused comparison's test no longer keeps a None return Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(llms): re-importing OpenAI's completion module restores its package attribute `test_openai_completion_module_is_imported` deletes the module from sys.modules and lets `LLM(...)` import it again, which binds the new module on `crewai.llms.providers.openai` as well. monkeypatch restored only the sys.modules entry, so every later test on that worker saw two module objects for one name: `patch("crewai.llms.providers.openai.completion.X")` patched the attribute's module while `from ... import X` read sys.modules'. On Python 3.10 that made tests/llms/azure/test_azure_responses.py fail whenever it shared a worker with test_openai.py and test_crew_loader.py, which the new tests' even split now did. The attribute is restored too. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(llm): model keys strip only a model's own prefix; a role builds from the declaration Two review findings on the model-for-model overlay: - `overlay_model_for_model` took the text after the first slash as another name for the model, so an aggregator's route (`openrouter/openai/gpt-4o`) matched a `model:openai/gpt-4o` or `model:gpt-4o` key meant for the native model. A prefix is now stripped only when it is a native provider's own and what remains has no slash; an instance on an aggregator is compared as `<provider>/<its model id>`. - With a role key and a model key both active, the model key mapped the declared llm first and the role's model was built like that mapped instance, losing the caller's key and endpoint. The overlay now remembers what the caller declared for every llm it built, and a role key is built from that declaration. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(llm): model keys read a provider's other names; a LiteLLM choice survives a missing SDK - The router's provider aliases are now one table, `crewai.llm.PROVIDER_ALIASES`, and `llm_overlay` compares model names through it: `google/x` and `gemini/x` are one model whether the llm is built in the block or was built before it (a prebuilt instance records `gemini`). An exact key still wins, and an aggregator's route is still compared whole. - When the declared model's SDK is missing, a caller's explicit `is_litellm=True` is kept for the mapped model instead of dropping to a native route. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(deploy): a git create sends the project id once, at the top level AMP reads project_id at the top level on every deploy route, so the git create no longer repeats it inside `deploy`. The redeploy docstring records why its optional body is safe on an AMP that predates project ids. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-30 10:53:42 -07:00
João MouraandClaude Opus 5.5	19d2ebcc8c	fix(cli): crewai eval exits 1 unless the gate passed, and says to log in when nothing was traced unattended (#7806) ... * fix(cli): `crewai eval` exits 1 unless the gate passed, and says to log in when nothing was traced unattended The exit code is what a CI job reads, and it said only whether the evaluation finished: a run whose goal gate FAILED exited 0, so a pipeline gating on `crewai eval` waved it through. It is the gate's now — 0 only for PASSED; a failed gate, one without a verdict, and an evaluation that stopped are 1. The command's help says so. With no terminal and no login, an anonymous run's trace stays on the machine even with tracing on (nobody is there to approve the upload). `crewai eval` then told the user to turn tracing on, which they had — the same loop again. When tracing is on and nobody is logged in, it now says what traces an unattended run: `crewai login`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(cli): an unreadable login is the reason given when nothing was traced unattended Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(cli): print the unattended reason as text, never markup Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-29 11:47:47 +05:30
João MouraandClaude Opus 5.5	4dcd19aa2e	fix(tracing): turning tracing on is the answer, and an Evaluate button in the TUI (#7765) ... * fix(tracing): turning tracing on is the answer; stop asking again Two things a user who ran `crewai eval` sees today. `crewai eval` offers to turn tracing on and run the crew. The run finishes — and the ephemeral buffer asks "Share this execution trace with CrewAI?", which is the same question again. Turning tracing on IS consent: `tracing_asked_for()` is true when `CREWAI_TRACING_ENABLED` says so or when `tracing=True` was passed in this context, and the buffer takes it as the answer. First-time auto-collection still asks, because nobody asked for that one. And the offer named the mechanism rather than the effect — "Turn tracing on (CREWAI_TRACING_ENABLED=true stays in .env) and run the crew now?", then "Tracing is on for this project (CREWAI_TRACING_ENABLED=true in .env)." The variable is how it works, not what the user is agreeing to; both lines now say what happens and leave the plumbing to the .env file it is written in. 491 telemetry tests and 62 eval-CLI tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tracing): the TUI's modal is a prompt too, and say where traces go Two follow-ups from running it. The terminal UI installs its own consent callback, so turning tracing on still bought a modal at the end of the run — the callback is consulted before the default prompt, and the check only guarded the prompt. A host's callback stays authoritative, because an embedder may answer no for reasons of its own; what changed is that the TUI's callback, which exists to ASK, now answers for itself when tracing was asked for. And having taken the variable out of the offer, the offer should say what saying yes means: "Turn tracing on and run the crew now? The run's trace is sent to CrewAI AMP", and afterwards "Tracing is on for this project — its runs are traced to CrewAI AMP." Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(tui): an Evaluate button beside View Traces and Deploy A run finishes in the terminal UI and the next question is usually "was that any good?" — which `crewai eval` answers, from the command line, about the run that just happened. The button is there now, with `e` bound to it. It behaves like Deploy rather than like View Traces: it leaves the UI and then runs the command, because `crewai eval` talks — it prints the link to follow, waits for the verdict and prints that — and none of it belongs inside a full-screen app. It calls `eval_crew()` itself rather than reimplementing any of it, so the button and the command can never say different things; a clean SystemExit is the command deciding, and a failure is reported without failing the run that already succeeded. Declarative flows get it too — same button, same tail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(telemetry): count `crewai eval`, not only the button that runs it The TUI's buttons each record a `cli_usage:` feature span, so the new Evaluate button was counted — but the command it runs was not, and most people will type `crewai eval` rather than press a button. Every evaluation now records `cli_usage:eval`; the button keeps its own `cli_usage:evaluate`, so the difference between the two is how many were started from the TUI. It counts the command and nothing about the run: no id, no project, no verdict. And it is wrapped like every other telemetry call here — a missing tracer or a refused span never stops an evaluation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(telemetry): say which run was evaluated, and by whom `project_id` already rides every span. What only this command knows is which run is being graded, whether the caller was logged in, and — when they are — which organization they are logged in to. A feature span can carry attributes now, and `cli_usage:eval` carries those three. What it does not carry is anything about the work: no inputs, no output, no verdict, no criteria. The execution id is the join key; what was found lives in AMP, where the evaluation itself is. An anonymous caller reports `authenticated=false` and an empty organization, and is not asked for one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Address the review: a terminal, a thread, and the right run Three findings worth acting on, one already fixed, and ruff. **The security review is right.** `prompt_user_for_trace_viewing` fails closed where no one can answer, so an anonymous run in CI has never uploaded. Treating the switch as consent everywhere would have changed that: a project `.env` copied into CI carries the variable, not the person. `tracing_asked_for()` now answers only where a prompt could have been shown — the double-ask this PR is for happens in a terminal, and nothing else moves. **The ContextVar never crosses the worker.** `set_tracing_enabled` runs in the crew's validator, on the thread the crew was BUILT on, and a flow never sets it at all — so `tracing=True` without the env var still opened the modal. The TUI reads the declaration off the object as well: the same yes, wherever written. **The button now names the run it watched.** `_chain_eval()` called `crewai eval` with no id, which looks up the last recorded run — a PREVIOUS run when this one was not traced, and an offer to run the crew again when there is no record at all. The TUI remembers what was recorded before the run, hands the new id to the command, and says "this run was not traced" rather than grading someone else's. **Already fixed**: telemetry is recorded after `get_or_create_project_id()`, not before — that moved when the span gained its attributes. 492 telemetry tests, 187 CLI tests. Ruff format and check clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Close all three doors the prompt closes, not just the terminal The security review named three places `prompt_user_for_trace_viewing` fails closed — no terminal, a suite under test, and a host that suppressed tracing messages — and the first pass only honoured the terminal. A run under test, or inside a host that asked for silence, could still have uploaded on the strength of a variable. The three conditions are one function now, `_prompt_can_be_shown()`, used by the prompt itself and by `tracing_asked_for`, so the rule and the prompt it answers cannot drift apart. Where the question could not have been put to anybody, there is nothing to answer and nothing leaves the machine. The terminal UI decides for itself, and says why: it silences crewAI's console to keep it out of its own layout, which is a reason a PROMPT cannot be shown, not a reason to ask a second time — it has a screen and somebody in front of it. 494 telemetry tests, 88 TUI tests. Ruff clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Tests: a real session for the consent rule, and a laid-out screen for a click The rule that a declared CREWAI_TRACING_ENABLED needs no modal only holds where a prompt could have been shown, and a test suite is one of the places it could not — so the three cases about a real session say a person is here. The consent click waited for the screen, which is not the same as waiting for its buttons to have a region; it now waits for the layout it clicks on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * A run started for an evaluation closes itself, even in a child process A project's crew and flow are run through `uv run …`, so the app lives in another process and the ContextVar saying "an evaluation is waiting" never reaches it: the screen stayed open and `crewai eval` waited behind it, which is the trap the handshake was meant to remove. The environment is the one thing that crosses into that process, so the command sets a variable there and clears it after; the id travels back the way it already did, through the project's last-run record. A run that FAILED leaves too — the command behind the screen has its own way of saying there is nothing to grade, and it cannot say it while somebody has to quit a screen first. And the spans a run exports reach AMP a moment after the run ends, sometimes later than that, so a command that just watched the run would ask for a trace still in flight and be told, correctly and uselessly, that there is nothing there. A run we know is fresh now waits up to two minutes and says it is waiting; an id somebody typed fails at once, as it always has. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * The evaluation happens in the app, on the screen the reader is already on Closing the app the moment the run ended and leaving somebody in a bare terminal, waiting some seconds for a link to appear and a browser to open, is worse than the screen it replaced. So the evaluation runs where the run does: `Evaluating this run` under the run's own line, the report link beside it, the browser opened from there, and `Goal gate PASSED · goal 5/5 · …` when it comes back. Nothing closes; the reader leaves when they are done reading, and the link is printed once more on the way out so it outlives the screen. The Evaluate button does the same thing now instead of quitting the app and chaining into the command, so there is one implementation of one experience — `_chain_eval` and `_want_eval` are gone. Pressing it again opens the report it already made rather than paying for a second evaluation. Underneath, `crewai eval`'s two steps became callable as values: they raise `EvaluationStoppedError` with the sentence to show instead of printing and exiting, and `evaluate_run()` is the app's one door — the url as soon as AMP gives it, every unfinished answer while it waits. The command keeps the terminal wording it had; when it ran the crew itself, it now has nothing left to say and steps aside for the app it opened. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * The button says which report it opens Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * The button works while it waits: it spins, and it stays pressable A dimmed label that never moves reads as a screen that has stopped. The Evaluate button now carries the same spinner the run's own line does, the clock goes back up to eight frames a second while the evaluation runs, and the button is no longer disabled — the report page is where the progress is, so pressing it while it waits is the useful thing to do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * View Traces opens the traces The button never had a link. crewAI grants one for every exported run and prints it after a run — but never under a TUI, where its console is kept out of the layout — so the app had nothing to open and said so, which read as a feature that had stopped working. The link is now recorded beside the rest of what crewAI writes when a run's spans reach AMP, and the button reads it off that record, matched to THIS app's execution rather than to whatever ran last in the project. When there is no link the notice says which of the two reasons it is: a run nobody traced, or a traced run AMP granted no viewer for — only one of those is the reader's to fix. Verified against a real anonymous run: pressing `t` opens app.crewai.com/crewai_plus/ephemeral_traces/<id> and shows it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * The review's five: a trust boundary, a fallback, a retry, an exit and a count **A project's `.env` cannot lend itself the login.** `evaluate_run()` built its AMP client after the project's `.env` had been loaded with override, so a `CREWAI_PLUS_URL` a project put there landed in the set of origins trusted with the saved `crewai login` bearer. The command reads that variable BEFORE the `.env` — a shell export is this machine speaking — but inside the app there is no such "before", so the variable is left out of the set entirely. The request still follows it; the credential does not. **A run no app evaluated is still evaluated.** The command returned as soon as a trace existed, assuming the app had graded it — but a conversational session never ends the way a crew does and a flow with human feedback takes the terminal instead, so `crewai eval` could exit 0 having graded nothing. The app now leaves a marker naming the run it started evaluating, and the command grades anything that marker does not name. **An evaluation that stopped can be tried again.** The button re-enabled itself after a failure but the action still treated any existing evaluation as finished, so a 502 could only be escaped by re-running the crew. **A login that exits is shown on screen.** `saved_login()` fails by exiting, which is not an `Exception` and so slipped past the worker's hands: the button could sit on "Evaluating…" forever with the reason gone with the exit. **An evaluation the app runs is counted.** `cli_usage:eval` is the count of evaluations that begin, and the ones that begin in the app were missing from it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * A stop is a sentence, and the app's own word settles what it graded **"1" is not a reason.** A credential that exists but cannot be read stopped the evaluation by printing its sentence and exiting — under the app the print lands beneath the layout and the exit carries only a code, so the screen said `1`. `saved_login()` now raises the stop like everything else that stops an evaluation, the command catches it where it builds the client and prints it as before, and the app shows it. The test that claimed to cover this raised a `SystemExit` carrying a sentence, which is not what `_fail` raises; it now breaks the credential store and reads what reaches the screen. The worker still catches an exit, and says that the reason did not survive rather than echoing its code. **The record is the project's last finish, not this run.** Another run finishing in the same project replaces it while the app is open, so "is the marker about the id in the record" could send the command off to evaluate a different execution. The marker is now the app's own word about the run it watched — the id it evaluated, or null when it looked and found nothing to grade — and the command trusts it over the record, matching on time rather than on an id that may already have been overwritten. No marker still means no app got here, which is the conversational session and the flow that takes the terminal, and the command grades those itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * A project can say what good means, and `crewai eval` carries it An evaluation grades a run against the crew's own `expected_output`s — which is the prompt that produced the output, so it catches a crew that ignored its own contract and nothing else. What discriminates lives in criteria somebody writes, and there was nowhere to write them. Now there is. After the first evaluation the criteria it used are written to `eval.jsonc` in the project — once, never over a file that exists, because the one thing worse than no criteria is criteria that vanish whenever somebody evaluates — and the line that says so points at what to do with it. From then on the file travels with every evaluation, as it was written, comments and all: what a criterion means is the grader's to read, not this command's. Both doors do it, since most evaluations now happen inside the run app: the app writes the file and names it in the line that outlives its screen, and `crewai eval` in a terminal says it there. The file is bounded at the same 64KB AMP holds the field to, so one that would be refused is never uploaded and the run is graded anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * The review's asks: anonymous eval stats, a handshake a project cannot forge Usage stats are anonymous, so the `crewai eval` count says whether the caller was logged in and nothing else — no execution id, no organization (the product decision on this PR). Which runs were evaluated, and by whom, is AMP's record. A feature span now keeps only the keys its own feature lists, in both telemetry classes; everything else — the dimensions every span carries included — is dropped. The run app's "an evaluation is waiting" handshake was an environment variable any project's `.env` could set, which would start an evaluation with the machine's login on a plain `crewai run`. Its value is now a random token that counts only while the command's file of that name exists in the user's crewAI data directory; a `.env` cannot create a file. And three smaller ones: when no app evaluated the run, the command grades the project's record only if it was written after this run began; the "Wrote eval.jsonc" line after the app closes now appears (the key it read was never set); an oversized eval.jsonc is said once, through the caller's note, rather than printed under the TUI on every retry. uv.lock is back to main's: the change was re-resolution drift, nothing this PR needs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * A report link is a web address, and an anonymous request names no organization Two from the security review of 52fa5f3ae. The report url comes from whichever AMP answered — which a project's `.env` may choose — and is printed as a link and opened without a click. It is now checked where it arrives, before anything shows or opens it: HTTPS, or plain HTTP to this machine, no credentials in it, no control characters. Anything else costs the link and nothing more. The host is not pinned: the report is served by the evaluation service, not by AMP. A request sent without the login — nobody logged in, or an AMP this machine is not logged in to — no longer carries the saved organization id. AMP reads it only beside a credential, so it said nothing to AMP and told an AMP the login does not go to which organization was asking. Dropped, as anonymous trace grants already drop it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-28 11:28:52 -07:00
João MouraandClaude Opus 5	bdd1bc6200	chore(cli): move crewai eval under experimental (#7707) ... Nothing about the command changes. It is still `crewai eval`, still listed in `--help`, still takes `--run`, still prints the same lines. Only the module moved, from `crewai_cli/eval_crew.py` to `crewai_cli/experimental/eval_crew.py`, with its tests alongside. The package says what the command's location could not: its output and its options may change between releases without the deprecation cycle a settled command gets. That is honest about where it stands — most of what it depends on lives outside this repository, in an AMP endpoint and the evaluator behind it, and both are new enough that the report's shape is still moving. The command is already built to absorb some of that, printing whatever the evaluation graded rather than a list of its own, but the promise it makes to a user should match. `crewai/experimental/` has the same role on the library side, so this is the convention the repository already has rather than a new one. The CLI had no experimental package before; it does now, and `crewai eval` is its first tenant. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-21 23:41:50 -07:00
3 people	f0704ebb22	feat(skills)!: promote Skills Repository out of experimental (#6579) ... * feat(skills)!: promote Skills Repository out of experimental The registry-backed Skills Repository (crewai skill create/publish/ install/list, @org/name refs, global cache) is now mainline: - CLI: `crewai skill ...` is a top-level group; the CREWAI_EXPERIMENTAL gate and the now-empty `crewai experimental` group are removed. - Runtime: registry.py, cache.py, and events.py move from crewai.experimental.skills into crewai.skills next to the loader; the require_experimental_skills() gate is gone. crewai.experimental.skills remains as a deprecated re-export shim. - Docs: concepts/skills now leads with the CLI workflow and documents the create -> publish -> install lifecycle. Linear: n/a (requested promotion) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(skills): org-scoped publish only + docs in all languages Skills are always scoped to the publishing organization, like tools: drop the --public/--private flags from `crewai skill publish` and always send is_public=False to the registry. CLI tests assert the flag is rejected and the API never receives a public publish. Translate the new CLI-first Quick Start and the create -> publish -> install lifecycle section into ar, pt-BR, and ko concepts/skills docs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(skills): address review comments on the promotion PR - Back-compat shim now aliases the old submodules in sys.modules so `crewai.experimental.skills.registry/cache/events` imports (and patch targets) resolve to the real crewai.skills modules, not just the package-root re-exports. - `crewai skill publish` actually enforces the git-state check that --force claims to skip: unsynced repos block publishing (mirroring tool publish); standalone skill dirs outside any git repo publish without a check. - Explicit UTF-8 encoding on SKILL.md and cache-metadata reads/writes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(skills): fail closed when git state cannot be validated on publish Follow deploy's pattern: construct git.Repository(fetch=False) and only treat "not a Git repository" as skippable — any other git error (fetch/auth/misconfiguration) now blocks publish with a --force escape hatch instead of silently bypassing the sync check. Also single-style imports in the shim test (CodeQL) with the dotted shim import covered via importlib. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(skills): fetch before sync check on publish; bump mcp past advisories Publish now refreshes remote-tracking refs (repository.fetch()) before is_synced(), so ahead/behind is judged against the actual remote rather than stale local refs; a failing fetch blocks publish with the --force escape hatch. Adds a fail-closed test for fetch errors. Raise mcp to >=1.28.1,<2 (locks 1.28.1): the ~=1.26.0 pin blocked GHSA-hvrp-rf83-w775 / GHSA-jpw9-pfvf-9f58 (fixed 1.27.2) and GHSA-vj7q-gjh5-988w (fixed 1.28.1), which were failing pip-audit on this PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Vinicius Brasil <vini@hey.com>	2026-07-17 14:21:15 +00:00
Greyson LaLonde	ed91100a0f	refactor(skills): move Skills Repository to experimental + CREWAI_EXPERIMENTAL gate ... Moves the registry/cache pieces of PR #5867 under crewai.experimental.skills and the CLI commands under `crewai experimental skill`. The stable local-file skills feature (loader, parser, validation, models) stays in crewai.skills. Both entry points now require CREWAI_EXPERIMENTAL=1: - resolve_registry_ref() calls require_experimental_skills() before resolving - The `crewai experimental` CLI group raises UsageError when the flag is unset SkillDownloadStarted/CompletedEvent move out of crewai.events.types.skill_events into crewai.experimental.skills.events. * refactor(skills): move 'version' off SkillFrontmatter into metadata The skill version is now stored as `metadata.version` rather than a top-level field on `SkillFrontmatter`. A `before` validator lifts any top-level YAML `version:` into `metadata['version']` so existing SKILL.md files keep parsing.	2026-05-28 09:38:10 -07:00