mirror of
https://github.com/crewAIInc/crewAI.git
synced 2026-10-21 17:36:42 +02:00
João MouraandClaude Opus 5.5
87c2868123
* feat(cli): rewrite the scaffolded AGENTS.md around the CrewAI + AMP workflow The template opened with an API reference and mentioned traces, eval and deployment as optional extras, so coding agents rarely used them. It now opens with the steps an agent follows in a session (set up, reuse what the organization has on AMP, run, trace and evaluate every change, deploy when the work has to run elsewhere), then a short reference of the facts models get wrong. About 3.3K tokens instead of 11.9K, with outdated claims corrected. Tests assert the behaviours the old wording carried, check every documented `crewai` command against the CLI, and keep numbered lists gapless with and without the evaluation blocks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(cli): index the features AGENTS.md no longer details, and guard the file The workflow rewrite dropped references to guardrails, knowledge, MCP, conversational and declarative flows, checkpoints and more. An "Also available" index under Reference names each one with its API and docs path. Tests now guard the shipped file on every PR: a reference manifest, a token budget, text that is wrong on main, imports that must resolve, python blocks that must parse, and docs paths that must exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(cli): add a core API section to AGENTS.md for models that don't know CrewAI Measured with OpenAI models, the workflow rewrite lost the basics they don't know without the file: flow decorators and state, the @CrewBase class, task fields, CrewOutput and custom tools. A short Core API section restores them and folds in the two crew and router facts it would otherwise repeat. The reference manifest now guards those core references too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: run the AGENTS.md guards on PRs that touch the template The guards live in lib/cli/tests, which the main test workflow doesn't run. A path-filtered job runs them when the template, its tests or the docs pages it links change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: run the AGENTS.md workflow only on pull requests against main Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(cli): close the gaps where AGENTS.md taught less than the file it replaces Question by question against today's file, V6 trailed on facts it had cut or worded loosely: the context window and date options, task output folders and human review, the guardrail's return shape, test targets before writes, the eval's structural fixes, logged-in trace visibility, partial selector misses, written IDs for silent failures, ZIP deploys with an origin, the flow memory root, and LiteLLM for providers outside the native set. Each is back, verified on main. Measured on Claude Code (Haiku, Sonnet) and Codex, no question now scores below today's file. The token budget rises to 4,800 to hold them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(cli): teach the hooks and planning that telemetry counts, and three facts the old file had The index named kickoff hooks and Crew(planning=True), but the features most projects use are interception hooks (@on(InterceptionPoint...) in crewai.hooks) and agent planning (Agent(planning_config=PlanningConfig(...))). Both lines now lead with those. Three facts today's file taught come back: OpenAI's Responses API, agents judge while code computes, and answering a paused deployed flow through AMP's human_feedback_requests API. Measured on 221 questions in Claude Code (Haiku, Sonnet) and Codex, no question scores below today's file. The budget rises to 5,000 tokens, guards cover the new lines, and the never-teach list allows naming reasoning=True as replaced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(devtools): agents-md-eval fails a PR whose AGENTS.md teaches coding agents less `devtools agents-md-eval` asks a 218-question bank in fresh, tool-less Claude Code and Codex sessions, once with the base branch's AGENTS.md and once with this branch's, grades each answer by its key, and exits non-zero when either agent scores more than 10% below the base. The AGENTS.md workflow runs it on PRs that change the template, the eval or its questions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(devtools): run the AGENTS.md eval per model and count sessions with no answer The action asks Claude Code with the latest Haiku and Sonnet and Codex with gpt-5.6-luna and gpt-5.6-sol, one job per model. A session that hangs, fails or returns nothing is retried once and then counted; past 2% unanswered the comparison is called unreliable instead of scored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: run the AGENTS.md eval only on pull requests against main Drops the manual trigger and the conditions it needed; a test pins the trigger. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(devtools): say why AGENTS.md eval sessions gave no answer Each unanswered session records its exit code and last error line (keys redacted) or a timeout, and the command prints the most common reasons, so a run called unreliable can be diagnosed from the CI log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(devtools): grade 'is a different command' as a warning, accept 'call the agent directly' Two answer-key misses found reading V6's answers: a correct answer that warns off `crewai update` tripped its trap, and "No. Call the agent directly" didn't count as needing no crew. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(devtools): B-065's trap fires only on advice to run crewai update Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(devtools): read a failed session's reason from stdout when stderr is empty Claude Code prints some failures, such as 'Not logged in', on stdout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(devtools): accept 'never a fix' in C-018's key Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(devtools): read typographic apostrophes and 'replaces' as warnings in traps Codex writes "don’t"; the negation check knew only "don't", so its warnings counted as recommendations. "X replaces reasoning=True" now reads as a warning too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(devtools): add hooks and agent planning questions; D-018 needs the tools, not the package name Three questions for features telemetry counts and the bank missed: tool-call hooks, LLM-call hooks and agent planning (221 questions). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(cli): AGENTS.md teaches that crewai eval runs the crew itself, logged in or out Since #8270, `crewai eval` runs the crew, traced, when the project changed since its last traced run. The file still said to `crewai run` first, and that a run without a terminal is traced only when logged in, so logged-out coding agents logged in or stopped instead of evaluating. It now says `crewai eval` works logged in or out, with or without a terminal, with no `crewai run` between rounds and no wait on a login. The CLI test's command parser stops at a group that runs without a subcommand (`crewai eval`, `crewai eval --models ...`), which `crewai eval` became when it gained `export`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(devtools): ask whether crewai eval needs a login, and read "doesn't" as a warning Two questions join the bank: can a logged-out coding agent with no terminal get a verdict, and does it need `crewai run` first. Three keys follow main: C-004 and B-051 accept `crewai eval` (it runs and traces the crew), C-032 accepts "flow code". Negations now include doesn't, didn't, aren't and can't, so "evaluation doesn't require an account" no longer trips the trap it warns against. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ci): run the AGENTS.md eval from the base commit, with the PR's files as data The eval job checked out the pull request and ran its devtools with both API keys in the environment, so a branch could print or send them. It now checks out the base commit, fetches the pull request's commit only to read its AGENTS.md and CLAUDE.md (`agents-md-eval --head <sha>`), and runs in the `agents-md-eval` environment, where the keys can sit behind a required reviewer. When the base branch has no eval yet, the job skips with a notice. The eval also names a session's real error: Claude Code can end its output with a JSON log line after the message ("Invalid API key"), which hid every Haiku failure. Item text and reasons print as text, so `[tool.crewai]` isn't dropped and `[/...]` can't raise MarkupError after the paid runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cli): AGENTS.md says deploys build from any git origin, not only GitHub `crewai deploy create` sends whatever `origin` URL the project has, and AMP builds from GitHub or another git host (`any_git`). The file said an origin meant GitHub, which pointed GitLab and other users at the wrong fix. The GitHub integration note now applies to GitHub only, and a guard keeps it so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ci): run the AGENTS.md eval on pull_request_target, so its keys reach only reviewed code Under pull_request a branch's own copy of the workflow runs, so a pull request could edit the eval's steps and still get the keys. The eval now has its own workflow on pull_request_target: GitHub runs it from the base branch, the job checks out the base commit, and the pull request's AGENTS.md and CLAUDE.md are fetched and read as data, never run. It runs for this repository's branches only, since pull_request_target would hand the keys to forks too, and it fetches the pull request's commit only once it knows it will run, so a skip can't fail on the fetch. The guards stay on pull_request, with no secrets, and no longer persist the checkout's token for the pull request's code to read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(devtools): accept "go-ahead" as asking first in C-052's key "Publishing writes to your organization, so it needs your go-ahead first" asks before publishing, but the key only matched ask, approve, confirm or permission, so correct answers scored wrong. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>