crewAI mints a project id into pyproject.toml and it is committed, so it is
the same id on every machine, in CI, and for a teammate. `crewai eval`
already read it on its way past and threw it away; it now travels with the
request, so a project's evaluations can be shown together rather than each
run standing alone.
Omitted entirely outside a crewAI project, where there is no id to send.
85 passed; ruff and mypy clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two CodeRabbit findings on the credential-routing change.
- _origin() accepted http://, so a trusted-but-cleartext AMP would still
have received the bearer token in a header. The credential now also
requires an encrypted connection: HTTPS, or plain HTTP to this machine
(localhost, its subdomains, loopback), which is the rule
TraceGrantClient already applies to collector grants. Anything else
reads the run anonymously and says which of the two reasons applies.
- A verdict's grades were checked with isinstance(grade, int), which
accepts True and 6; both would have printed as real grades and let the
command exit 0. A grade is now an exact int in 1..5, or null.
Tests: a trusted http origin gets no credential, localhost does, the
encryption rule itself over nine origins, and four more malformed verdicts
(bool, 6, 0, 4.5). 56 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two review findings.
- Loading the project's .env (so `crewai eval` asks the AMP the run was
traced to) also let a project's .env choose where the saved bearer token
goes. The request still follows the project's CREWAI_PLUS_URL, because
that is how a self-hosted project is wired and the run really is there,
but the credential now goes only to an origin this machine is logged in
to: `crewai enterprise configure`'s saved settings, an address already
exported in this shell (read before .env is loaded), or app.crewai.com.
Anywhere else the run is read anonymously and the command says so, naming
`crewai enterprise configure`. The wider hole is not this command's:
`crewai run` sends the same token to the same .env-chosen URL, and that
is worth a separate look.
- saved_login() caught every exception and returned None, so an unreadable
credential store — a rotated key, a directory left owned by root — read
as "anonymous", quietly spending the run's one anonymous read and then
refusing a user who believes they are logged in. Only AuthError means
anonymous now; anything else is reported with its cause and a pointer to
`crewai login`. This matches tracing_credential() in the library, which
catches AuthError alone.
Tests: a project pointing elsewhere is read anonymously with the message
and no token; a shell-exported AMP is trusted; the configured AMP keeps the
token and says nothing; the origin rule itself; AuthError versus an
unreadable store. 41 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
João's call (2026-09-20): the confirm is y/n with Y as the default. The
prompt still says tracing stays on in .env and that the crew runs now.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review findings (CodeRabbit, the PR gate):
- The record's amp_base_url was handed to PlusAPI beside the saved login,
so a modified .crewai/last_run.json could send the token to any origin.
The client is now built from the configured AMP only (CREWAI_PLUS_URL,
the saved settings, app.crewai.com); the project's .env is loaded first,
as `crewai run` loads it, so the configured AMP is the one the run was
traced to. A record naming another address gets a one-line note and no
credential.
- The offer to turn tracing on and run the crew defaults to no and says
the .env change stays; Enter no longer spends a crew run.
- A `done` answer whose verdict is missing or malformed (no gate, grades
not an object, a grade not an int or null) is a protocol error, exit 1,
instead of an INCONCLUSIVE line with exit 0 or an AttributeError.
Tests for each: a foreign origin in the record with a saved token, the
.env-loaded same-AMP case, seven malformed verdicts, the prompt's text and
default. 59 passed (eval + plus_api); ruff and mypy clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review findings, each reproduced before the fix:
- The run-it-now offer wrote CREWAI_TRACING_ENABLED into ./.env before
checking the directory is a crewAI project, then run_crew() died on a
missing pyproject.toml with a traceback. Now: no pyproject.toml → one
sentence, exit 1, nothing written.
- httpx errors (AMP unreachable, a timeout) surfaced as tracebacks. Now
the start says "Could not reach AMP to start the evaluation: …"; while
waiting, an unreachable AMP or a 5xx is retried up to POLL_RETRIES
consecutive times, then reported with the URL — the evaluation keeps
running server-side either way.
- The POST now carries a 120 s timeout (AMP reads the run's spans inside
it), the poll 30 s.
- A 200 whose body has no known status (a non-dict, no status, a status
outside queued/running/done/failed) polled forever. Now it stops with
the status it saw and the URL.
- A 404 without a JSON message read "AMP holds no run <evaluation id>"
while polling. _refused takes "run <id>" / "evaluation <id>" and says
"AMP answered 404 for <subject>" — AMP's own message still wins.
- The record's amp_base_url was ignored; the CLI now evaluates the run at
the AMP it was traced to. --run keeps the configured AMP.
- The post-run explanation names the third cause: a crewai older than the
version that records the last run.
Tests for each, plus the previously untested paths: Ctrl-C exits 130, a
2xx without an id, a refusal mid-poll, DMN opens no browser, --run skips
the offer. 51 passed in lib/cli/tests (eval + plus_api).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`crewai eval` reads `.crewai/last_run.json` — the record crewAI writes
when a traced run's spans reach Wharf — and asks AMP to evaluate that
run: POST /crewai_plus/api/v1/tracing/evaluations with the execution id,
sending the saved `crewai login` when there is one and nothing otherwise.
AMP answers with an evaluation id and a URL; the command prints the URL,
opens it, waits for the verdict and prints it (goal gate and the four
grades), exit 1 only when the evaluation itself failed. `--run
EXECUTION_ID` evaluates another run.
With no traced run recorded it offers to turn tracing on for the project
(`CREWAI_TRACING_ENABLED=true` in .env, set_key so nothing else in the
file moves) and run the crew now with `crewai run`; without a terminal, or
declined, it prints the three steps instead. A run that leaves no record
behind is explained, never guessed at.
AMP's refusals are printed in its own words: a run that needs an account
(401 account_required), a refused credential (then `crewai login`), a run
AMP does not hold (404), rate limiting (429 with Retry-After), and any
other status with AMP's message. The two AMP calls live on the CLI's
PlusAPI subclass, so no crewai-core release is needed.
Who may evaluate what is AMP's decision, not the command's: an anonymous
run once without an account, then it needs one; a run traced while logged
in for that organization's members; a deployment execution for members
who may see its traces.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
