Two review findings.
- Loading the project's .env (so `crewai eval` asks the AMP the run was
traced to) also let a project's .env choose where the saved bearer token
goes. The request still follows the project's CREWAI_PLUS_URL, because
that is how a self-hosted project is wired and the run really is there,
but the credential now goes only to an origin this machine is logged in
to: `crewai enterprise configure`'s saved settings, an address already
exported in this shell (read before .env is loaded), or app.crewai.com.
Anywhere else the run is read anonymously and the command says so, naming
`crewai enterprise configure`. The wider hole is not this command's:
`crewai run` sends the same token to the same .env-chosen URL, and that
is worth a separate look.
- saved_login() caught every exception and returned None, so an unreadable
credential store — a rotated key, a directory left owned by root — read
as "anonymous", quietly spending the run's one anonymous read and then
refusing a user who believes they are logged in. Only AuthError means
anonymous now; anything else is reported with its cause and a pointer to
`crewai login`. This matches tracing_credential() in the library, which
catches AuthError alone.
Tests: a project pointing elsewhere is read anonymously with the message
and no token; a shell-exported AMP is trusted; the configured AMP keeps the
token and says nothing; the origin rule itself; AuthError versus an
unreadable store. 41 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
João's call (2026-09-20): the confirm is y/n with Y as the default. The
prompt still says tracing stays on in .env and that the crew runs now.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review findings (CodeRabbit, the PR gate):
- The record's amp_base_url was handed to PlusAPI beside the saved login,
so a modified .crewai/last_run.json could send the token to any origin.
The client is now built from the configured AMP only (CREWAI_PLUS_URL,
the saved settings, app.crewai.com); the project's .env is loaded first,
as `crewai run` loads it, so the configured AMP is the one the run was
traced to. A record naming another address gets a one-line note and no
credential.
- The offer to turn tracing on and run the crew defaults to no and says
the .env change stays; Enter no longer spends a crew run.
- A `done` answer whose verdict is missing or malformed (no gate, grades
not an object, a grade not an int or null) is a protocol error, exit 1,
instead of an INCONCLUSIVE line with exit 0 or an AttributeError.
Tests for each: a foreign origin in the record with a saved token, the
.env-loaded same-AMP case, seven malformed verdicts, the prompt's text and
default. 59 passed (eval + plus_api); ruff and mypy clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review findings, each reproduced before the fix:
- The run-it-now offer wrote CREWAI_TRACING_ENABLED into ./.env before
checking the directory is a crewAI project, then run_crew() died on a
missing pyproject.toml with a traceback. Now: no pyproject.toml → one
sentence, exit 1, nothing written.
- httpx errors (AMP unreachable, a timeout) surfaced as tracebacks. Now
the start says "Could not reach AMP to start the evaluation: …"; while
waiting, an unreachable AMP or a 5xx is retried up to POLL_RETRIES
consecutive times, then reported with the URL — the evaluation keeps
running server-side either way.
- The POST now carries a 120 s timeout (AMP reads the run's spans inside
it), the poll 30 s.
- A 200 whose body has no known status (a non-dict, no status, a status
outside queued/running/done/failed) polled forever. Now it stops with
the status it saw and the URL.
- A 404 without a JSON message read "AMP holds no run <evaluation id>"
while polling. _refused takes "run <id>" / "evaluation <id>" and says
"AMP answered 404 for <subject>" — AMP's own message still wins.
- The record's amp_base_url was ignored; the CLI now evaluates the run at
the AMP it was traced to. --run keeps the configured AMP.
- The post-run explanation names the third cause: a crewai older than the
version that records the last run.
Tests for each, plus the previously untested paths: Ctrl-C exits 130, a
2xx without an id, a refusal mid-poll, DMN opens no browser, --run skips
the offer. 51 passed in lib/cli/tests (eval + plus_api).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`crewai eval` reads `.crewai/last_run.json` — the record crewAI writes
when a traced run's spans reach Wharf — and asks AMP to evaluate that
run: POST /crewai_plus/api/v1/tracing/evaluations with the execution id,
sending the saved `crewai login` when there is one and nothing otherwise.
AMP answers with an evaluation id and a URL; the command prints the URL,
opens it, waits for the verdict and prints it (goal gate and the four
grades), exit 1 only when the evaluation itself failed. `--run
EXECUTION_ID` evaluates another run.
With no traced run recorded it offers to turn tracing on for the project
(`CREWAI_TRACING_ENABLED=true` in .env, set_key so nothing else in the
file moves) and run the crew now with `crewai run`; without a terminal, or
declined, it prints the three steps instead. A run that leaves no record
behind is explained, never guessed at.
AMP's refusals are printed in its own words: a run that needs an account
(401 account_required), a refused credential (then `crewai login`), a run
AMP does not hold (404), rate limiting (429 with Retry-After), and any
other status with AMP's message. The two AMP calls live on the CLI's
PlusAPI subclass, so no crewai-core release is needed.
Who may evaluate what is AMP's decision, not the command's: an anonymous
run once without an account, then it needs one; a run traced while logged
in for that organization's members; a deployment execution for members
who may see its traces.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
