fix(cli): crewai eval — guard the project, survive AMP blips, name the right subject, read the whole record

Review findings, each reproduced before the fix:

- The run-it-now offer wrote CREWAI_TRACING_ENABLED into ./.env before
  checking the directory is a crewAI project, then run_crew() died on a
  missing pyproject.toml with a traceback. Now: no pyproject.toml → one
  sentence, exit 1, nothing written.
- httpx errors (AMP unreachable, a timeout) surfaced as tracebacks. Now
  the start says "Could not reach AMP to start the evaluation: …"; while
  waiting, an unreachable AMP or a 5xx is retried up to POLL_RETRIES
  consecutive times, then reported with the URL — the evaluation keeps
  running server-side either way.
- The POST now carries a 120 s timeout (AMP reads the run's spans inside
  it), the poll 30 s.
- A 200 whose body has no known status (a non-dict, no status, a status
  outside queued/running/done/failed) polled forever. Now it stops with
  the status it saw and the URL.
- A 404 without a JSON message read "AMP holds no run <evaluation id>"
  while polling. _refused takes "run <id>" / "evaluation <id>" and says
  "AMP answered 404 for <subject>" — AMP's own message still wins.
- The record's amp_base_url was ignored; the CLI now evaluates the run at
  the AMP it was traced to. --run keeps the configured AMP.
- The post-run explanation names the third cause: a crewai older than the
  version that records the last run.

Tests for each, plus the previously untested paths: Ctrl-C exits 130, a
2xx without an id, a refusal mid-poll, DMN opens no browser, --run skips
the offer. 51 passed in lib/cli/tests (eval + plus_api).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Joao Moura
2026-09-20 01:51:13 -07:00
parent 8640cda042
commit 5ac9c052d8
4 changed files with 204 additions and 31 deletions

View File

@@ -29,6 +29,7 @@ class TestPlusAPI(unittest.TestCase):
"POST",
"/crewai_plus/api/v1/tracing/evaluations",
json={"execution_id": "6f31fe1a-20bd-4bfe-a011-25d6b9341f62"},
timeout=120.0, # AMP reads the run's spans inside this request
)
self.assertEqual(response, mock_response)
@@ -40,7 +41,7 @@ class TestPlusAPI(unittest.TestCase):
response = self.api.get_evaluation("ev-1")
mock_make_request.assert_called_once_with(
"GET", "/crewai_plus/api/v1/tracing/evaluations/ev-1"
"GET", "/crewai_plus/api/v1/tracing/evaluations/ev-1", timeout=30.0
)
self.assertEqual(response, mock_response)