João MouraandClaude Opus 5.5 b0c16c162a fix(cli): crewai eval waits out AMP's rate limit instead of quitting (#8148) ...

* fix(cli): crewai eval waits out AMP's rate limit instead of quitting

On prod (2026-10-14) a `crewai eval --models` comparison ended its CLI with
exit 1 on one poll AMP answered 429 ("Too many evaluation requests. Please
retry later." — AMP allows 30 evaluation requests a minute per IP), while
the deployment went on running the comparison. A 429 mid-poll is now waited
out: Retry-After seconds, never less than a poll's interval nor more than
RATE_LIMIT_MAX_WAIT_SECONDS (120), a missing or odd header a minute; then
the poll goes on, however many come. It is not counted as a miss: being
asked to slow down is no reason to stop waiting for a running evaluation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(cli): a rate limit that never lifts ends the wait; Retry-After as RFC 9110 has it

Review on #8148:
- Rate-limited without a break for RATE_LIMITED_FOR_AT_MOST_SECONDS (15
  minutes, counted over the waits the 429s asked for and reset by any other
  answer), the wait stops: "AMP kept rate-limiting this command for 15
  minutes, so it stopped waiting; the evaluation keeps running at <url>."
  A limit that passes is still ridden out; CI, agents and the TUI no longer
  wait without end.
- Retry-After is read in both forms RFC 9110 allows, delay-seconds and an
  HTTP-date (a date already past waits a poll's interval); a negative or
  non-finite number, or words, waits a minute, as documented — NaN and the
  infinities no longer slip past that.
- The test compares the whole list of waits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(cli): the passing-limit case checks every poll it made

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(cli): the rate-limit count never overruns, and any other answer restarts it

CodeRabbit on #8148 (outside the diff):
- A 5xx between 429s is an answer too: every non-429 answer now restarts
  the count, before the 5xx branch continues.
- The last wait is what is left of RATE_LIMITED_FOR_AT_MOST_SECONDS, so the
  wait never runs past fifteen minutes (it could overrun by up to 119 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(cli): a 429 breaks a streak of unreachable or 5xx polls

CodeRabbit on #8148: POLL_RETRIES counts a consecutive streak of
unreachable or 5xx polls, and a 429 is AMP answering. Four 503s, a 429 and
one more 503 stopped the wait; the 429 now resets the streak.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(cli): one deadline over every kind of refusal while waiting

iris on #8148: a 503 reset the rate-limit count and a 429 reset the 5xx
streak, so refusals taking turns could keep CI and agent callers waiting
without end. The two counts are now one monotonic deadline (_clock):
polls that get no answer about the evaluation — rate-limited, failing,
unreachable, in any mix — for UNANSWERED_FOR_AT_MOST_SECONDS (15 minutes)
without a break end the wait, and only AMP's answer (a 200) restarts it.
The last wait is still what is left of the deadline. The sentence names the
cause: "AMP kept rate-limiting this command for 15 minutes" when every
refusal was a 429, else "AMP gave this command no answer about the
evaluation for 15 minutes (rate limits, errors, no connection)".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

2026-10-14 06:37:57 -07:00

..

__init__.py

chore(cli): move crewai eval under experimental (#7707)

2026-09-21 23:41:50 -07:00

test_eval_crew.py

fix(cli): crewai eval waits out AMP's rate limit instead of quitting (#8148)

2026-10-14 06:37:57 -07:00