Compare commits

...

5 Commits

Author SHA1 Message Date
Georgi Atsev
d38ceafb68 feat(llms): add deepinfra as an openai-compatible provider (#7459)
Some checks are pending
CodeQL Advanced / Analyze (python) (push) Waiting to run
CodeQL Advanced / Analyze (actions) (push) Waiting to run
Check Documentation Broken Links / Check broken links (push) Waiting to run
Vulnerability Scan / Detect changes (push) Waiting to run
Vulnerability Scan / pip-audit (push) Blocked by required conditions
* feat(llms): add deepinfra as an openai-compatible provider

Register deepinfra in the OpenAI-compatible provider registry so
LLM(model="deepinfra/<org>/<model>") routes natively with DEEPINFRA_API_KEY
and no LiteLLM install. Model ids keep their org/model form,
DEEPINFRA_BASE_URL overrides the endpoint, and the docs accordion is added
in en, ar, ko and pt-BR, stating that the provider uses the native OpenAI
client and needs no LiteLLM install.

The model id check requires exactly one slash with both parts non-empty, and
it also runs on the explicit provider="deepinfra" path, where a malformed id
raises a ValueError that shows the expected form.

deepinfra/ model strings previously routed through LiteLLM; is_litellm=True
keeps that path.

* fix(llms): reject malformed deepinfra/ ids before the LiteLLM fallback

A deepinfra/<model> reference that is not org/model fell through to
LiteLLM, so the same bad id failed later and differently depending on
what was installed. The prefixed form now raises the same ValueError as
provider="deepinfra", with the example and the echoed value written the
way the caller wrote the id; both branches share one helper.

---------

Co-authored-by: João Moura <joaomdmoura@gmail.com>
2026-10-09 05:12:19 -03:00
João Moura
b3c4409bc5 feat(cli): record why an evaluation stopped before its verdict (#7963)
* feat(cli): record why an evaluation stopped before its verdict

`cli_usage:evaluate` counts presses of Evaluate and `cli_usage:eval` counts
evaluations that started, but about four in five presses never become an
evaluation and nothing said why. Every stop now counts once as
`cli_usage:eval_stopped:<reason>` from a closed list: untraced, declined,
no_project, login_unreadable, network_error, amp_401/403/404/4xx/5xx,
invalid_response, unexpected. The reason rides on EvaluationStoppedError from
where it is raised; messages, exit codes and screens are unchanged.

A run that `crewai eval` hands to the run app and that turns out untraced is
counted by the command only, so one stop is never counted twice.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(cli): count each eval stop once, with the reason shown

- the run app answers for a delegated run it could not start, so the waiting crewai eval does not start it again and count the stop twice
- an unreadable saved login on the unattended path counts as login_unreadable, not untraced
- an evaluation AMP itself failed counts as evaluation_failed, in the CLI and the app; --models unchanged

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 04:50:12 -03:00
João Moura
ed71a6f4d6 crewai eval --models: per-run cost and time, and no repeated progress lines (#7965)
* crewai eval --models: per-run cost and time, and no repeated progress lines

The comparison table printed each setup's cost and time summed over all
its runs, while the brief and the report say them per run. It now prints
cost_per_run / seconds_per_run under 'cost / run' and 'time / run' (the
totals, headed 'cost' / 'time', only when the service sends no per-run
figure; a column never mixes the two).

Progress deduped only against the previous line, so parallel setups made
polls alternate and the same lines printed again and again. Each line is
now said once per wait.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Keep the per-run unit when a model is unpriced, and explain its dash

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 23:15:40 -07:00
Stoyan Dimitrov
274fba60f9 COR-891 Fix SQLite flow initialization contention (#7821)
Some checks failed
Build uv cache / build-cache (3.10) (push) Waiting to run
Build uv cache / build-cache (3.11) (push) Waiting to run
Build uv cache / build-cache (3.12) (push) Waiting to run
Build uv cache / build-cache (3.13) (push) Waiting to run
CodeQL Advanced / Analyze (actions) (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled
Check Documentation Broken Links / Check broken links (push) Has been cancelled
Vulnerability Scan / Detect changes (push) Has been cancelled
Vulnerability Scan / pip-audit (push) Has been cancelled
* COR-891 Fix SQLite flow initialization contention

* fix: update vulnerable auth dependencies (COR-891)

* COR-891 Fix SQLite flow initialization contention
2026-10-08 16:19:30 -07:00
João Moura
24a1ddf23f feat(cli): record what the run app's Deploy met before an attempt (#7964)
About half the presses of Deploy in the run app never reach a deployment
attempt, and nothing said why. With no saved login the press raises before the
attempt is counted and prompts for one. That prompt now counts once as
`cli_usage:deploy_login_prompted`, and a press that still never reaches an
attempt counts as `cli_usage:deploy_stopped:<reason>`: login_still_required,
login_abandoned or login_failed. Counting re-raises; what the user sees is
unchanged.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 20:03:47 -03:00
17 changed files with 1183 additions and 48 deletions

View File

@@ -1012,6 +1012,39 @@ mode: "wide"
uv add 'crewai[litellm]'
```
</Accordion>
<Accordion title="DeepInfra">
يتصل CrewAI بـ DeepInfra عبر عميل OpenAI الأصلي، لذا لا حاجة لتثبيت LiteLLM.
عيّن متغيرات البيئة التالية في ملف `.env`:
```toml Code
# Required
DEEPINFRA_API_KEY=<your-api-key>
# Optional, point at a proxy or a private deployment endpoint
DEEPINFRA_BASE_URL=https://api.deepinfra.com/v1/openai
```
مثال الاستخدام في مشروع CrewAI:
```python Code
llm = LLM(
model="deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731",
temperature=0.7
)
```
<Info>
ميزات DeepInfra:
- نماذج مفتوحة الأوزان (Llama وQwen وDeepSeek وGLM وMistral وغيرها) خلف API واحد متوافق مع OpenAI
- استدعاء الأدوات والبث والمخرجات المنظمة بمخطط JSON
- تسعير لكل token وطول السياق منشوران في كتالوج النماذج
</Info>
معرّفات النماذج بصيغة `org/model` أصلًا، لذا يتكوّن المرجع الكامل من ثلاثة أجزاء،
مثل `deepinfra/meta-llama/Llama-4-Scout-17B-16E-Instruct`. راجع
[كتالوج النماذج المنسّق لـ CrewAI](https://api.deepinfra.com/v1/openai/models?filter=with_meta&sort_by=crewai)،
وهو عام ولا يتطلب مصادقة، للحصول على المعرّفات الحالية وأطوال السياق والأسعار.
</Accordion>
</AccordionGroup>
## بث الاستجابات

View File

@@ -1155,6 +1155,39 @@ In this section, you'll find detailed examples that help you select, configure,
uv add 'crewai[litellm]'
```
</Accordion>
<Accordion title="DeepInfra">
CrewAI talks to DeepInfra through its native OpenAI client, so no LiteLLM install is needed.
Set the following environment variables in your `.env` file:
```toml Code
# Required
DEEPINFRA_API_KEY=<your-api-key>
# Optional, point at a proxy or a private deployment endpoint
DEEPINFRA_BASE_URL=https://api.deepinfra.com/v1/openai
```
Example usage in your CrewAI project:
```python Code
llm = LLM(
model="deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731",
temperature=0.7
)
```
<Info>
DeepInfra features:
- Open-weight models (Llama, Qwen, DeepSeek, GLM, Mistral and more) behind one OpenAI-compatible API
- Tool calling, streaming and JSON schema structured outputs
- Per-token pricing and context length published in the model catalog
</Info>
Model ids are already in `org/model` form, so a full reference carries three segments,
for example `deepinfra/meta-llama/Llama-4-Scout-17B-16E-Instruct`. See the
[model catalog curated for CrewAI](https://api.deepinfra.com/v1/openai/models?filter=with_meta&sort_by=crewai),
which is public and needs no authentication, for current ids, context lengths and pricing.
</Accordion>
</AccordionGroup>
## Streaming Responses

View File

@@ -755,6 +755,39 @@ CrewAI는 고유한 기능, 인증 방법, 모델 역량을 제공하는 다양
uv add 'crewai[litellm]'
```
</Accordion>
<Accordion title="DeepInfra">
CrewAI는 네이티브 OpenAI 클라이언트로 DeepInfra에 연결하므로 LiteLLM 설치가 필요하지 않습니다.
`.env` 파일에 다음 환경 변수를 설정하십시오:
```toml Code
# Required
DEEPINFRA_API_KEY=<your-api-key>
# Optional, point at a proxy or a private deployment endpoint
DEEPINFRA_BASE_URL=https://api.deepinfra.com/v1/openai
```
CrewAI 프로젝트에서의 예시 사용법:
```python Code
llm = LLM(
model="deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731",
temperature=0.7
)
```
<Info>
DeepInfra 특징:
- 하나의 OpenAI 호환 API로 제공되는 오픈 웨이트 모델 (Llama, Qwen, DeepSeek, GLM, Mistral 등)
- 도구 호출, 스트리밍, JSON 스키마 구조화 출력 지원
- 모델 카탈로그에 공개된 토큰 단위 가격과 컨텍스트 길이
</Info>
모델 ID는 이미 `org/model` 형식이므로 전체 참조는 세 개의 세그먼트로 구성됩니다.
예: `deepinfra/meta-llama/Llama-4-Scout-17B-16E-Instruct`. 현재 ID, 컨텍스트 길이, 가격은
인증 없이 공개된 [CrewAI용으로 선별된 모델 카탈로그](https://api.deepinfra.com/v1/openai/models?filter=with_meta&sort_by=crewai)를
참조하세요.
</Accordion>
</AccordionGroup>
## 스트리밍 응답

View File

@@ -728,6 +728,39 @@ Nesta seção, você encontrará exemplos detalhados que ajudam a selecionar, co
uv add 'crewai[litellm]'
```
</Accordion>
<Accordion title="DeepInfra">
O CrewAI se comunica com o DeepInfra pelo cliente OpenAI nativo, então não é necessário instalar o LiteLLM.
Defina as seguintes variáveis de ambiente no seu arquivo `.env`:
```toml Code
# Required
DEEPINFRA_API_KEY=<your-api-key>
# Optional, point at a proxy or a private deployment endpoint
DEEPINFRA_BASE_URL=https://api.deepinfra.com/v1/openai
```
Exemplo de uso em seu projeto CrewAI:
```python Code
llm = LLM(
model="deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731",
temperature=0.7
)
```
<Info>
Recursos do DeepInfra:
- Modelos de pesos abertos (Llama, Qwen, DeepSeek, GLM, Mistral e outros) por trás de uma única API compatível com OpenAI
- Chamada de ferramentas, streaming e saídas estruturadas com JSON schema
- Preço por token e tamanho de contexto publicados no catálogo de modelos
</Info>
Os IDs dos modelos já estão no formato `org/model`, então uma referência completa tem três segmentos,
por exemplo `deepinfra/meta-llama/Llama-4-Scout-17B-16E-Instruct`. Consulte o
[catálogo de modelos selecionado para o CrewAI](https://api.deepinfra.com/v1/openai/models?filter=with_meta&sort_by=crewai),
que é público e não exige autenticação, para obter os IDs atuais, tamanhos de contexto e preços.
</Accordion>
</AccordionGroup>
## Respostas em streaming

View File

@@ -1355,6 +1355,9 @@ FooterKey .footer-key--key {
self._record_tui_button_click("evaluate")
traced = self._traced_execution_id()
if traced is None:
from crewai_cli.experimental.eval_crew import record_stop
record_stop("untraced")
self.notify(
"This run was not traced, so there is nothing to evaluate. "
"Turn tracing on and run it again.",
@@ -1387,6 +1390,7 @@ FooterKey .footer-key--key {
from crewai_cli.experimental.eval_crew import (
EvaluationStoppedError,
evaluate_run,
record_stop,
)
def back(handler: Callable[..., Any], *args: Any) -> None:
@@ -1401,21 +1405,24 @@ FooterKey .footer-key--key {
note=lambda text: back(self._evaluation_note, text),
)
except EvaluationStoppedError as stopped:
back(self._evaluation_failed, str(stopped))
return
reason, message = stopped.reason, str(stopped)
except SystemExit as exit_:
# Nothing on this path should exit any more — a stop is a value
# now — but an exit carries a code, not a reason, and "1" on screen
# is worse than saying plainly that the reason did not survive.
back(
self._evaluation_failed,
f"The evaluation stopped without saying why (exit {exit_.code}).",
)
return
reason = "unexpected"
message = f"The evaluation stopped without saying why (exit {exit_.code})."
except Exception as error: # a client bug is still an answer to show
back(self._evaluation_failed, f"{type(error).__name__}: {error}")
reason, message = "unexpected", f"{type(error).__name__}: {error}"
else:
back(self._evaluation_finished, finished)
return
back(self._evaluation_finished, finished)
# Answered for, even though it stopped before AMP started it: a
# `crewai eval` waiting behind this screen would otherwise try the run
# again, and count the same stop twice.
self._say_what_became_of_the_evaluation(execution_id)
record_stop(reason)
back(self._evaluation_failed, message)
def _evaluation_started(self, started: dict[str, Any]) -> None:
if self._evaluation is None:

View File

@@ -30,7 +30,7 @@ import os
from pathlib import Path
import sys
import time
from typing import Any, NoReturn, cast
from typing import Any, Literal, NoReturn, cast
from urllib.parse import urlparse
import uuid
import webbrowser
@@ -98,6 +98,41 @@ def _record_usage(*, logged_in: bool) -> None:
pass
StopReason = Literal[
"untraced",
"declined",
"no_project",
"login_unreadable",
"network_error",
"amp_401",
"amp_403",
"amp_404",
"amp_4xx",
"amp_5xx",
"invalid_response",
"evaluation_failed",
"unexpected",
]
def record_stop(reason: StopReason) -> None:
"""Count an evaluation that was asked for and did not reach a verdict, and why.
`cli_usage:eval` counts the evaluations that started; this is the other
side, so the gap between pressing Evaluate and seeing a verdict has a cause.
The reason is a word from a closed list and never the message, which can
carry AMP's own words or a local path.
"""
try:
from crewai_core.telemetry import Telemetry
telemetry = Telemetry()
telemetry.set_tracer()
telemetry.feature_usage_span(f"cli_usage:eval_stopped:{reason}")
except Exception: # noqa: S110 - telemetry must never break a command
pass
NOT_TRACED = (
"The run finished but no trace was recorded: the run may have failed, sharing the "
"trace was declined, or this project's crewai is older than the version that records "
@@ -110,9 +145,14 @@ class EvaluationStoppedError(RuntimeError):
Raised rather than printed-and-exited, because the same two functions serve
the terminal and the run app: one of them owns the screen, and a line
printed underneath it is a smear nobody asked for.
printed underneath it is a smear nobody asked for. REASON says why in one
word, for the usage count; an unclassified stop is "unexpected".
"""
def __init__(self, message: str, *, reason: StopReason = "unexpected") -> None:
super().__init__(message)
self.reason: StopReason = reason
def _note(text: str, style: str = "dim") -> None:
console.print(Text(text), style=style)
@@ -139,7 +179,7 @@ def eval_crew(run_id: str | None = None) -> None:
try:
client = _amp_client(trusted)
except EvaluationStoppedError as stopped:
_fail(str(stopped))
_stop(stopped)
recorded_amp = str(record.get("amp_base_url") or "").rstrip("/")
if not run_id and recorded_amp and recorded_amp != client.base_url.rstrip("/"):
console.print(
@@ -156,7 +196,7 @@ def eval_crew(run_id: str | None = None) -> None:
project_id=project_id,
)
except EvaluationStoppedError as stopped:
_fail(str(stopped))
_stop(stopped)
# After, not before: `cli_usage:eval` counts an evaluation, and a refused
# request — a run AMP does not hold, a credential it will not take — is not
# one. `_start_evaluation` raises rather than returning on those.
@@ -174,12 +214,14 @@ def eval_crew(run_id: str | None = None) -> None:
try:
finished = _wait(client, started["id"], url)
except EvaluationStoppedError as stopped:
_fail(str(stopped))
_stop(stopped)
except KeyboardInterrupt:
console.print(
Text(f"\nStill running{f' at {url}' if url else ''}."), style="yellow"
)
raise SystemExit(130) from None
if finished.get("status") == "failed":
record_stop("evaluation_failed")
_print_verdict(finished, url)
_print_brief(finished)
_say_where_the_criteria_live(write_eval_config(finished))
@@ -362,6 +404,8 @@ def evaluate_run(
record_evaluation_outcome(execution_id)
on_started(started)
finished = _wait(client, started["id"], started.get("url"), on_status=on_status)
if finished.get("status") == "failed":
record_stop("evaluation_failed")
written = write_eval_config(finished)
if written is not None:
finished = {**finished, "wrote_eval_config": written.name}
@@ -513,7 +557,8 @@ def saved_login() -> str | None:
# take the sentence with it.
raise EvaluationStoppedError(
f"Could not read the saved login ({type(error).__name__}: {error}). "
"Run `crewai login` again, or `crewai eval` will not know who you are."
"Run `crewai login` again, or `crewai eval` will not know who you are.",
reason="login_unreadable",
) from error
@@ -526,6 +571,7 @@ def _run_and_let_the_app_evaluate() -> str | None:
flow that took the terminal), and says so itself when nothing was traced.
"""
if not Path("pyproject.toml").is_file():
record_stop("no_project")
_fail(
"No crewAI project here (no pyproject.toml). Run `crewai eval` from the project's "
"directory, or name a run: `crewai eval --run EXECUTION_ID`."
@@ -538,13 +584,16 @@ def _run_and_let_the_app_evaluate() -> str | None:
if is_dmn_mode_enabled() or not sys.stdin.isatty():
# `Text`, never markup: the reason may be an OS error's own words, and
# its `[Errno 13]` would be read as a style tag.
console.print(Text(_nothing_traced_unattended() or steps), style="yellow")
reason, message = _nothing_traced_unattended() or ("untraced", steps)
record_stop(reason)
console.print(Text(message), style="yellow")
raise SystemExit(1)
if not click.confirm(
"No traced run is recorded in this project. Turn tracing on and run the crew now? "
"The run's trace is sent to CrewAI AMP.",
default=True, # João, 2026-09-20: y/n with Y as the default — the prompt says what Enter does
):
record_stop("declined")
console.print(steps, style="yellow")
raise SystemExit(0)
@@ -569,6 +618,7 @@ def _run_and_let_the_app_evaluate() -> str | None:
if marker is not None:
if marker.get("execution_id"):
return None
record_stop("untraced")
console.print(NOT_TRACED, style="bold red")
raise SystemExit(1)
@@ -582,6 +632,7 @@ def _run_and_let_the_app_evaluate() -> str | None:
if traced:
return str(traced)
record_stop("untraced")
console.print(NOT_TRACED, style="bold red")
raise SystemExit(1)
@@ -599,7 +650,7 @@ def _recorded_since(record: dict[str, Any], began: datetime) -> bool:
return when >= began - timedelta(seconds=1)
def _nothing_traced_unattended() -> str | None:
def _nothing_traced_unattended() -> tuple[StopReason, str] | None:
"""Why a run with tracing on left nothing to evaluate, when nobody was there.
An anonymous run asks before its trace leaves the machine, and a process with
@@ -607,7 +658,7 @@ def _nothing_traced_unattended() -> str | None:
not. Telling that user to turn tracing on sends them round the same loop;
logging in is what makes an unattended run traced. None when tracing is off
or there is a login: the ordinary steps are the right ones then. A login
that cannot be read says so instead.
that cannot be read says so instead, and is counted as that.
"""
if os.environ.get(TRACING_ENV_VAR, "").strip().lower() not in ("true", "1"):
return None
@@ -617,8 +668,8 @@ def _nothing_traced_unattended() -> str | None:
except EvaluationStoppedError as unreadable:
# A login that exists and cannot be read is the reason, and its
# sentence says what to do about it.
return str(unreadable)
return (
return unreadable.reason, str(unreadable)
return "untraced", (
"No traced run is recorded in this project. Tracing is on, but a run nobody is "
"watching is only traced when you are logged in: run `crewai login`, then "
"`crewai run` and `crewai eval` again."
@@ -657,7 +708,8 @@ def _start_evaluation(
)
except httpx.HTTPError as error:
raise EvaluationStoppedError(
f"Could not reach AMP to start the evaluation: {error}"
f"Could not reach AMP to start the evaluation: {error}",
reason="network_error",
) from error
# The run finished moments ago and its spans are still on their way:
# waiting is the answer, not a 404 the reader can do nothing with.
@@ -699,10 +751,12 @@ def _accepted(
payload["url"] = url
return payload
raise EvaluationStoppedError(
f"AMP answered without an evaluation id ({response.status_code})."
f"AMP answered without an evaluation id ({response.status_code}).",
reason="invalid_response",
)
raise EvaluationStoppedError(
_refusal_message(response, subject, about_a_deployment=about_a_deployment)
_refusal_message(response, subject, about_a_deployment=about_a_deployment),
reason=_refusal_reason(response),
)
@@ -764,30 +818,39 @@ def _wait(
misses += 1
if misses >= POLL_RETRIES:
raise EvaluationStoppedError(
f"Could not reach AMP while waiting ({error}); the evaluation keeps running{where}."
f"Could not reach AMP while waiting ({error}); the evaluation keeps running{where}.",
reason="network_error",
) from error
time.sleep(POLL_SECONDS)
continue
if response.status_code >= 500:
misses += 1
if misses >= POLL_RETRIES:
raise EvaluationStoppedError(_refusal_message(response, subject))
raise EvaluationStoppedError(
_refusal_message(response, subject),
reason=_refusal_reason(response),
)
time.sleep(POLL_SECONDS)
continue
if response.status_code != 200:
raise EvaluationStoppedError(_refusal_message(response, subject))
raise EvaluationStoppedError(
_refusal_message(response, subject),
reason=_refusal_reason(response),
)
misses = 0
payload = _payload(response) or {}
status = payload.get("status")
if status == "done" and not well_formed(payload):
raise EvaluationStoppedError(
f"AMP answered done without a {answer} (protocol error); follow it{where or ' on AMP'}."
f"AMP answered done without a {answer} (protocol error); follow it{where or ' on AMP'}.",
reason="invalid_response",
)
if status in FINISHED:
return payload
if status not in STATUSES:
raise EvaluationStoppedError(
f"AMP answered without a known evaluation status ({status!r}); follow it{where or ' on AMP'}."
f"AMP answered without a known evaluation status ({status!r}); follow it{where or ' on AMP'}.",
reason="invalid_response",
)
if on_status is not None:
on_status(payload)
@@ -1032,12 +1095,15 @@ def eval_models(models_text: str, deployment_id: str | None = None) -> None:
_open(url)
console.print("Waiting for the comparison…", style="dim")
shown: list[str] = []
# Every line once: setups run in parallel, so consecutive polls alternate
# between them ("judging the final output", "0 of 3 setups done", again),
# and a line said before carries nothing new.
shown: set[str] = set()
def show_progress(payload: dict[str, Any]) -> None:
line = _progress_line(payload)
if line and (not shown or shown[-1] != line):
shown.append(line)
if line and line not in shown:
shown.add(line)
_note(line)
try:
@@ -1125,13 +1191,15 @@ def _print_comparison(finished: dict[str, Any], url: str | None) -> None:
rows: list[dict[str, Any]] = comparison["models"]
grades = {area: [_grade_of(row, area) for row in rows] for area in GRADE_COLUMNS}
costs = [_number(row.get("cost_usd")) for row in rows]
seconds = [_number(row.get("seconds")) for row in rows]
# Per run, as the brief and the report say it; the totals over every run of
# a setup only from a service that sends no per-run figure.
costs, cost_header = _per_run(rows, "cost_per_run", "cost_usd", "cost")
seconds, time_header = _per_run(rows, "seconds_per_run", "seconds", "time")
best = {area: _best(values, max) for area, values in grades.items()}
cheapest, fastest = _best(costs, min), _best(seconds, min)
table = Table(show_edge=False, pad_edge=False)
for header in ("model", *GRADE_COLUMNS, "cost", "time"):
for header in ("model", *GRADE_COLUMNS, cost_header, time_header):
table.add_column(header, justify="left" if header == "model" else "right")
for n, row in enumerate(rows):
label = Text(str(row.get("label") or row.get("key") or "?"))
@@ -1152,6 +1220,14 @@ def _print_comparison(finished: dict[str, Any], url: str | None) -> None:
)
table.add_row(*cells)
console.print(table)
# A dash is a model the service could not price, or a setup with no time
# recorded — said once, so a `/ run` column with a gap is not a mystery.
if any(cost is None for cost in costs):
console.print(Text("— in cost: no price is known for that model", style="dim"))
if any(took is None for took in seconds):
console.print(
Text("— in time: no time was recorded for that setup", style="dim")
)
suggestions = _top_suggestions(comparison.get("suggestions"))
if suggestions:
@@ -1175,6 +1251,26 @@ def _print_comparison(finished: dict[str, Any], url: str | None) -> None:
console.print(Text(f"Full report: {url}"))
def _per_run(
rows: list[dict[str, Any]], per_run: str, total: str, noun: str
) -> tuple[list[float | None], str]:
"""One column's values and header: the per-run figure when every row that
has a value has one (`noun / run`), else every row's total (`noun`) — never
a column that mixes the two. A row with neither (a model the service could
not price) does not decide the unit: one unpriced model would otherwise turn
the whole column back into totals, which the brief beside it does not use.
Its dash is explained under the table."""
totals = [_number(row.get(total)) for row in rows]
runs = [_number(row.get(per_run)) for row in rows]
if any(value is not None for value in runs) and all(
run is not None
for run, whole in zip(runs, totals, strict=True)
if whole is not None
):
return runs, f"{noun} / run"
return totals, noun
def _grade_of(row: dict[str, Any], area: str) -> int | None:
grades = row.get("grades")
grade = grades.get(area) if isinstance(grades, dict) else None
@@ -1338,6 +1434,27 @@ def _refused(response: httpx.Response, subject: str) -> None:
_fail(_refusal_message(response, subject))
def _refusal_reason(response: httpx.Response) -> StopReason:
"""AMP's refusal by status class, for the usage count."""
code = response.status_code
if code == 401:
return "amp_401"
if code == 403:
return "amp_403"
if code == 404:
return "amp_404"
if 400 <= code < 500:
return "amp_4xx"
if code >= 500:
return "amp_5xx"
return "invalid_response"
def _stop(stopped: EvaluationStoppedError) -> NoReturn:
record_stop(stopped.reason)
_fail(str(stopped))
def _fail(message: str) -> NoReturn:
# `Text`, because most of what reaches here is AMP's own sentence and a
# `Console` parses square brackets. No caller relies on markup; the colour

View File

@@ -6,7 +6,7 @@ from pathlib import Path
import re
import subprocess
import sys
from typing import TYPE_CHECKING, Any
from typing import TYPE_CHECKING, Any, Literal
import click
from crewai_core.constants import CREWAI_TRAINED_AGENTS_FILE_ENV
@@ -493,6 +493,30 @@ def _run_json_crew_in_project_env(
return None
DeployButtonEvent = Literal[
"deploy_login_prompted",
"deploy_stopped:login_still_required",
"deploy_stopped:login_abandoned",
"deploy_stopped:login_failed",
]
def _record_deploy_button(event: DeployButtonEvent) -> None:
"""Count what the run app's Deploy met before a deployment attempt.
`cli_usage:deploy` counts the press and `Create Crew Deployment` the
attempt; a press that never became an attempt said nothing about why.
"""
try:
from crewai_core.telemetry import Telemetry
telemetry = Telemetry()
telemetry.set_tracer()
telemetry.feature_usage_span(f"cli_usage:{event}")
except Exception: # noqa: S110 - telemetry must never break a command
pass
def _chain_deploy() -> None:
from rich.console import Console
@@ -516,11 +540,20 @@ def _chain_deploy() -> None:
except AuthenticationRequiredError:
from crewai_cli.authentication.main import AuthenticationCommand
_record_deploy_button("deploy_login_prompted")
console.print()
AuthenticationCommand().login()
try:
AuthenticationCommand().login()
except KeyboardInterrupt:
_record_deploy_button("deploy_stopped:login_abandoned")
raise
except Exception:
_record_deploy_button("deploy_stopped:login_failed")
raise
try:
DeployCommand().create_crew(confirm=True, skip_validate=True, source="tui")
except AuthenticationRequiredError:
_record_deploy_button("deploy_stopped:login_still_required")
console.print(
"\nDeploy failed: authentication is still required.\n",
style="bold red",

View File

@@ -1331,6 +1331,70 @@ def test_models_are_compared_on_the_deployment_and_the_table_printed(deployed, m
assert "a fourth" not in out
def test_cost_and_time_are_per_run_when_the_service_sends_them(deployed, monkeypatch, capsys):
per_run = comparison()
per_run["models"][0].update(cost_per_run=0.0020, seconds_per_run=4.5)
per_run["models"][1].update(cost_per_run=0.0004, seconds_per_run=2.5)
install(monkeypatch, FakeModelsAMP(statuses=[httpx.Response(
200, json={"id": "ev-9", "status": "done", "url": URL, "comparison": per_run})]))
eval_module.eval_models(MODELS)
out = capsys.readouterr().out
assert "cost / run" in out and "time / run" in out
mini = next(line for line in out.splitlines() if "Poem composer: openai/gpt-4o-mini" in line)
base = next(line for line in out.splitlines() if "gpt-5.6-sol" in line)
assert "$0.0004 ★" in mini and "2.5s ★" in mini
assert "$0.0020" in base and "4.5s" in base
# the totals are not what the column says
assert "$0.0012" not in out and "9.5s" not in out
def test_a_column_never_mixes_per_run_and_totals(deployed, monkeypatch, capsys):
partial = comparison()
partial["models"][1].update(cost_per_run=0.0004) # one row only: the totals stand
install(monkeypatch, FakeModelsAMP(statuses=[httpx.Response(
200, json={"id": "ev-9", "status": "done", "url": URL, "comparison": partial})]))
eval_module.eval_models(MODELS)
out = capsys.readouterr().out
assert "cost / run" not in out and "time / run" not in out
assert "$0.0012 ★" in out and "$0.0004" not in out
def test_an_unpriced_model_keeps_the_per_run_unit_and_its_dash_is_explained(deployed, monkeypatch, capsys):
unpriced = comparison()
unpriced["models"][0].update(cost_per_run=0.0020, seconds_per_run=4.5)
unpriced["models"][1].update(cost_usd=None, cost_per_run=None, seconds_per_run=2.5)
install(monkeypatch, FakeModelsAMP(statuses=[httpx.Response(
200, json={"id": "ev-9", "status": "done", "url": URL, "comparison": unpriced})]))
eval_module.eval_models(MODELS)
out = capsys.readouterr().out
assert "cost / run" in out and "time / run" in out
mini = next(line for line in out.splitlines() if "Poem composer: openai/gpt-4o-mini" in line)
assert "—" in mini and "$0.0020" in out
assert "— in cost: no price is known for that model" in out
assert "— in time" not in out
def test_a_progress_line_is_said_once_even_when_parallel_setups_alternate(deployed, monkeypatch, capsys):
def at(message):
return httpx.Response(200, json={"id": "ev-9", "status": "running", "progress": {"message": message}})
alternating = [at("judging the final output"), at("0 of 3 setups done")] * 4
install(monkeypatch, FakeModelsAMP(statuses=[*alternating, at("1 of 3 setups done"), compared()]))
eval_module.eval_models(MODELS)
out = capsys.readouterr().out
assert out.count("judging the final output") == 1
assert out.count("0 of 3 setups done") == 1
assert out.index("0 of 3 setups done") < out.index("1 of 3 setups done")
def test_the_comparison_is_counted_with_the_models_and_nothing_that_names_it(deployed, monkeypatch):
spans: list[tuple[str, dict[str, str]]] = []
@@ -1456,7 +1520,7 @@ def test_a_model_without_its_provider_exits_one_before_anything_is_sent(deployed
assert "provider/model" in capsys.readouterr().out
def test_a_failed_comparison_exits_one_with_the_reason(deployed, monkeypatch, capsys):
def test_a_failed_comparison_exits_one_with_the_reason(deployed, monkeypatch, capsys, usage):
failed = httpx.Response(200, json={"id": "ev-9", "status": "failed", "error": "the deployment answered 500"})
install(monkeypatch, FakeModelsAMP(statuses=[failed]))
@@ -1465,6 +1529,8 @@ def test_a_failed_comparison_exits_one_with_the_reason(deployed, monkeypatch, ca
assert stopped.value.code == 1
assert "Comparison failed: the deployment answered 500" in capsys.readouterr().out
# `--models` compares models; it is not the evaluation the stop count is about
assert not [feature for feature in usage if "eval_stopped" in feature]
def test_done_without_a_comparison_is_a_protocol_error(deployed, monkeypatch, capsys):
@@ -1709,3 +1775,301 @@ def test_the_help_tells_an_agent_it_gets_a_brief():
) in help_text
assert "That is when AMP provides the brief: otherwise its link, when AMP sends only that, else nothing more" in help_text
assert "it prints a markdown brief after the comparison when AMP provides one" in help_text
# --- why an evaluation that was asked for never reached a verdict -----------
_REAL_SAVED_LOGIN = eval_module.saved_login
@pytest.fixture
def usage(monkeypatch):
"""Every usage feature counted, in order."""
features: list[str] = []
class FakeTelemetry:
def set_tracer(self) -> None:
pass
def feature_usage_span(self, feature, attributes=None) -> None:
features.append(feature)
monkeypatch.setattr("crewai_core.telemetry.Telemetry", FakeTelemetry)
return features
@pytest.mark.parametrize(
("status", "reason"),
[(401, "amp_401"), (403, "amp_403"), (404, "amp_404"), (422, "amp_4xx"), (503, "amp_5xx")],
)
def test_a_refused_start_counts_why_and_says_what_it_always_said(
project, monkeypatch, capsys, usage, status, reason
):
directory, _ = project
record_last_run(directory) # not a run that just ended: a 404 is not waited out
refusal = httpx.Response(status, json={"detail": "refused for a reason AMP states"})
install(monkeypatch, FakeAMP(create=refusal))
with pytest.raises(SystemExit) as exited:
eval_module.eval_crew()
assert exited.value.code == 1
assert usage == [f"cli_usage:eval_stopped:{reason}"]
expected = eval_module._refusal_message(refusal, f"run {EXECUTION_ID}")
assert expected in capsys.readouterr().out
def test_an_unreachable_amp_at_the_start_counts_as_a_network_error(project, monkeypatch, usage):
directory, _ = project
record_last_run(directory)
class Unreachable(FakeAMP):
def create_evaluation(self, execution_id, *, eval_config=None, project_id=None):
raise httpx.ConnectError("no route to host")
install(monkeypatch, Unreachable())
with pytest.raises(SystemExit, match="1"):
eval_module.eval_crew()
assert usage == ["cli_usage:eval_stopped:network_error"]
def test_a_start_without_an_evaluation_id_counts_as_an_invalid_response(project, monkeypatch, usage):
directory, _ = project
record_last_run(directory)
install(monkeypatch, FakeAMP(create=httpx.Response(202, json={"status": "queued"})))
with pytest.raises(SystemExit, match="1"):
eval_module.eval_crew()
assert usage == ["cli_usage:eval_stopped:invalid_response"]
@pytest.mark.parametrize(
("answer", "reason"),
[
(httpx.Response(200, json={"id": "ev-1", "status": "mystery"}), "invalid_response"),
(httpx.Response(200, json={"id": "ev-1", "status": "done"}), "invalid_response"),
(httpx.Response(403, json={}), "amp_403"),
],
)
def test_an_evaluation_that_stops_while_waiting_was_counted_and_says_why(
project, monkeypatch, usage, answer, reason
):
"""It started, so `cli_usage:eval` counted it; the stop is the second half."""
directory, _ = project
record_last_run(directory)
install(monkeypatch, FakeAMP(statuses=[answer]))
with pytest.raises(SystemExit, match="1"):
eval_module.eval_crew()
assert usage == ["cli_usage:eval", f"cli_usage:eval_stopped:{reason}"]
def test_losing_amp_while_waiting_counts_as_a_network_error(project, monkeypatch, usage):
directory, _ = project
record_last_run(directory)
class GoneQuiet(FakeAMP):
def get_evaluation(self, evaluation_id):
raise httpx.ReadTimeout("no answer")
install(monkeypatch, GoneQuiet())
with pytest.raises(SystemExit, match="1"):
eval_module.eval_crew()
assert usage == ["cli_usage:eval", "cli_usage:eval_stopped:network_error"]
def test_a_saved_login_that_cannot_be_read_counts_as_such(project, monkeypatch, usage):
directory, _ = project
record_last_run(directory)
monkeypatch.setattr(eval_module, "saved_login", _REAL_SAVED_LOGIN)
def broken_store():
raise OSError("permission denied")
monkeypatch.setattr(eval_module, "get_auth_token", broken_store)
install(monkeypatch, FakeAMP())
with pytest.raises(SystemExit, match="1"):
eval_module.eval_crew()
assert usage == ["cli_usage:eval_stopped:login_unreadable"]
def test_an_evaluation_amp_failed_is_counted_and_says_what_it_always_said(
project, monkeypatch, capsys, usage
):
directory, _ = project
record_last_run(directory)
failed = httpx.Response(200, json={"id": "ev-1", "status": "failed", "error": "the judge was unreachable"})
install(monkeypatch, FakeAMP(statuses=[failed]))
with pytest.raises(SystemExit) as exited:
eval_module.eval_crew()
assert exited.value.code == 1
assert usage == ["cli_usage:eval", "cli_usage:eval_stopped:evaluation_failed"]
assert "Evaluation failed: the judge was unreachable" in capsys.readouterr().out
def test_an_evaluation_amp_failed_in_the_run_app_is_counted_and_still_returned(
project, monkeypatch, usage
):
failed = httpx.Response(200, json={"id": "ev-1", "status": "failed", "error": "the judge was unreachable"})
install(monkeypatch, FakeAMP(statuses=[failed]))
finished = eval_module.evaluate_run(EXECUTION_ID, on_started=lambda started: None)
assert finished["status"] == "failed" # the app still shows AMP's reason
assert usage == ["cli_usage:eval", "cli_usage:eval_stopped:evaluation_failed"]
def test_an_unreadable_login_with_nobody_to_ask_counts_as_such(
project, monkeypatch, capsys, usage
):
"""Tracing on, no terminal, and a saved login that cannot be read: the login
is the reason shown, so it is the reason counted — not "untraced"."""
directory, _ = project
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: False)
monkeypatch.setenv("CREWAI_TRACING_ENABLED", "true")
monkeypatch.setattr(eval_module, "saved_login", _REAL_SAVED_LOGIN)
def broken_store():
raise OSError("permission denied")
monkeypatch.setattr(eval_module, "get_auth_token", broken_store)
with pytest.raises(SystemExit, match="1"):
eval_module.eval_crew()
assert usage == ["cli_usage:eval_stopped:login_unreadable"]
assert "crewai login" in capsys.readouterr().out
def test_a_start_the_run_app_was_refused_is_not_tried_again_by_the_command(
project, monkeypatch, usage
):
"""`crewai eval` runs the crew, the app it opens tries the evaluation and AMP
refuses the start. The app has shown the refusal and counted it once; the
command waiting behind it must not start the same run again."""
import contextlib
from crewai_cli.crew_run_tui import CrewRunApp
directory, _ = project
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
monkeypatch.setattr(eval_module.click, "confirm", lambda *args, **kwargs: True)
monkeypatch.setattr(eval_module, "_enable_tracing", lambda: None)
amp = install(monkeypatch, FakeAMP(create=httpx.Response(401, json={"detail": "Bad credentials"})))
monkeypatch.setattr(
CrewRunApp, "call_from_thread", lambda self, handler, *args: handler(*args)
)
shown: list[str] = []
monkeypatch.setattr(CrewRunApp, "_evaluation_failed", lambda self, message: shown.append(message))
holder: dict[str, str | None] = {"execution_id": None}
@contextlib.contextmanager
def watched():
yield holder
def the_app_runs_and_evaluates() -> None:
holder["execution_id"] = EXECUTION_ID # the app saw the run traced
CrewRunApp()._evaluate_now(EXECUTION_ID)
monkeypatch.setattr("crewai_cli.crew_run_tui.evaluating_after_run", watched)
monkeypatch.setattr(run_crew_module, "run_crew", the_app_runs_and_evaluates)
eval_module.eval_crew()
assert [call for call in amp.calls if call[0] == "create"] == [("create", EXECUTION_ID)]
assert usage == ["cli_usage:eval_stopped:amp_401"]
assert len(shown) == 1
def test_outside_a_project_counts_as_no_project(project, usage):
with pytest.raises(SystemExit, match="1"):
eval_module.eval_crew()
assert usage == ["cli_usage:eval_stopped:no_project"]
def test_nothing_traced_with_nobody_to_ask_counts_as_untraced(project, monkeypatch, usage):
directory, _ = project
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: False)
with pytest.raises(SystemExit, match="1"):
eval_module.eval_crew()
assert usage == ["cli_usage:eval_stopped:untraced"]
def test_declining_to_run_with_tracing_on_counts_as_declined(project, monkeypatch, usage):
directory, _ = project
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
monkeypatch.setattr(eval_module.click, "confirm", lambda *args, **kwargs: False)
with pytest.raises(SystemExit) as exited:
eval_module.eval_crew()
assert exited.value.code == 0
assert usage == ["cli_usage:eval_stopped:declined"]
@pytest.mark.parametrize("app_saw_the_run", [True, False])
def test_a_run_that_left_no_trace_counts_as_untraced_once(
project, monkeypatch, usage, app_saw_the_run
):
"""Whether the run app looked and found nothing, or no app got to the run,
the command is the one that counts it; the app counts nothing here."""
import contextlib
directory, _ = project
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
monkeypatch.setattr(eval_module.click, "confirm", lambda *args, **kwargs: True)
monkeypatch.setattr(eval_module, "_enable_tracing", lambda: None)
monkeypatch.setattr(run_crew_module, "run_crew", lambda: None)
@contextlib.contextmanager
def watched():
yield {"execution_id": None}
monkeypatch.setattr("crewai_cli.crew_run_tui.evaluating_after_run", watched)
monkeypatch.setattr(
eval_module,
"evaluation_marker",
lambda after: {"execution_id": None} if app_saw_the_run else None,
)
with pytest.raises(SystemExit, match="1"):
eval_module.eval_crew()
assert usage == ["cli_usage:eval_stopped:untraced"]
def test_a_failing_usage_count_never_changes_what_the_command_does(project, monkeypatch, capsys):
directory, _ = project
record_last_run(directory)
class BrokenTelemetry:
def set_tracer(self) -> None:
raise RuntimeError("exporter is down")
monkeypatch.setattr("crewai_core.telemetry.Telemetry", BrokenTelemetry)
refusal = httpx.Response(403, json={"detail": "not yours"})
install(monkeypatch, FakeAMP(create=refusal))
with pytest.raises(SystemExit) as exited:
eval_module.eval_crew()
assert exited.value.code == 1
assert eval_module._refusal_message(refusal, f"run {EXECUTION_ID}") in capsys.readouterr().out

View File

@@ -1,5 +1,5 @@
import contextvars
from datetime import datetime
from datetime import datetime, timezone
import time
from types import SimpleNamespace
from unittest.mock import Mock
@@ -43,6 +43,7 @@ from crewai_cli.crew_run_tui import (
_format_json_in_text,
_try_parse_structured,
)
import httpx
import pytest
from rich.text import Text
@@ -2231,3 +2232,229 @@ def test_format_json_in_text_still_pretty_prints_valid_json() -> None:
assert _format_json_in_text('data: {"a": 1} and [...]') == (
"data: " + '{\n "a": 1\n}' + " and [...]"
)
# --- what the run app's Deploy met before a deployment attempt ---------------
@pytest.fixture
def deploy_usage(monkeypatch) -> list[str]:
features: list[str] = []
class FakeTelemetry:
def set_tracer(self) -> None:
pass
def feature_usage_span(self, feature, attributes=None) -> None:
features.append(feature)
monkeypatch.setattr("crewai_core.telemetry.Telemetry", FakeTelemetry)
return features
def _deploy_doubles(monkeypatch, *, creates, login):
calls: list[str] = []
class FakeDeployCommand:
def create_crew(self, **kwargs) -> None:
calls.append("create")
outcome = creates.pop(0)
if outcome is not None:
raise outcome
class FakeAuthenticationCommand:
def login(self) -> None:
calls.append("login")
if login is not None:
raise login
monkeypatch.setattr("crewai_cli.deploy.main.DeployCommand", FakeDeployCommand)
monkeypatch.setattr(
"crewai_cli.authentication.main.AuthenticationCommand",
FakeAuthenticationCommand,
)
return calls
def test_a_logged_in_deploy_counts_no_hurdle(monkeypatch, deploy_usage) -> None:
calls = _deploy_doubles(monkeypatch, creates=[None], login=None)
run_crew._chain_deploy()
assert calls == ["create"]
assert deploy_usage == []
def test_a_login_prompt_that_succeeds_counts_the_prompt_only(monkeypatch, deploy_usage) -> None:
calls = _deploy_doubles(
monkeypatch, creates=[AuthenticationRequiredError(), None], login=None
)
run_crew._chain_deploy()
assert calls == ["create", "login", "create"]
assert deploy_usage == ["cli_usage:deploy_login_prompted"]
def test_still_not_logged_in_after_the_prompt_counts_why_and_says_so(
monkeypatch, capsys, deploy_usage
) -> None:
_deploy_doubles(
monkeypatch,
creates=[AuthenticationRequiredError(), AuthenticationRequiredError()],
login=None,
)
run_crew._chain_deploy()
assert deploy_usage == [
"cli_usage:deploy_login_prompted",
"cli_usage:deploy_stopped:login_still_required",
]
assert "Deploy failed: authentication is still required." in capsys.readouterr().out
@pytest.mark.parametrize(
("raised", "reason"),
[(KeyboardInterrupt(), "login_abandoned"), (httpx.HTTPError("access_denied"), "login_failed")],
)
def test_a_login_that_ends_badly_is_counted_and_still_raised(
monkeypatch, deploy_usage, raised, reason
) -> None:
"""Counting must not change what the user sees: the error still escapes."""
calls = _deploy_doubles(
monkeypatch, creates=[AuthenticationRequiredError()], login=raised
)
with pytest.raises(type(raised)):
run_crew._chain_deploy()
assert calls == ["create", "login"]
assert deploy_usage == [
"cli_usage:deploy_login_prompted",
f"cli_usage:deploy_stopped:{reason}",
]
def test_a_failing_usage_count_never_changes_the_deploy(monkeypatch, capsys) -> None:
class BrokenTelemetry:
def set_tracer(self) -> None:
raise RuntimeError("exporter is down")
monkeypatch.setattr("crewai_core.telemetry.Telemetry", BrokenTelemetry)
calls = _deploy_doubles(
monkeypatch,
creates=[AuthenticationRequiredError(), AuthenticationRequiredError()],
login=None,
)
run_crew._chain_deploy()
assert calls == ["create", "login", "create"]
assert "Deploy failed: authentication is still required." in capsys.readouterr().out
# --- why an evaluation asked for on this screen never reached a verdict ------
@pytest.fixture
def stops(monkeypatch) -> list[str]:
"""The `cli_usage:eval_stopped:*` features counted, in order."""
features: list[str] = []
class FakeTelemetry:
def set_tracer(self) -> None:
pass
def feature_usage_span(self, feature, attributes=None) -> None:
features.append(feature)
monkeypatch.setattr("crewai_core.telemetry.Telemetry", FakeTelemetry)
return features
def test_pressing_evaluate_on_an_untraced_run_counts_the_press_and_why_it_stopped(
monkeypatch, stops
) -> None:
app = CrewRunApp()
app._status = "completed"
app._telemetry = Mock()
app._execution_uuid = "run-this-app"
monkeypatch.setattr(crew_run_tui, "_trace_was_recorded", lambda uuid: False)
monkeypatch.setattr(CrewRunApp, "notify", lambda self, *a, **k: None)
app.action_evaluate_crew()
app._telemetry.feature_usage_span.assert_called_once_with("cli_usage:evaluate")
assert stops == ["cli_usage:eval_stopped:untraced"]
def test_a_run_started_by_crewai_eval_leaves_the_untraced_count_to_the_command(
monkeypatch, stops
) -> None:
"""The command waiting behind this screen says "not traced" and counts it;
counting it here as well would count one stop twice."""
monkeypatch.setattr(crew_run_tui, "_trace_was_recorded", lambda uuid: False)
monkeypatch.setattr(CrewRunApp, "notify", lambda self, *a, **k: None)
with crew_run_tui.evaluating_after_run():
app = CrewRunApp()
app._execution_uuid = "run-this-app"
app._evaluate_if_one_is_waiting()
assert stops == []
@pytest.mark.parametrize(
("raised", "reason"),
[
("amp_403", "amp_403"),
(SystemExit(1), "unexpected"),
(RuntimeError("client bug"), "unexpected"),
],
)
def test_an_evaluation_that_stops_on_this_screen_counts_why_and_answers_for_the_run(
monkeypatch, tmp_path, stops, raised, reason
) -> None:
"""The marker tells a `crewai eval` waiting behind this screen that the run
was answered for, so it neither starts it again nor counts the stop twice."""
from crewai_cli.experimental.eval_crew import (
EvaluationStoppedError,
evaluation_marker,
)
if isinstance(raised, str):
raised = EvaluationStoppedError("AMP answered 403.", reason=raised)
app = CrewRunApp()
app._evaluation = {"state": "starting"}
monkeypatch.setattr(
CrewRunApp, "call_from_thread", lambda self, handler, *args: handler(*args)
)
monkeypatch.setattr(
"crewai_cli.experimental.eval_crew.evaluate_run", Mock(side_effect=raised)
)
monkeypatch.chdir(tmp_path)
began = datetime.now(timezone.utc)
app._evaluate_now("run-this-app")
assert app._evaluation["state"] == "failed"
assert stops == [f"cli_usage:eval_stopped:{reason}"]
marker = evaluation_marker(after=began)
assert marker is not None and marker["execution_id"] == "run-this-app"
def test_an_evaluation_that_finishes_counts_no_stop(monkeypatch, stops) -> None:
app = CrewRunApp()
app._evaluation = {"state": "starting"}
monkeypatch.setattr(
CrewRunApp, "call_from_thread", lambda self, handler, *args: handler(*args)
)
monkeypatch.setattr(
"crewai_cli.experimental.eval_crew.evaluate_run",
Mock(return_value={"status": "done", "verdict": {"gate": "passed"}}),
)
app._evaluate_now("run-this-app")
assert stops == []

View File

@@ -30,6 +30,7 @@ dependencies = [
# Authentication and Security
"python-dotenv>=1.2.2,<2",
"pyjwt>=2.15.0,<3",
"oauthlib>=4.0.0,<5",
# Configuration and Utils
"click>=8.1.7,<9",
"appdirs~=1.4.4",

View File

@@ -8,6 +8,7 @@ import json
import os
from pathlib import Path
import sqlite3
from time import monotonic, sleep
from typing import TYPE_CHECKING, Any
from crewai_core.lock_store import lock as store_lock
@@ -22,6 +23,10 @@ if TYPE_CHECKING:
from crewai.flow.async_feedback.types import PendingFeedbackContext
_SQLITE_BUSY_TIMEOUT_SECONDS = 30.0
_SCHEMA_RETRY_DELAY_SECONDS = 0.05
def _json_default(obj: Any) -> Any:
"""Fallback serializer for non-primitive types in JSON dumps."""
if isinstance(obj, BaseModel):
@@ -85,9 +90,21 @@ class SQLiteFlowPersistence(FlowPersistence):
def init_db(self) -> None:
"""Create the necessary tables if they don't exist."""
deadline = monotonic() + _SQLITE_BUSY_TIMEOUT_SECONDS
while True:
try:
self._create_schema()
return
except sqlite3.OperationalError as exc: # noqa: PERF203
if "locked" not in str(exc).lower() or monotonic() >= deadline:
raise
sleep(_SCHEMA_RETRY_DELAY_SECONDS)
def _create_schema(self) -> None:
with (
store_lock(self._lock_name),
closing(sqlite3.connect(self.db_path, timeout=30)) as conn,
closing(
sqlite3.connect(self.db_path, timeout=_SQLITE_BUSY_TIMEOUT_SECONDS)
) as conn,
conn,
):
conn.execute("PRAGMA journal_mode=WAL")
@@ -190,7 +207,9 @@ class SQLiteFlowPersistence(FlowPersistence):
with (
store_lock(self._lock_name),
closing(sqlite3.connect(self.db_path, timeout=30)) as conn,
closing(
sqlite3.connect(self.db_path, timeout=_SQLITE_BUSY_TIMEOUT_SECONDS)
) as conn,
conn,
):
self._save_state_sql(conn, flow_uuid, method_name, state_dict)
@@ -204,7 +223,12 @@ class SQLiteFlowPersistence(FlowPersistence):
Returns:
The most recent state as a dictionary, or None if no state exists
"""
with closing(sqlite3.connect(self.db_path, timeout=30)) as conn, conn:
with (
closing(
sqlite3.connect(self.db_path, timeout=_SQLITE_BUSY_TIMEOUT_SECONDS)
) as conn,
conn,
):
cursor = conn.execute(
"""
SELECT state_json
@@ -242,7 +266,9 @@ class SQLiteFlowPersistence(FlowPersistence):
with (
store_lock(self._lock_name),
closing(sqlite3.connect(self.db_path, timeout=30)) as conn,
closing(
sqlite3.connect(self.db_path, timeout=_SQLITE_BUSY_TIMEOUT_SECONDS)
) as conn,
conn,
):
self._save_state_sql(conn, flow_uuid, context.method_name, state_dict)
@@ -280,7 +306,12 @@ class SQLiteFlowPersistence(FlowPersistence):
# Import here to avoid circular imports
from crewai.flow.async_feedback.types import PendingFeedbackContext
with closing(sqlite3.connect(self.db_path, timeout=30)) as conn, conn:
with (
closing(
sqlite3.connect(self.db_path, timeout=_SQLITE_BUSY_TIMEOUT_SECONDS)
) as conn,
conn,
):
cursor = conn.execute(
"""
SELECT state_json, context_json
@@ -306,7 +337,9 @@ class SQLiteFlowPersistence(FlowPersistence):
"""
with (
store_lock(self._lock_name),
closing(sqlite3.connect(self.db_path, timeout=30)) as conn,
closing(
sqlite3.connect(self.db_path, timeout=_SQLITE_BUSY_TIMEOUT_SECONDS)
) as conn,
conn,
):
conn.execute(

View File

@@ -197,6 +197,7 @@ SUPPORTED_NATIVE_PROVIDERS: Final[list[str]] = [
"cerebras",
"dashscope",
"snowflake",
"deepinfra",
]
@@ -221,6 +222,7 @@ PROVIDER_ALIASES: Final[dict[str, str]] = {
"cerebras": "cerebras",
"dashscope": "dashscope",
"snowflake": "snowflake",
"deepinfra": "deepinfra",
}
@@ -444,6 +446,12 @@ class LLM(BaseLLM):
# OpenRouter uses org/model format but accepts anything
return True
if provider == "deepinfra":
# DeepInfra ids are exactly org/model, so a full reference is
# deepinfra/<org>/<model>: one slash with both parts non-empty.
parts = model_lower.split("/")
return len(parts) == 2 and all(parts)
if provider == "snowflake":
return True
@@ -458,7 +466,8 @@ class LLM(BaseLLM):
a ``<prefix>/<model>`` string goes native when the prefix is a native
provider and the model is one it knows (or, for ``openai/``, when a
custom endpoint is configured); a bare model name infers its provider.
``native_class`` is ``None`` when the call falls back to LiteLLM.
``native_class`` is ``None`` when the call falls back to LiteLLM; a
DeepInfra id that is not ``org/model`` raises instead of falling through.
``kwargs`` is read, never mutated.
"""
custom_openai = bool(kwargs.get("custom_openai", False))
@@ -481,10 +490,16 @@ class LLM(BaseLLM):
provider = explicit_provider
use_native = True
model_string = model
if provider == "deepinfra":
cls._require_deepinfra_model(model)
elif "/" in model:
prefix, _, model_part = model.partition("/")
canonical_provider = PROVIDER_ALIASES.get(prefix.lower())
if canonical_provider == "deepinfra":
# Raise here rather than fall through to LiteLLM, where the same
# malformed id fails later and differently per environment.
cls._require_deepinfra_model(model_part, prefix=f"{prefix}/")
valid_native_model = bool(
canonical_provider
@@ -514,6 +529,20 @@ class LLM(BaseLLM):
native_class = None
return _Route(provider, model_string, native_class, custom_openai_route)
@classmethod
def _require_deepinfra_model(cls, model: str, prefix: str = "") -> None:
"""Raise unless ``model`` is the ``org/model`` form DeepInfra serves.
``prefix`` is the ``deepinfra/`` the caller wrote before ``model``, if
any, so the example and the echoed value match what they passed.
"""
if cls._matches_provider_pattern(model, "deepinfra"):
return
raise ValueError(
"DeepInfra model ids are org/model, for example "
f"'{prefix}deepseek-ai/DeepSeek-V4-Flash-0731'; got '{prefix}{model}'"
)
@classmethod
def _validate_model_in_constants(cls, model: str, provider: str) -> bool:
"""Validate if a model name exists in the provider's constants or matches provider patterns.
@@ -650,6 +679,7 @@ class LLM(BaseLLM):
"hosted_vllm",
"cerebras",
"dashscope",
"deepinfra",
}
if provider in openai_compatible_providers:
from crewai.llms.providers.openai_compatible.completion import (

View File

@@ -90,6 +90,12 @@ OPENAI_COMPATIBLE_PROVIDERS: dict[str, ProviderConfig] = {
base_url_env="DASHSCOPE_BASE_URL",
api_key_required=True,
),
"deepinfra": ProviderConfig(
base_url="https://api.deepinfra.com/v1/openai",
api_key_env="DEEPINFRA_API_KEY",
base_url_env="DEEPINFRA_BASE_URL",
api_key_required=True,
),
}
_OLLAMA_DEFAULT_PORT = 11434
@@ -142,6 +148,7 @@ class OpenAICompatibleCompletion(OpenAICompletion):
- hosted_vllm: vLLM server (https://github.com/vllm-project/vllm)
- cerebras: Cerebras (https://cerebras.ai)
- dashscope: Alibaba Dashscope/Qwen (https://dashscope.aliyun.com)
- deepinfra: DeepInfra (https://deepinfra.com)
Example:
# Using provider prefix

View File

@@ -95,6 +95,14 @@ class TestProviderRegistry:
assert config.api_key_env == "DASHSCOPE_API_KEY"
assert config.api_key_required is True
def test_deepinfra_config(self):
"""Test DeepInfra provider configuration."""
config = OPENAI_COMPATIBLE_PROVIDERS["deepinfra"]
assert config.base_url == "https://api.deepinfra.com/v1/openai"
assert config.api_key_env == "DEEPINFRA_API_KEY"
assert config.base_url_env == "DEEPINFRA_BASE_URL"
assert config.api_key_required is True
class TestNormalizeOllamaBaseUrl:
"""Tests for _normalize_ollama_base_url helper."""
@@ -162,6 +170,15 @@ class TestOpenAICompatibleCompletion:
if original is not None:
os.environ[env_key] = original
def test_deepinfra_missing_api_key_names_env_var(self):
"""DeepInfra requires a key; the error tells the user which env var to set."""
with patch.dict(os.environ, {}, clear=True), pytest.raises(
ValueError, match="DEEPINFRA_API_KEY"
):
OpenAICompatibleCompletion(
model="deepseek-ai/DeepSeek-V4-Flash-0731", provider="deepinfra"
)
def test_api_key_from_env(self):
"""Test API key is read from environment variable."""
with patch.dict(os.environ, {"DEEPSEEK_API_KEY": "test-key-from-env"}):
@@ -312,6 +329,87 @@ class TestLLMIntegration:
assert llm.provider == "dashscope"
assert llm.base_url == "https://my-dashscope.example.com/v1"
def test_llm_creates_openai_compatible_for_deepinfra(self):
"""Test LLM factory creates OpenAICompatibleCompletion for DeepInfra."""
with patch.dict(os.environ, {"DEEPINFRA_API_KEY": "test-key"}):
llm = LLM(model="deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731")
assert isinstance(llm, OpenAICompatibleCompletion)
assert llm.provider == "deepinfra"
# DeepInfra ids are org/model, so the part after the provider
# prefix is itself two segments and must reach the API intact.
assert llm.model == "deepseek-ai/DeepSeek-V4-Flash-0731"
assert llm.base_url == "https://api.deepinfra.com/v1/openai"
def test_deepinfra_base_url_env_override(self):
"""DEEPINFRA_BASE_URL redirects DeepInfra to a proxy or private endpoint."""
with patch.dict(
os.environ,
{
"DEEPINFRA_API_KEY": "test-key",
"DEEPINFRA_BASE_URL": "https://proxy.example.com/v1/openai",
},
):
llm = LLM(model="deepinfra/meta-llama/Llama-4-Scout-17B-16E-Instruct")
assert isinstance(llm, OpenAICompatibleCompletion)
assert llm.base_url == "https://proxy.example.com/v1/openai"
def test_llm_creates_openai_compatible_for_deepinfra_explicit_provider(self):
"""Explicit provider="deepinfra" routes natively without the model prefix."""
with patch.dict(os.environ, {"DEEPINFRA_API_KEY": "test-key"}):
llm = LLM(model="deepseek-ai/DeepSeek-V4-Flash-0731", provider="deepinfra")
assert isinstance(llm, OpenAICompatibleCompletion)
assert llm.provider == "deepinfra"
assert llm.model == "deepseek-ai/DeepSeek-V4-Flash-0731"
@pytest.mark.parametrize(
"model",
[
"deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731",
"DeepSeek-V4-Flash-0731",
"deepseek-ai/DeepSeek-V4-Flash-0731/",
],
)
def test_llm_explicit_deepinfra_provider_rejects_malformed_model(self, model):
"""provider="deepinfra" applies the org/model check before building a client."""
with patch.dict(os.environ, {"DEEPINFRA_API_KEY": "test-key"}), pytest.raises(
ValueError, match="org/model"
):
LLM(model=model, provider="deepinfra")
@pytest.mark.parametrize(
"model",
[
"deepinfra/DeepSeek-V4-Flash-0731",
"deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731/",
"deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731/extra",
],
)
def test_llm_rejects_malformed_deepinfra_prefixed_model(self, model):
"""A malformed deepinfra/... id raises the org/model error, never reaching LiteLLM."""
with patch.dict(os.environ, {"DEEPINFRA_API_KEY": "test-key"}), pytest.raises(
ValueError, match="org/model"
) as exc_info:
LLM(model=model)
assert model in str(exc_info.value)
@pytest.mark.parametrize(
("model", "expected"),
[
("deepseek-ai/DeepSeek-V4-Flash-0731", True),
("Qwen/Qwen3.5-27B", True),
("meta-llama/Llama-4-Scout-17B-16E-Instruct", True),
("", False),
("qwen", False),
("Qwen/", False),
("/Qwen3.5-27B", False),
("deepseek-ai/DeepSeek-V4-Flash-0731/extra", False),
("deepseek-ai/DeepSeek-V4-Flash-0731/", False),
],
)
def test_deepinfra_requires_an_org_and_a_model(self, model, expected):
"""DeepInfra references are exactly org/model: one slash, both parts present."""
assert LLM._matches_provider_pattern(model, "deepinfra") is expected
def test_llm_with_explicit_provider(self):
"""Test LLM with explicit provider parameter."""
with patch.dict(os.environ, {"DEEPSEEK_API_KEY": "test-key"}):

View File

@@ -2,17 +2,27 @@
from __future__ import annotations
from concurrent.futures import ThreadPoolExecutor
from contextlib import closing
import json
import os
import sqlite3
from pathlib import Path
import sqlite3
import subprocess
import sys
from threading import Barrier
import pytest
from crewai.flow.async_feedback.types import PendingFeedbackContext
from crewai.flow.persistence import sqlite as sqlite_module
from crewai.flow.persistence.sqlite import SQLiteFlowPersistence
from crewai.state.provider.sqlite_provider import SqliteProvider
_CONCURRENT_START_TIMEOUT_SECONDS = 10
_CHILD_PROCESS_TIMEOUT_SECONDS = 60
def _track_connections(
monkeypatch: pytest.MonkeyPatch,
) -> list[sqlite3.Connection]:
@@ -111,6 +121,79 @@ def test_flow_persistence_failed_write_rolls_back_and_closes(
assert persistence.load_state("flow-1") is None
@pytest.mark.parametrize("existing_db", [False, True])
def test_parallel_flow_initialization_does_not_use_store_lock(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, existing_db: bool
) -> None:
db_path = tmp_path / "flows.db"
if existing_db:
persistence = SQLiteFlowPersistence(str(db_path))
persistence.save_state("prior-flow", "start", {"step": 1})
persistence.save_pending_feedback("prior-flow", _make_context(), {"step": 2})
def reject_store_lock(*args: object, **kwargs: object) -> None:
raise AssertionError("initialization must not acquire a shared store lock")
monkeypatch.setattr(sqlite_module, "store_lock", reject_store_lock)
barrier = Barrier(16)
def initialize(_: int) -> SQLiteFlowPersistence:
barrier.wait(timeout=_CONCURRENT_START_TIMEOUT_SECONDS)
return SQLiteFlowPersistence(str(db_path))
with ThreadPoolExecutor(max_workers=16) as pool:
persistence_instances = list(pool.map(initialize, range(16)))
assert len(persistence_instances) == 16
with closing(sqlite3.connect(db_path)) as conn:
assert conn.execute("PRAGMA journal_mode").fetchone() == ("wal",)
assert {
row[0]
for row in conn.execute(
"SELECT name FROM sqlite_master WHERE type = 'table'"
)
} >= {"flow_states", "pending_feedback"}
if existing_db:
assert persistence_instances[0].load_state("prior-flow") == {"step": 2}
pending = persistence_instances[0].load_pending_feedback("prior-flow")
assert pending is not None and pending[0] == {"step": 2}
@pytest.mark.parametrize("existing_db", [False, True])
def test_parallel_processes_initialize_flow_db(
tmp_path: Path, existing_db: bool
) -> None:
db_path = tmp_path / "flows.db"
if existing_db:
persistence = SQLiteFlowPersistence(str(db_path))
persistence.save_state("prior-flow", "start", {"step": 1})
barrier = Barrier(2)
script = (
"import sys; from crewai.flow.persistence.sqlite import SQLiteFlowPersistence; "
"SQLiteFlowPersistence(sys.argv[1])"
)
def initialize(_: int) -> None:
barrier.wait(timeout=_CONCURRENT_START_TIMEOUT_SECONDS)
subprocess.run(
[sys.executable, "-c", script, str(db_path)],
check=True,
timeout=_CHILD_PROCESS_TIMEOUT_SECONDS,
)
with ThreadPoolExecutor(max_workers=2) as pool:
list(pool.map(initialize, range(2)))
with closing(sqlite3.connect(db_path)) as conn:
assert conn.execute("PRAGMA journal_mode").fetchone() == ("wal",)
persistence = SQLiteFlowPersistence(str(db_path))
if existing_db:
assert persistence.load_state("prior-flow") == {"step": 1}
persistence.save_state("new-flow", "start", {"step": 2})
assert persistence.load_state("new-flow") == {"step": 2}
def test_sqlite_provider_closes_every_connection(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:

View File

@@ -172,7 +172,7 @@ info = "Commits must follow Conventional Commits 1.0.0."
[tool.uv]
exclude-newer = "3 days"
# These security fixes are newer than the global supply-chain cutoff.
exclude-newer-package = { msgpack = "2026-06-20T00:00:00Z", pydantic-settings = "2026-06-20T00:00:00Z", langsmith = "2026-06-20T00:00:00Z" }
exclude-newer-package = { msgpack = "2026-06-20T00:00:00Z", pydantic-settings = "2026-06-20T00:00:00Z", langsmith = "2026-06-20T00:00:00Z", oauthlib = "2026-09-29T00:00:00Z" }
# composio-core pins rich<14 but textual requires rich>=14.
# onnxruntime 1.24+ dropped Python 3.10 wheels; cap it so qdrant[fastembed] resolves on 3.10.

3
uv.lock generated
View File

@@ -23,6 +23,7 @@ exclude-newer-span = "P3D"
[options.exclude-newer-package]
msgpack = "2026-06-20T00:00:00Z"
langsmith = "2026-06-20T00:00:00Z"
oauthlib = "2026-09-29T00:00:00Z"
pydantic-settings = "2026-06-20T00:00:00Z"
[manifest]
@@ -1448,6 +1449,7 @@ dependencies = [
{ name = "jsonref" },
{ name = "lancedb" },
{ name = "mcp" },
{ name = "oauthlib" },
{ name = "openai" },
{ name = "openpyxl" },
{ name = "opentelemetry-api" },
@@ -1565,6 +1567,7 @@ requires-dist = [
{ name = "litellm", marker = "extra == 'litellm'", specifier = ">=1.88.6,<2" },
{ name = "mcp", specifier = "~=1.28.1" },
{ name = "mem0ai", marker = "extra == 'mem0'", specifier = ">=2.0.0,<3" },
{ name = "oauthlib", specifier = ">=4.0.0,<5" },
{ name = "openai", specifier = ">=2.30.0,<3" },
{ name = "openpyxl", specifier = "~=3.1.5" },
{ name = "openpyxl", marker = "extra == 'openpyxl'", specifier = "~=3.1.5" },