mirror of
https://github.com/crewAIInc/crewAI.git
synced 2026-10-09 09:57:46 +00:00
Compare commits
5 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d38ceafb68 | ||
|
|
b3c4409bc5 | ||
|
|
ed71a6f4d6 | ||
|
|
274fba60f9 | ||
|
|
24a1ddf23f |
@@ -1012,6 +1012,39 @@ mode: "wide"
|
||||
uv add 'crewai[litellm]'
|
||||
```
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="DeepInfra">
|
||||
يتصل CrewAI بـ DeepInfra عبر عميل OpenAI الأصلي، لذا لا حاجة لتثبيت LiteLLM.
|
||||
|
||||
عيّن متغيرات البيئة التالية في ملف `.env`:
|
||||
```toml Code
|
||||
# Required
|
||||
DEEPINFRA_API_KEY=<your-api-key>
|
||||
|
||||
# Optional, point at a proxy or a private deployment endpoint
|
||||
DEEPINFRA_BASE_URL=https://api.deepinfra.com/v1/openai
|
||||
```
|
||||
|
||||
مثال الاستخدام في مشروع CrewAI:
|
||||
```python Code
|
||||
llm = LLM(
|
||||
model="deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731",
|
||||
temperature=0.7
|
||||
)
|
||||
```
|
||||
|
||||
<Info>
|
||||
ميزات DeepInfra:
|
||||
- نماذج مفتوحة الأوزان (Llama وQwen وDeepSeek وGLM وMistral وغيرها) خلف API واحد متوافق مع OpenAI
|
||||
- استدعاء الأدوات والبث والمخرجات المنظمة بمخطط JSON
|
||||
- تسعير لكل token وطول السياق منشوران في كتالوج النماذج
|
||||
</Info>
|
||||
|
||||
معرّفات النماذج بصيغة `org/model` أصلًا، لذا يتكوّن المرجع الكامل من ثلاثة أجزاء،
|
||||
مثل `deepinfra/meta-llama/Llama-4-Scout-17B-16E-Instruct`. راجع
|
||||
[كتالوج النماذج المنسّق لـ CrewAI](https://api.deepinfra.com/v1/openai/models?filter=with_meta&sort_by=crewai)،
|
||||
وهو عام ولا يتطلب مصادقة، للحصول على المعرّفات الحالية وأطوال السياق والأسعار.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
## بث الاستجابات
|
||||
|
||||
@@ -1155,6 +1155,39 @@ In this section, you'll find detailed examples that help you select, configure,
|
||||
uv add 'crewai[litellm]'
|
||||
```
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="DeepInfra">
|
||||
CrewAI talks to DeepInfra through its native OpenAI client, so no LiteLLM install is needed.
|
||||
|
||||
Set the following environment variables in your `.env` file:
|
||||
```toml Code
|
||||
# Required
|
||||
DEEPINFRA_API_KEY=<your-api-key>
|
||||
|
||||
# Optional, point at a proxy or a private deployment endpoint
|
||||
DEEPINFRA_BASE_URL=https://api.deepinfra.com/v1/openai
|
||||
```
|
||||
|
||||
Example usage in your CrewAI project:
|
||||
```python Code
|
||||
llm = LLM(
|
||||
model="deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731",
|
||||
temperature=0.7
|
||||
)
|
||||
```
|
||||
|
||||
<Info>
|
||||
DeepInfra features:
|
||||
- Open-weight models (Llama, Qwen, DeepSeek, GLM, Mistral and more) behind one OpenAI-compatible API
|
||||
- Tool calling, streaming and JSON schema structured outputs
|
||||
- Per-token pricing and context length published in the model catalog
|
||||
</Info>
|
||||
|
||||
Model ids are already in `org/model` form, so a full reference carries three segments,
|
||||
for example `deepinfra/meta-llama/Llama-4-Scout-17B-16E-Instruct`. See the
|
||||
[model catalog curated for CrewAI](https://api.deepinfra.com/v1/openai/models?filter=with_meta&sort_by=crewai),
|
||||
which is public and needs no authentication, for current ids, context lengths and pricing.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
## Streaming Responses
|
||||
|
||||
@@ -755,6 +755,39 @@ CrewAI는 고유한 기능, 인증 방법, 모델 역량을 제공하는 다양
|
||||
uv add 'crewai[litellm]'
|
||||
```
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="DeepInfra">
|
||||
CrewAI는 네이티브 OpenAI 클라이언트로 DeepInfra에 연결하므로 LiteLLM 설치가 필요하지 않습니다.
|
||||
|
||||
`.env` 파일에 다음 환경 변수를 설정하십시오:
|
||||
```toml Code
|
||||
# Required
|
||||
DEEPINFRA_API_KEY=<your-api-key>
|
||||
|
||||
# Optional, point at a proxy or a private deployment endpoint
|
||||
DEEPINFRA_BASE_URL=https://api.deepinfra.com/v1/openai
|
||||
```
|
||||
|
||||
CrewAI 프로젝트에서의 예시 사용법:
|
||||
```python Code
|
||||
llm = LLM(
|
||||
model="deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731",
|
||||
temperature=0.7
|
||||
)
|
||||
```
|
||||
|
||||
<Info>
|
||||
DeepInfra 특징:
|
||||
- 하나의 OpenAI 호환 API로 제공되는 오픈 웨이트 모델 (Llama, Qwen, DeepSeek, GLM, Mistral 등)
|
||||
- 도구 호출, 스트리밍, JSON 스키마 구조화 출력 지원
|
||||
- 모델 카탈로그에 공개된 토큰 단위 가격과 컨텍스트 길이
|
||||
</Info>
|
||||
|
||||
모델 ID는 이미 `org/model` 형식이므로 전체 참조는 세 개의 세그먼트로 구성됩니다.
|
||||
예: `deepinfra/meta-llama/Llama-4-Scout-17B-16E-Instruct`. 현재 ID, 컨텍스트 길이, 가격은
|
||||
인증 없이 공개된 [CrewAI용으로 선별된 모델 카탈로그](https://api.deepinfra.com/v1/openai/models?filter=with_meta&sort_by=crewai)를
|
||||
참조하세요.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
## 스트리밍 응답
|
||||
|
||||
@@ -728,6 +728,39 @@ Nesta seção, você encontrará exemplos detalhados que ajudam a selecionar, co
|
||||
uv add 'crewai[litellm]'
|
||||
```
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="DeepInfra">
|
||||
O CrewAI se comunica com o DeepInfra pelo cliente OpenAI nativo, então não é necessário instalar o LiteLLM.
|
||||
|
||||
Defina as seguintes variáveis de ambiente no seu arquivo `.env`:
|
||||
```toml Code
|
||||
# Required
|
||||
DEEPINFRA_API_KEY=<your-api-key>
|
||||
|
||||
# Optional, point at a proxy or a private deployment endpoint
|
||||
DEEPINFRA_BASE_URL=https://api.deepinfra.com/v1/openai
|
||||
```
|
||||
|
||||
Exemplo de uso em seu projeto CrewAI:
|
||||
```python Code
|
||||
llm = LLM(
|
||||
model="deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731",
|
||||
temperature=0.7
|
||||
)
|
||||
```
|
||||
|
||||
<Info>
|
||||
Recursos do DeepInfra:
|
||||
- Modelos de pesos abertos (Llama, Qwen, DeepSeek, GLM, Mistral e outros) por trás de uma única API compatível com OpenAI
|
||||
- Chamada de ferramentas, streaming e saídas estruturadas com JSON schema
|
||||
- Preço por token e tamanho de contexto publicados no catálogo de modelos
|
||||
</Info>
|
||||
|
||||
Os IDs dos modelos já estão no formato `org/model`, então uma referência completa tem três segmentos,
|
||||
por exemplo `deepinfra/meta-llama/Llama-4-Scout-17B-16E-Instruct`. Consulte o
|
||||
[catálogo de modelos selecionado para o CrewAI](https://api.deepinfra.com/v1/openai/models?filter=with_meta&sort_by=crewai),
|
||||
que é público e não exige autenticação, para obter os IDs atuais, tamanhos de contexto e preços.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
## Respostas em streaming
|
||||
|
||||
@@ -1355,6 +1355,9 @@ FooterKey .footer-key--key {
|
||||
self._record_tui_button_click("evaluate")
|
||||
traced = self._traced_execution_id()
|
||||
if traced is None:
|
||||
from crewai_cli.experimental.eval_crew import record_stop
|
||||
|
||||
record_stop("untraced")
|
||||
self.notify(
|
||||
"This run was not traced, so there is nothing to evaluate. "
|
||||
"Turn tracing on and run it again.",
|
||||
@@ -1387,6 +1390,7 @@ FooterKey .footer-key--key {
|
||||
from crewai_cli.experimental.eval_crew import (
|
||||
EvaluationStoppedError,
|
||||
evaluate_run,
|
||||
record_stop,
|
||||
)
|
||||
|
||||
def back(handler: Callable[..., Any], *args: Any) -> None:
|
||||
@@ -1401,21 +1405,24 @@ FooterKey .footer-key--key {
|
||||
note=lambda text: back(self._evaluation_note, text),
|
||||
)
|
||||
except EvaluationStoppedError as stopped:
|
||||
back(self._evaluation_failed, str(stopped))
|
||||
return
|
||||
reason, message = stopped.reason, str(stopped)
|
||||
except SystemExit as exit_:
|
||||
# Nothing on this path should exit any more — a stop is a value
|
||||
# now — but an exit carries a code, not a reason, and "1" on screen
|
||||
# is worse than saying plainly that the reason did not survive.
|
||||
back(
|
||||
self._evaluation_failed,
|
||||
f"The evaluation stopped without saying why (exit {exit_.code}).",
|
||||
)
|
||||
return
|
||||
reason = "unexpected"
|
||||
message = f"The evaluation stopped without saying why (exit {exit_.code})."
|
||||
except Exception as error: # a client bug is still an answer to show
|
||||
back(self._evaluation_failed, f"{type(error).__name__}: {error}")
|
||||
reason, message = "unexpected", f"{type(error).__name__}: {error}"
|
||||
else:
|
||||
back(self._evaluation_finished, finished)
|
||||
return
|
||||
back(self._evaluation_finished, finished)
|
||||
# Answered for, even though it stopped before AMP started it: a
|
||||
# `crewai eval` waiting behind this screen would otherwise try the run
|
||||
# again, and count the same stop twice.
|
||||
self._say_what_became_of_the_evaluation(execution_id)
|
||||
record_stop(reason)
|
||||
back(self._evaluation_failed, message)
|
||||
|
||||
def _evaluation_started(self, started: dict[str, Any]) -> None:
|
||||
if self._evaluation is None:
|
||||
|
||||
@@ -30,7 +30,7 @@ import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import time
|
||||
from typing import Any, NoReturn, cast
|
||||
from typing import Any, Literal, NoReturn, cast
|
||||
from urllib.parse import urlparse
|
||||
import uuid
|
||||
import webbrowser
|
||||
@@ -98,6 +98,41 @@ def _record_usage(*, logged_in: bool) -> None:
|
||||
pass
|
||||
|
||||
|
||||
StopReason = Literal[
|
||||
"untraced",
|
||||
"declined",
|
||||
"no_project",
|
||||
"login_unreadable",
|
||||
"network_error",
|
||||
"amp_401",
|
||||
"amp_403",
|
||||
"amp_404",
|
||||
"amp_4xx",
|
||||
"amp_5xx",
|
||||
"invalid_response",
|
||||
"evaluation_failed",
|
||||
"unexpected",
|
||||
]
|
||||
|
||||
|
||||
def record_stop(reason: StopReason) -> None:
|
||||
"""Count an evaluation that was asked for and did not reach a verdict, and why.
|
||||
|
||||
`cli_usage:eval` counts the evaluations that started; this is the other
|
||||
side, so the gap between pressing Evaluate and seeing a verdict has a cause.
|
||||
The reason is a word from a closed list and never the message, which can
|
||||
carry AMP's own words or a local path.
|
||||
"""
|
||||
try:
|
||||
from crewai_core.telemetry import Telemetry
|
||||
|
||||
telemetry = Telemetry()
|
||||
telemetry.set_tracer()
|
||||
telemetry.feature_usage_span(f"cli_usage:eval_stopped:{reason}")
|
||||
except Exception: # noqa: S110 - telemetry must never break a command
|
||||
pass
|
||||
|
||||
|
||||
NOT_TRACED = (
|
||||
"The run finished but no trace was recorded: the run may have failed, sharing the "
|
||||
"trace was declined, or this project's crewai is older than the version that records "
|
||||
@@ -110,9 +145,14 @@ class EvaluationStoppedError(RuntimeError):
|
||||
|
||||
Raised rather than printed-and-exited, because the same two functions serve
|
||||
the terminal and the run app: one of them owns the screen, and a line
|
||||
printed underneath it is a smear nobody asked for.
|
||||
printed underneath it is a smear nobody asked for. REASON says why in one
|
||||
word, for the usage count; an unclassified stop is "unexpected".
|
||||
"""
|
||||
|
||||
def __init__(self, message: str, *, reason: StopReason = "unexpected") -> None:
|
||||
super().__init__(message)
|
||||
self.reason: StopReason = reason
|
||||
|
||||
|
||||
def _note(text: str, style: str = "dim") -> None:
|
||||
console.print(Text(text), style=style)
|
||||
@@ -139,7 +179,7 @@ def eval_crew(run_id: str | None = None) -> None:
|
||||
try:
|
||||
client = _amp_client(trusted)
|
||||
except EvaluationStoppedError as stopped:
|
||||
_fail(str(stopped))
|
||||
_stop(stopped)
|
||||
recorded_amp = str(record.get("amp_base_url") or "").rstrip("/")
|
||||
if not run_id and recorded_amp and recorded_amp != client.base_url.rstrip("/"):
|
||||
console.print(
|
||||
@@ -156,7 +196,7 @@ def eval_crew(run_id: str | None = None) -> None:
|
||||
project_id=project_id,
|
||||
)
|
||||
except EvaluationStoppedError as stopped:
|
||||
_fail(str(stopped))
|
||||
_stop(stopped)
|
||||
# After, not before: `cli_usage:eval` counts an evaluation, and a refused
|
||||
# request — a run AMP does not hold, a credential it will not take — is not
|
||||
# one. `_start_evaluation` raises rather than returning on those.
|
||||
@@ -174,12 +214,14 @@ def eval_crew(run_id: str | None = None) -> None:
|
||||
try:
|
||||
finished = _wait(client, started["id"], url)
|
||||
except EvaluationStoppedError as stopped:
|
||||
_fail(str(stopped))
|
||||
_stop(stopped)
|
||||
except KeyboardInterrupt:
|
||||
console.print(
|
||||
Text(f"\nStill running{f' at {url}' if url else ''}."), style="yellow"
|
||||
)
|
||||
raise SystemExit(130) from None
|
||||
if finished.get("status") == "failed":
|
||||
record_stop("evaluation_failed")
|
||||
_print_verdict(finished, url)
|
||||
_print_brief(finished)
|
||||
_say_where_the_criteria_live(write_eval_config(finished))
|
||||
@@ -362,6 +404,8 @@ def evaluate_run(
|
||||
record_evaluation_outcome(execution_id)
|
||||
on_started(started)
|
||||
finished = _wait(client, started["id"], started.get("url"), on_status=on_status)
|
||||
if finished.get("status") == "failed":
|
||||
record_stop("evaluation_failed")
|
||||
written = write_eval_config(finished)
|
||||
if written is not None:
|
||||
finished = {**finished, "wrote_eval_config": written.name}
|
||||
@@ -513,7 +557,8 @@ def saved_login() -> str | None:
|
||||
# take the sentence with it.
|
||||
raise EvaluationStoppedError(
|
||||
f"Could not read the saved login ({type(error).__name__}: {error}). "
|
||||
"Run `crewai login` again, or `crewai eval` will not know who you are."
|
||||
"Run `crewai login` again, or `crewai eval` will not know who you are.",
|
||||
reason="login_unreadable",
|
||||
) from error
|
||||
|
||||
|
||||
@@ -526,6 +571,7 @@ def _run_and_let_the_app_evaluate() -> str | None:
|
||||
flow that took the terminal), and says so itself when nothing was traced.
|
||||
"""
|
||||
if not Path("pyproject.toml").is_file():
|
||||
record_stop("no_project")
|
||||
_fail(
|
||||
"No crewAI project here (no pyproject.toml). Run `crewai eval` from the project's "
|
||||
"directory, or name a run: `crewai eval --run EXECUTION_ID`."
|
||||
@@ -538,13 +584,16 @@ def _run_and_let_the_app_evaluate() -> str | None:
|
||||
if is_dmn_mode_enabled() or not sys.stdin.isatty():
|
||||
# `Text`, never markup: the reason may be an OS error's own words, and
|
||||
# its `[Errno 13]` would be read as a style tag.
|
||||
console.print(Text(_nothing_traced_unattended() or steps), style="yellow")
|
||||
reason, message = _nothing_traced_unattended() or ("untraced", steps)
|
||||
record_stop(reason)
|
||||
console.print(Text(message), style="yellow")
|
||||
raise SystemExit(1)
|
||||
if not click.confirm(
|
||||
"No traced run is recorded in this project. Turn tracing on and run the crew now? "
|
||||
"The run's trace is sent to CrewAI AMP.",
|
||||
default=True, # João, 2026-09-20: y/n with Y as the default — the prompt says what Enter does
|
||||
):
|
||||
record_stop("declined")
|
||||
console.print(steps, style="yellow")
|
||||
raise SystemExit(0)
|
||||
|
||||
@@ -569,6 +618,7 @@ def _run_and_let_the_app_evaluate() -> str | None:
|
||||
if marker is not None:
|
||||
if marker.get("execution_id"):
|
||||
return None
|
||||
record_stop("untraced")
|
||||
console.print(NOT_TRACED, style="bold red")
|
||||
raise SystemExit(1)
|
||||
|
||||
@@ -582,6 +632,7 @@ def _run_and_let_the_app_evaluate() -> str | None:
|
||||
if traced:
|
||||
return str(traced)
|
||||
|
||||
record_stop("untraced")
|
||||
console.print(NOT_TRACED, style="bold red")
|
||||
raise SystemExit(1)
|
||||
|
||||
@@ -599,7 +650,7 @@ def _recorded_since(record: dict[str, Any], began: datetime) -> bool:
|
||||
return when >= began - timedelta(seconds=1)
|
||||
|
||||
|
||||
def _nothing_traced_unattended() -> str | None:
|
||||
def _nothing_traced_unattended() -> tuple[StopReason, str] | None:
|
||||
"""Why a run with tracing on left nothing to evaluate, when nobody was there.
|
||||
|
||||
An anonymous run asks before its trace leaves the machine, and a process with
|
||||
@@ -607,7 +658,7 @@ def _nothing_traced_unattended() -> str | None:
|
||||
not. Telling that user to turn tracing on sends them round the same loop;
|
||||
logging in is what makes an unattended run traced. None when tracing is off
|
||||
or there is a login: the ordinary steps are the right ones then. A login
|
||||
that cannot be read says so instead.
|
||||
that cannot be read says so instead, and is counted as that.
|
||||
"""
|
||||
if os.environ.get(TRACING_ENV_VAR, "").strip().lower() not in ("true", "1"):
|
||||
return None
|
||||
@@ -617,8 +668,8 @@ def _nothing_traced_unattended() -> str | None:
|
||||
except EvaluationStoppedError as unreadable:
|
||||
# A login that exists and cannot be read is the reason, and its
|
||||
# sentence says what to do about it.
|
||||
return str(unreadable)
|
||||
return (
|
||||
return unreadable.reason, str(unreadable)
|
||||
return "untraced", (
|
||||
"No traced run is recorded in this project. Tracing is on, but a run nobody is "
|
||||
"watching is only traced when you are logged in: run `crewai login`, then "
|
||||
"`crewai run` and `crewai eval` again."
|
||||
@@ -657,7 +708,8 @@ def _start_evaluation(
|
||||
)
|
||||
except httpx.HTTPError as error:
|
||||
raise EvaluationStoppedError(
|
||||
f"Could not reach AMP to start the evaluation: {error}"
|
||||
f"Could not reach AMP to start the evaluation: {error}",
|
||||
reason="network_error",
|
||||
) from error
|
||||
# The run finished moments ago and its spans are still on their way:
|
||||
# waiting is the answer, not a 404 the reader can do nothing with.
|
||||
@@ -699,10 +751,12 @@ def _accepted(
|
||||
payload["url"] = url
|
||||
return payload
|
||||
raise EvaluationStoppedError(
|
||||
f"AMP answered without an evaluation id ({response.status_code})."
|
||||
f"AMP answered without an evaluation id ({response.status_code}).",
|
||||
reason="invalid_response",
|
||||
)
|
||||
raise EvaluationStoppedError(
|
||||
_refusal_message(response, subject, about_a_deployment=about_a_deployment)
|
||||
_refusal_message(response, subject, about_a_deployment=about_a_deployment),
|
||||
reason=_refusal_reason(response),
|
||||
)
|
||||
|
||||
|
||||
@@ -764,30 +818,39 @@ def _wait(
|
||||
misses += 1
|
||||
if misses >= POLL_RETRIES:
|
||||
raise EvaluationStoppedError(
|
||||
f"Could not reach AMP while waiting ({error}); the evaluation keeps running{where}."
|
||||
f"Could not reach AMP while waiting ({error}); the evaluation keeps running{where}.",
|
||||
reason="network_error",
|
||||
) from error
|
||||
time.sleep(POLL_SECONDS)
|
||||
continue
|
||||
if response.status_code >= 500:
|
||||
misses += 1
|
||||
if misses >= POLL_RETRIES:
|
||||
raise EvaluationStoppedError(_refusal_message(response, subject))
|
||||
raise EvaluationStoppedError(
|
||||
_refusal_message(response, subject),
|
||||
reason=_refusal_reason(response),
|
||||
)
|
||||
time.sleep(POLL_SECONDS)
|
||||
continue
|
||||
if response.status_code != 200:
|
||||
raise EvaluationStoppedError(_refusal_message(response, subject))
|
||||
raise EvaluationStoppedError(
|
||||
_refusal_message(response, subject),
|
||||
reason=_refusal_reason(response),
|
||||
)
|
||||
misses = 0
|
||||
payload = _payload(response) or {}
|
||||
status = payload.get("status")
|
||||
if status == "done" and not well_formed(payload):
|
||||
raise EvaluationStoppedError(
|
||||
f"AMP answered done without a {answer} (protocol error); follow it{where or ' on AMP'}."
|
||||
f"AMP answered done without a {answer} (protocol error); follow it{where or ' on AMP'}.",
|
||||
reason="invalid_response",
|
||||
)
|
||||
if status in FINISHED:
|
||||
return payload
|
||||
if status not in STATUSES:
|
||||
raise EvaluationStoppedError(
|
||||
f"AMP answered without a known evaluation status ({status!r}); follow it{where or ' on AMP'}."
|
||||
f"AMP answered without a known evaluation status ({status!r}); follow it{where or ' on AMP'}.",
|
||||
reason="invalid_response",
|
||||
)
|
||||
if on_status is not None:
|
||||
on_status(payload)
|
||||
@@ -1032,12 +1095,15 @@ def eval_models(models_text: str, deployment_id: str | None = None) -> None:
|
||||
_open(url)
|
||||
|
||||
console.print("Waiting for the comparison…", style="dim")
|
||||
shown: list[str] = []
|
||||
# Every line once: setups run in parallel, so consecutive polls alternate
|
||||
# between them ("judging the final output", "0 of 3 setups done", again),
|
||||
# and a line said before carries nothing new.
|
||||
shown: set[str] = set()
|
||||
|
||||
def show_progress(payload: dict[str, Any]) -> None:
|
||||
line = _progress_line(payload)
|
||||
if line and (not shown or shown[-1] != line):
|
||||
shown.append(line)
|
||||
if line and line not in shown:
|
||||
shown.add(line)
|
||||
_note(line)
|
||||
|
||||
try:
|
||||
@@ -1125,13 +1191,15 @@ def _print_comparison(finished: dict[str, Any], url: str | None) -> None:
|
||||
rows: list[dict[str, Any]] = comparison["models"]
|
||||
|
||||
grades = {area: [_grade_of(row, area) for row in rows] for area in GRADE_COLUMNS}
|
||||
costs = [_number(row.get("cost_usd")) for row in rows]
|
||||
seconds = [_number(row.get("seconds")) for row in rows]
|
||||
# Per run, as the brief and the report say it; the totals over every run of
|
||||
# a setup only from a service that sends no per-run figure.
|
||||
costs, cost_header = _per_run(rows, "cost_per_run", "cost_usd", "cost")
|
||||
seconds, time_header = _per_run(rows, "seconds_per_run", "seconds", "time")
|
||||
best = {area: _best(values, max) for area, values in grades.items()}
|
||||
cheapest, fastest = _best(costs, min), _best(seconds, min)
|
||||
|
||||
table = Table(show_edge=False, pad_edge=False)
|
||||
for header in ("model", *GRADE_COLUMNS, "cost", "time"):
|
||||
for header in ("model", *GRADE_COLUMNS, cost_header, time_header):
|
||||
table.add_column(header, justify="left" if header == "model" else "right")
|
||||
for n, row in enumerate(rows):
|
||||
label = Text(str(row.get("label") or row.get("key") or "?"))
|
||||
@@ -1152,6 +1220,14 @@ def _print_comparison(finished: dict[str, Any], url: str | None) -> None:
|
||||
)
|
||||
table.add_row(*cells)
|
||||
console.print(table)
|
||||
# A dash is a model the service could not price, or a setup with no time
|
||||
# recorded — said once, so a `/ run` column with a gap is not a mystery.
|
||||
if any(cost is None for cost in costs):
|
||||
console.print(Text("— in cost: no price is known for that model", style="dim"))
|
||||
if any(took is None for took in seconds):
|
||||
console.print(
|
||||
Text("— in time: no time was recorded for that setup", style="dim")
|
||||
)
|
||||
|
||||
suggestions = _top_suggestions(comparison.get("suggestions"))
|
||||
if suggestions:
|
||||
@@ -1175,6 +1251,26 @@ def _print_comparison(finished: dict[str, Any], url: str | None) -> None:
|
||||
console.print(Text(f"Full report: {url}"))
|
||||
|
||||
|
||||
def _per_run(
|
||||
rows: list[dict[str, Any]], per_run: str, total: str, noun: str
|
||||
) -> tuple[list[float | None], str]:
|
||||
"""One column's values and header: the per-run figure when every row that
|
||||
has a value has one (`noun / run`), else every row's total (`noun`) — never
|
||||
a column that mixes the two. A row with neither (a model the service could
|
||||
not price) does not decide the unit: one unpriced model would otherwise turn
|
||||
the whole column back into totals, which the brief beside it does not use.
|
||||
Its dash is explained under the table."""
|
||||
totals = [_number(row.get(total)) for row in rows]
|
||||
runs = [_number(row.get(per_run)) for row in rows]
|
||||
if any(value is not None for value in runs) and all(
|
||||
run is not None
|
||||
for run, whole in zip(runs, totals, strict=True)
|
||||
if whole is not None
|
||||
):
|
||||
return runs, f"{noun} / run"
|
||||
return totals, noun
|
||||
|
||||
|
||||
def _grade_of(row: dict[str, Any], area: str) -> int | None:
|
||||
grades = row.get("grades")
|
||||
grade = grades.get(area) if isinstance(grades, dict) else None
|
||||
@@ -1338,6 +1434,27 @@ def _refused(response: httpx.Response, subject: str) -> None:
|
||||
_fail(_refusal_message(response, subject))
|
||||
|
||||
|
||||
def _refusal_reason(response: httpx.Response) -> StopReason:
|
||||
"""AMP's refusal by status class, for the usage count."""
|
||||
code = response.status_code
|
||||
if code == 401:
|
||||
return "amp_401"
|
||||
if code == 403:
|
||||
return "amp_403"
|
||||
if code == 404:
|
||||
return "amp_404"
|
||||
if 400 <= code < 500:
|
||||
return "amp_4xx"
|
||||
if code >= 500:
|
||||
return "amp_5xx"
|
||||
return "invalid_response"
|
||||
|
||||
|
||||
def _stop(stopped: EvaluationStoppedError) -> NoReturn:
|
||||
record_stop(stopped.reason)
|
||||
_fail(str(stopped))
|
||||
|
||||
|
||||
def _fail(message: str) -> NoReturn:
|
||||
# `Text`, because most of what reaches here is AMP's own sentence and a
|
||||
# `Console` parses square brackets. No caller relies on markup; the colour
|
||||
|
||||
@@ -6,7 +6,7 @@ from pathlib import Path
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from typing import TYPE_CHECKING, Any, Literal
|
||||
|
||||
import click
|
||||
from crewai_core.constants import CREWAI_TRAINED_AGENTS_FILE_ENV
|
||||
@@ -493,6 +493,30 @@ def _run_json_crew_in_project_env(
|
||||
return None
|
||||
|
||||
|
||||
DeployButtonEvent = Literal[
|
||||
"deploy_login_prompted",
|
||||
"deploy_stopped:login_still_required",
|
||||
"deploy_stopped:login_abandoned",
|
||||
"deploy_stopped:login_failed",
|
||||
]
|
||||
|
||||
|
||||
def _record_deploy_button(event: DeployButtonEvent) -> None:
|
||||
"""Count what the run app's Deploy met before a deployment attempt.
|
||||
|
||||
`cli_usage:deploy` counts the press and `Create Crew Deployment` the
|
||||
attempt; a press that never became an attempt said nothing about why.
|
||||
"""
|
||||
try:
|
||||
from crewai_core.telemetry import Telemetry
|
||||
|
||||
telemetry = Telemetry()
|
||||
telemetry.set_tracer()
|
||||
telemetry.feature_usage_span(f"cli_usage:{event}")
|
||||
except Exception: # noqa: S110 - telemetry must never break a command
|
||||
pass
|
||||
|
||||
|
||||
def _chain_deploy() -> None:
|
||||
from rich.console import Console
|
||||
|
||||
@@ -516,11 +540,20 @@ def _chain_deploy() -> None:
|
||||
except AuthenticationRequiredError:
|
||||
from crewai_cli.authentication.main import AuthenticationCommand
|
||||
|
||||
_record_deploy_button("deploy_login_prompted")
|
||||
console.print()
|
||||
AuthenticationCommand().login()
|
||||
try:
|
||||
AuthenticationCommand().login()
|
||||
except KeyboardInterrupt:
|
||||
_record_deploy_button("deploy_stopped:login_abandoned")
|
||||
raise
|
||||
except Exception:
|
||||
_record_deploy_button("deploy_stopped:login_failed")
|
||||
raise
|
||||
try:
|
||||
DeployCommand().create_crew(confirm=True, skip_validate=True, source="tui")
|
||||
except AuthenticationRequiredError:
|
||||
_record_deploy_button("deploy_stopped:login_still_required")
|
||||
console.print(
|
||||
"\nDeploy failed: authentication is still required.\n",
|
||||
style="bold red",
|
||||
|
||||
@@ -1331,6 +1331,70 @@ def test_models_are_compared_on_the_deployment_and_the_table_printed(deployed, m
|
||||
assert "a fourth" not in out
|
||||
|
||||
|
||||
def test_cost_and_time_are_per_run_when_the_service_sends_them(deployed, monkeypatch, capsys):
|
||||
per_run = comparison()
|
||||
per_run["models"][0].update(cost_per_run=0.0020, seconds_per_run=4.5)
|
||||
per_run["models"][1].update(cost_per_run=0.0004, seconds_per_run=2.5)
|
||||
install(monkeypatch, FakeModelsAMP(statuses=[httpx.Response(
|
||||
200, json={"id": "ev-9", "status": "done", "url": URL, "comparison": per_run})]))
|
||||
|
||||
eval_module.eval_models(MODELS)
|
||||
|
||||
out = capsys.readouterr().out
|
||||
assert "cost / run" in out and "time / run" in out
|
||||
mini = next(line for line in out.splitlines() if "Poem composer: openai/gpt-4o-mini" in line)
|
||||
base = next(line for line in out.splitlines() if "gpt-5.6-sol" in line)
|
||||
assert "$0.0004 ★" in mini and "2.5s ★" in mini
|
||||
assert "$0.0020" in base and "4.5s" in base
|
||||
# the totals are not what the column says
|
||||
assert "$0.0012" not in out and "9.5s" not in out
|
||||
|
||||
|
||||
def test_a_column_never_mixes_per_run_and_totals(deployed, monkeypatch, capsys):
|
||||
partial = comparison()
|
||||
partial["models"][1].update(cost_per_run=0.0004) # one row only: the totals stand
|
||||
install(monkeypatch, FakeModelsAMP(statuses=[httpx.Response(
|
||||
200, json={"id": "ev-9", "status": "done", "url": URL, "comparison": partial})]))
|
||||
|
||||
eval_module.eval_models(MODELS)
|
||||
|
||||
out = capsys.readouterr().out
|
||||
assert "cost / run" not in out and "time / run" not in out
|
||||
assert "$0.0012 ★" in out and "$0.0004" not in out
|
||||
|
||||
|
||||
def test_an_unpriced_model_keeps_the_per_run_unit_and_its_dash_is_explained(deployed, monkeypatch, capsys):
|
||||
unpriced = comparison()
|
||||
unpriced["models"][0].update(cost_per_run=0.0020, seconds_per_run=4.5)
|
||||
unpriced["models"][1].update(cost_usd=None, cost_per_run=None, seconds_per_run=2.5)
|
||||
install(monkeypatch, FakeModelsAMP(statuses=[httpx.Response(
|
||||
200, json={"id": "ev-9", "status": "done", "url": URL, "comparison": unpriced})]))
|
||||
|
||||
eval_module.eval_models(MODELS)
|
||||
|
||||
out = capsys.readouterr().out
|
||||
assert "cost / run" in out and "time / run" in out
|
||||
mini = next(line for line in out.splitlines() if "Poem composer: openai/gpt-4o-mini" in line)
|
||||
assert "—" in mini and "$0.0020" in out
|
||||
assert "— in cost: no price is known for that model" in out
|
||||
assert "— in time" not in out
|
||||
|
||||
|
||||
def test_a_progress_line_is_said_once_even_when_parallel_setups_alternate(deployed, monkeypatch, capsys):
|
||||
def at(message):
|
||||
return httpx.Response(200, json={"id": "ev-9", "status": "running", "progress": {"message": message}})
|
||||
|
||||
alternating = [at("judging the final output"), at("0 of 3 setups done")] * 4
|
||||
install(monkeypatch, FakeModelsAMP(statuses=[*alternating, at("1 of 3 setups done"), compared()]))
|
||||
|
||||
eval_module.eval_models(MODELS)
|
||||
|
||||
out = capsys.readouterr().out
|
||||
assert out.count("judging the final output") == 1
|
||||
assert out.count("0 of 3 setups done") == 1
|
||||
assert out.index("0 of 3 setups done") < out.index("1 of 3 setups done")
|
||||
|
||||
|
||||
def test_the_comparison_is_counted_with_the_models_and_nothing_that_names_it(deployed, monkeypatch):
|
||||
spans: list[tuple[str, dict[str, str]]] = []
|
||||
|
||||
@@ -1456,7 +1520,7 @@ def test_a_model_without_its_provider_exits_one_before_anything_is_sent(deployed
|
||||
assert "provider/model" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_a_failed_comparison_exits_one_with_the_reason(deployed, monkeypatch, capsys):
|
||||
def test_a_failed_comparison_exits_one_with_the_reason(deployed, monkeypatch, capsys, usage):
|
||||
failed = httpx.Response(200, json={"id": "ev-9", "status": "failed", "error": "the deployment answered 500"})
|
||||
install(monkeypatch, FakeModelsAMP(statuses=[failed]))
|
||||
|
||||
@@ -1465,6 +1529,8 @@ def test_a_failed_comparison_exits_one_with_the_reason(deployed, monkeypatch, ca
|
||||
|
||||
assert stopped.value.code == 1
|
||||
assert "Comparison failed: the deployment answered 500" in capsys.readouterr().out
|
||||
# `--models` compares models; it is not the evaluation the stop count is about
|
||||
assert not [feature for feature in usage if "eval_stopped" in feature]
|
||||
|
||||
|
||||
def test_done_without_a_comparison_is_a_protocol_error(deployed, monkeypatch, capsys):
|
||||
@@ -1709,3 +1775,301 @@ def test_the_help_tells_an_agent_it_gets_a_brief():
|
||||
) in help_text
|
||||
assert "That is when AMP provides the brief: otherwise its link, when AMP sends only that, else nothing more" in help_text
|
||||
assert "it prints a markdown brief after the comparison when AMP provides one" in help_text
|
||||
|
||||
|
||||
# --- why an evaluation that was asked for never reached a verdict -----------
|
||||
|
||||
_REAL_SAVED_LOGIN = eval_module.saved_login
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def usage(monkeypatch):
|
||||
"""Every usage feature counted, in order."""
|
||||
features: list[str] = []
|
||||
|
||||
class FakeTelemetry:
|
||||
def set_tracer(self) -> None:
|
||||
pass
|
||||
|
||||
def feature_usage_span(self, feature, attributes=None) -> None:
|
||||
features.append(feature)
|
||||
|
||||
monkeypatch.setattr("crewai_core.telemetry.Telemetry", FakeTelemetry)
|
||||
return features
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("status", "reason"),
|
||||
[(401, "amp_401"), (403, "amp_403"), (404, "amp_404"), (422, "amp_4xx"), (503, "amp_5xx")],
|
||||
)
|
||||
def test_a_refused_start_counts_why_and_says_what_it_always_said(
|
||||
project, monkeypatch, capsys, usage, status, reason
|
||||
):
|
||||
directory, _ = project
|
||||
record_last_run(directory) # not a run that just ended: a 404 is not waited out
|
||||
refusal = httpx.Response(status, json={"detail": "refused for a reason AMP states"})
|
||||
install(monkeypatch, FakeAMP(create=refusal))
|
||||
|
||||
with pytest.raises(SystemExit) as exited:
|
||||
eval_module.eval_crew()
|
||||
|
||||
assert exited.value.code == 1
|
||||
assert usage == [f"cli_usage:eval_stopped:{reason}"]
|
||||
expected = eval_module._refusal_message(refusal, f"run {EXECUTION_ID}")
|
||||
assert expected in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_an_unreachable_amp_at_the_start_counts_as_a_network_error(project, monkeypatch, usage):
|
||||
directory, _ = project
|
||||
record_last_run(directory)
|
||||
|
||||
class Unreachable(FakeAMP):
|
||||
def create_evaluation(self, execution_id, *, eval_config=None, project_id=None):
|
||||
raise httpx.ConnectError("no route to host")
|
||||
|
||||
install(monkeypatch, Unreachable())
|
||||
|
||||
with pytest.raises(SystemExit, match="1"):
|
||||
eval_module.eval_crew()
|
||||
|
||||
assert usage == ["cli_usage:eval_stopped:network_error"]
|
||||
|
||||
|
||||
def test_a_start_without_an_evaluation_id_counts_as_an_invalid_response(project, monkeypatch, usage):
|
||||
directory, _ = project
|
||||
record_last_run(directory)
|
||||
install(monkeypatch, FakeAMP(create=httpx.Response(202, json={"status": "queued"})))
|
||||
|
||||
with pytest.raises(SystemExit, match="1"):
|
||||
eval_module.eval_crew()
|
||||
|
||||
assert usage == ["cli_usage:eval_stopped:invalid_response"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("answer", "reason"),
|
||||
[
|
||||
(httpx.Response(200, json={"id": "ev-1", "status": "mystery"}), "invalid_response"),
|
||||
(httpx.Response(200, json={"id": "ev-1", "status": "done"}), "invalid_response"),
|
||||
(httpx.Response(403, json={}), "amp_403"),
|
||||
],
|
||||
)
|
||||
def test_an_evaluation_that_stops_while_waiting_was_counted_and_says_why(
|
||||
project, monkeypatch, usage, answer, reason
|
||||
):
|
||||
"""It started, so `cli_usage:eval` counted it; the stop is the second half."""
|
||||
directory, _ = project
|
||||
record_last_run(directory)
|
||||
install(monkeypatch, FakeAMP(statuses=[answer]))
|
||||
|
||||
with pytest.raises(SystemExit, match="1"):
|
||||
eval_module.eval_crew()
|
||||
|
||||
assert usage == ["cli_usage:eval", f"cli_usage:eval_stopped:{reason}"]
|
||||
|
||||
|
||||
def test_losing_amp_while_waiting_counts_as_a_network_error(project, monkeypatch, usage):
|
||||
directory, _ = project
|
||||
record_last_run(directory)
|
||||
|
||||
class GoneQuiet(FakeAMP):
|
||||
def get_evaluation(self, evaluation_id):
|
||||
raise httpx.ReadTimeout("no answer")
|
||||
|
||||
install(monkeypatch, GoneQuiet())
|
||||
|
||||
with pytest.raises(SystemExit, match="1"):
|
||||
eval_module.eval_crew()
|
||||
|
||||
assert usage == ["cli_usage:eval", "cli_usage:eval_stopped:network_error"]
|
||||
|
||||
|
||||
def test_a_saved_login_that_cannot_be_read_counts_as_such(project, monkeypatch, usage):
|
||||
directory, _ = project
|
||||
record_last_run(directory)
|
||||
monkeypatch.setattr(eval_module, "saved_login", _REAL_SAVED_LOGIN)
|
||||
|
||||
def broken_store():
|
||||
raise OSError("permission denied")
|
||||
|
||||
monkeypatch.setattr(eval_module, "get_auth_token", broken_store)
|
||||
install(monkeypatch, FakeAMP())
|
||||
|
||||
with pytest.raises(SystemExit, match="1"):
|
||||
eval_module.eval_crew()
|
||||
|
||||
assert usage == ["cli_usage:eval_stopped:login_unreadable"]
|
||||
|
||||
|
||||
def test_an_evaluation_amp_failed_is_counted_and_says_what_it_always_said(
|
||||
project, monkeypatch, capsys, usage
|
||||
):
|
||||
directory, _ = project
|
||||
record_last_run(directory)
|
||||
failed = httpx.Response(200, json={"id": "ev-1", "status": "failed", "error": "the judge was unreachable"})
|
||||
install(monkeypatch, FakeAMP(statuses=[failed]))
|
||||
|
||||
with pytest.raises(SystemExit) as exited:
|
||||
eval_module.eval_crew()
|
||||
|
||||
assert exited.value.code == 1
|
||||
assert usage == ["cli_usage:eval", "cli_usage:eval_stopped:evaluation_failed"]
|
||||
assert "Evaluation failed: the judge was unreachable" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_an_evaluation_amp_failed_in_the_run_app_is_counted_and_still_returned(
|
||||
project, monkeypatch, usage
|
||||
):
|
||||
failed = httpx.Response(200, json={"id": "ev-1", "status": "failed", "error": "the judge was unreachable"})
|
||||
install(monkeypatch, FakeAMP(statuses=[failed]))
|
||||
|
||||
finished = eval_module.evaluate_run(EXECUTION_ID, on_started=lambda started: None)
|
||||
|
||||
assert finished["status"] == "failed" # the app still shows AMP's reason
|
||||
assert usage == ["cli_usage:eval", "cli_usage:eval_stopped:evaluation_failed"]
|
||||
|
||||
|
||||
def test_an_unreadable_login_with_nobody_to_ask_counts_as_such(
|
||||
project, monkeypatch, capsys, usage
|
||||
):
|
||||
"""Tracing on, no terminal, and a saved login that cannot be read: the login
|
||||
is the reason shown, so it is the reason counted — not "untraced"."""
|
||||
directory, _ = project
|
||||
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
|
||||
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: False)
|
||||
monkeypatch.setenv("CREWAI_TRACING_ENABLED", "true")
|
||||
monkeypatch.setattr(eval_module, "saved_login", _REAL_SAVED_LOGIN)
|
||||
|
||||
def broken_store():
|
||||
raise OSError("permission denied")
|
||||
|
||||
monkeypatch.setattr(eval_module, "get_auth_token", broken_store)
|
||||
|
||||
with pytest.raises(SystemExit, match="1"):
|
||||
eval_module.eval_crew()
|
||||
|
||||
assert usage == ["cli_usage:eval_stopped:login_unreadable"]
|
||||
assert "crewai login" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_a_start_the_run_app_was_refused_is_not_tried_again_by_the_command(
|
||||
project, monkeypatch, usage
|
||||
):
|
||||
"""`crewai eval` runs the crew, the app it opens tries the evaluation and AMP
|
||||
refuses the start. The app has shown the refusal and counted it once; the
|
||||
command waiting behind it must not start the same run again."""
|
||||
import contextlib
|
||||
|
||||
from crewai_cli.crew_run_tui import CrewRunApp
|
||||
|
||||
directory, _ = project
|
||||
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
|
||||
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
|
||||
monkeypatch.setattr(eval_module.click, "confirm", lambda *args, **kwargs: True)
|
||||
monkeypatch.setattr(eval_module, "_enable_tracing", lambda: None)
|
||||
amp = install(monkeypatch, FakeAMP(create=httpx.Response(401, json={"detail": "Bad credentials"})))
|
||||
monkeypatch.setattr(
|
||||
CrewRunApp, "call_from_thread", lambda self, handler, *args: handler(*args)
|
||||
)
|
||||
shown: list[str] = []
|
||||
monkeypatch.setattr(CrewRunApp, "_evaluation_failed", lambda self, message: shown.append(message))
|
||||
holder: dict[str, str | None] = {"execution_id": None}
|
||||
|
||||
@contextlib.contextmanager
|
||||
def watched():
|
||||
yield holder
|
||||
|
||||
def the_app_runs_and_evaluates() -> None:
|
||||
holder["execution_id"] = EXECUTION_ID # the app saw the run traced
|
||||
CrewRunApp()._evaluate_now(EXECUTION_ID)
|
||||
|
||||
monkeypatch.setattr("crewai_cli.crew_run_tui.evaluating_after_run", watched)
|
||||
monkeypatch.setattr(run_crew_module, "run_crew", the_app_runs_and_evaluates)
|
||||
|
||||
eval_module.eval_crew()
|
||||
|
||||
assert [call for call in amp.calls if call[0] == "create"] == [("create", EXECUTION_ID)]
|
||||
assert usage == ["cli_usage:eval_stopped:amp_401"]
|
||||
assert len(shown) == 1
|
||||
|
||||
|
||||
def test_outside_a_project_counts_as_no_project(project, usage):
|
||||
with pytest.raises(SystemExit, match="1"):
|
||||
eval_module.eval_crew()
|
||||
|
||||
assert usage == ["cli_usage:eval_stopped:no_project"]
|
||||
|
||||
|
||||
def test_nothing_traced_with_nobody_to_ask_counts_as_untraced(project, monkeypatch, usage):
|
||||
directory, _ = project
|
||||
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
|
||||
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: False)
|
||||
|
||||
with pytest.raises(SystemExit, match="1"):
|
||||
eval_module.eval_crew()
|
||||
|
||||
assert usage == ["cli_usage:eval_stopped:untraced"]
|
||||
|
||||
|
||||
def test_declining_to_run_with_tracing_on_counts_as_declined(project, monkeypatch, usage):
|
||||
directory, _ = project
|
||||
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
|
||||
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
|
||||
monkeypatch.setattr(eval_module.click, "confirm", lambda *args, **kwargs: False)
|
||||
|
||||
with pytest.raises(SystemExit) as exited:
|
||||
eval_module.eval_crew()
|
||||
|
||||
assert exited.value.code == 0
|
||||
assert usage == ["cli_usage:eval_stopped:declined"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("app_saw_the_run", [True, False])
|
||||
def test_a_run_that_left_no_trace_counts_as_untraced_once(
|
||||
project, monkeypatch, usage, app_saw_the_run
|
||||
):
|
||||
"""Whether the run app looked and found nothing, or no app got to the run,
|
||||
the command is the one that counts it; the app counts nothing here."""
|
||||
import contextlib
|
||||
|
||||
directory, _ = project
|
||||
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
|
||||
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
|
||||
monkeypatch.setattr(eval_module.click, "confirm", lambda *args, **kwargs: True)
|
||||
monkeypatch.setattr(eval_module, "_enable_tracing", lambda: None)
|
||||
monkeypatch.setattr(run_crew_module, "run_crew", lambda: None)
|
||||
|
||||
@contextlib.contextmanager
|
||||
def watched():
|
||||
yield {"execution_id": None}
|
||||
|
||||
monkeypatch.setattr("crewai_cli.crew_run_tui.evaluating_after_run", watched)
|
||||
monkeypatch.setattr(
|
||||
eval_module,
|
||||
"evaluation_marker",
|
||||
lambda after: {"execution_id": None} if app_saw_the_run else None,
|
||||
)
|
||||
|
||||
with pytest.raises(SystemExit, match="1"):
|
||||
eval_module.eval_crew()
|
||||
|
||||
assert usage == ["cli_usage:eval_stopped:untraced"]
|
||||
|
||||
|
||||
def test_a_failing_usage_count_never_changes_what_the_command_does(project, monkeypatch, capsys):
|
||||
directory, _ = project
|
||||
record_last_run(directory)
|
||||
|
||||
class BrokenTelemetry:
|
||||
def set_tracer(self) -> None:
|
||||
raise RuntimeError("exporter is down")
|
||||
|
||||
monkeypatch.setattr("crewai_core.telemetry.Telemetry", BrokenTelemetry)
|
||||
refusal = httpx.Response(403, json={"detail": "not yours"})
|
||||
install(monkeypatch, FakeAMP(create=refusal))
|
||||
|
||||
with pytest.raises(SystemExit) as exited:
|
||||
eval_module.eval_crew()
|
||||
|
||||
assert exited.value.code == 1
|
||||
assert eval_module._refusal_message(refusal, f"run {EXECUTION_ID}") in capsys.readouterr().out
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import contextvars
|
||||
from datetime import datetime
|
||||
from datetime import datetime, timezone
|
||||
import time
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import Mock
|
||||
@@ -43,6 +43,7 @@ from crewai_cli.crew_run_tui import (
|
||||
_format_json_in_text,
|
||||
_try_parse_structured,
|
||||
)
|
||||
import httpx
|
||||
import pytest
|
||||
from rich.text import Text
|
||||
|
||||
@@ -2231,3 +2232,229 @@ def test_format_json_in_text_still_pretty_prints_valid_json() -> None:
|
||||
assert _format_json_in_text('data: {"a": 1} and [...]') == (
|
||||
"data: " + '{\n "a": 1\n}' + " and [...]"
|
||||
)
|
||||
|
||||
|
||||
# --- what the run app's Deploy met before a deployment attempt ---------------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def deploy_usage(monkeypatch) -> list[str]:
|
||||
features: list[str] = []
|
||||
|
||||
class FakeTelemetry:
|
||||
def set_tracer(self) -> None:
|
||||
pass
|
||||
|
||||
def feature_usage_span(self, feature, attributes=None) -> None:
|
||||
features.append(feature)
|
||||
|
||||
monkeypatch.setattr("crewai_core.telemetry.Telemetry", FakeTelemetry)
|
||||
return features
|
||||
|
||||
|
||||
def _deploy_doubles(monkeypatch, *, creates, login):
|
||||
calls: list[str] = []
|
||||
|
||||
class FakeDeployCommand:
|
||||
def create_crew(self, **kwargs) -> None:
|
||||
calls.append("create")
|
||||
outcome = creates.pop(0)
|
||||
if outcome is not None:
|
||||
raise outcome
|
||||
|
||||
class FakeAuthenticationCommand:
|
||||
def login(self) -> None:
|
||||
calls.append("login")
|
||||
if login is not None:
|
||||
raise login
|
||||
|
||||
monkeypatch.setattr("crewai_cli.deploy.main.DeployCommand", FakeDeployCommand)
|
||||
monkeypatch.setattr(
|
||||
"crewai_cli.authentication.main.AuthenticationCommand",
|
||||
FakeAuthenticationCommand,
|
||||
)
|
||||
return calls
|
||||
|
||||
|
||||
def test_a_logged_in_deploy_counts_no_hurdle(monkeypatch, deploy_usage) -> None:
|
||||
calls = _deploy_doubles(monkeypatch, creates=[None], login=None)
|
||||
|
||||
run_crew._chain_deploy()
|
||||
|
||||
assert calls == ["create"]
|
||||
assert deploy_usage == []
|
||||
|
||||
|
||||
def test_a_login_prompt_that_succeeds_counts_the_prompt_only(monkeypatch, deploy_usage) -> None:
|
||||
calls = _deploy_doubles(
|
||||
monkeypatch, creates=[AuthenticationRequiredError(), None], login=None
|
||||
)
|
||||
|
||||
run_crew._chain_deploy()
|
||||
|
||||
assert calls == ["create", "login", "create"]
|
||||
assert deploy_usage == ["cli_usage:deploy_login_prompted"]
|
||||
|
||||
|
||||
def test_still_not_logged_in_after_the_prompt_counts_why_and_says_so(
|
||||
monkeypatch, capsys, deploy_usage
|
||||
) -> None:
|
||||
_deploy_doubles(
|
||||
monkeypatch,
|
||||
creates=[AuthenticationRequiredError(), AuthenticationRequiredError()],
|
||||
login=None,
|
||||
)
|
||||
|
||||
run_crew._chain_deploy()
|
||||
|
||||
assert deploy_usage == [
|
||||
"cli_usage:deploy_login_prompted",
|
||||
"cli_usage:deploy_stopped:login_still_required",
|
||||
]
|
||||
assert "Deploy failed: authentication is still required." in capsys.readouterr().out
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("raised", "reason"),
|
||||
[(KeyboardInterrupt(), "login_abandoned"), (httpx.HTTPError("access_denied"), "login_failed")],
|
||||
)
|
||||
def test_a_login_that_ends_badly_is_counted_and_still_raised(
|
||||
monkeypatch, deploy_usage, raised, reason
|
||||
) -> None:
|
||||
"""Counting must not change what the user sees: the error still escapes."""
|
||||
calls = _deploy_doubles(
|
||||
monkeypatch, creates=[AuthenticationRequiredError()], login=raised
|
||||
)
|
||||
|
||||
with pytest.raises(type(raised)):
|
||||
run_crew._chain_deploy()
|
||||
|
||||
assert calls == ["create", "login"]
|
||||
assert deploy_usage == [
|
||||
"cli_usage:deploy_login_prompted",
|
||||
f"cli_usage:deploy_stopped:{reason}",
|
||||
]
|
||||
|
||||
|
||||
def test_a_failing_usage_count_never_changes_the_deploy(monkeypatch, capsys) -> None:
|
||||
class BrokenTelemetry:
|
||||
def set_tracer(self) -> None:
|
||||
raise RuntimeError("exporter is down")
|
||||
|
||||
monkeypatch.setattr("crewai_core.telemetry.Telemetry", BrokenTelemetry)
|
||||
calls = _deploy_doubles(
|
||||
monkeypatch,
|
||||
creates=[AuthenticationRequiredError(), AuthenticationRequiredError()],
|
||||
login=None,
|
||||
)
|
||||
|
||||
run_crew._chain_deploy()
|
||||
|
||||
assert calls == ["create", "login", "create"]
|
||||
assert "Deploy failed: authentication is still required." in capsys.readouterr().out
|
||||
|
||||
|
||||
# --- why an evaluation asked for on this screen never reached a verdict ------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def stops(monkeypatch) -> list[str]:
|
||||
"""The `cli_usage:eval_stopped:*` features counted, in order."""
|
||||
features: list[str] = []
|
||||
|
||||
class FakeTelemetry:
|
||||
def set_tracer(self) -> None:
|
||||
pass
|
||||
|
||||
def feature_usage_span(self, feature, attributes=None) -> None:
|
||||
features.append(feature)
|
||||
|
||||
monkeypatch.setattr("crewai_core.telemetry.Telemetry", FakeTelemetry)
|
||||
return features
|
||||
|
||||
|
||||
def test_pressing_evaluate_on_an_untraced_run_counts_the_press_and_why_it_stopped(
|
||||
monkeypatch, stops
|
||||
) -> None:
|
||||
app = CrewRunApp()
|
||||
app._status = "completed"
|
||||
app._telemetry = Mock()
|
||||
app._execution_uuid = "run-this-app"
|
||||
monkeypatch.setattr(crew_run_tui, "_trace_was_recorded", lambda uuid: False)
|
||||
monkeypatch.setattr(CrewRunApp, "notify", lambda self, *a, **k: None)
|
||||
|
||||
app.action_evaluate_crew()
|
||||
|
||||
app._telemetry.feature_usage_span.assert_called_once_with("cli_usage:evaluate")
|
||||
assert stops == ["cli_usage:eval_stopped:untraced"]
|
||||
|
||||
|
||||
def test_a_run_started_by_crewai_eval_leaves_the_untraced_count_to_the_command(
|
||||
monkeypatch, stops
|
||||
) -> None:
|
||||
"""The command waiting behind this screen says "not traced" and counts it;
|
||||
counting it here as well would count one stop twice."""
|
||||
monkeypatch.setattr(crew_run_tui, "_trace_was_recorded", lambda uuid: False)
|
||||
monkeypatch.setattr(CrewRunApp, "notify", lambda self, *a, **k: None)
|
||||
with crew_run_tui.evaluating_after_run():
|
||||
app = CrewRunApp()
|
||||
app._execution_uuid = "run-this-app"
|
||||
app._evaluate_if_one_is_waiting()
|
||||
|
||||
assert stops == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("raised", "reason"),
|
||||
[
|
||||
("amp_403", "amp_403"),
|
||||
(SystemExit(1), "unexpected"),
|
||||
(RuntimeError("client bug"), "unexpected"),
|
||||
],
|
||||
)
|
||||
def test_an_evaluation_that_stops_on_this_screen_counts_why_and_answers_for_the_run(
|
||||
monkeypatch, tmp_path, stops, raised, reason
|
||||
) -> None:
|
||||
"""The marker tells a `crewai eval` waiting behind this screen that the run
|
||||
was answered for, so it neither starts it again nor counts the stop twice."""
|
||||
from crewai_cli.experimental.eval_crew import (
|
||||
EvaluationStoppedError,
|
||||
evaluation_marker,
|
||||
)
|
||||
|
||||
if isinstance(raised, str):
|
||||
raised = EvaluationStoppedError("AMP answered 403.", reason=raised)
|
||||
app = CrewRunApp()
|
||||
app._evaluation = {"state": "starting"}
|
||||
monkeypatch.setattr(
|
||||
CrewRunApp, "call_from_thread", lambda self, handler, *args: handler(*args)
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"crewai_cli.experimental.eval_crew.evaluate_run", Mock(side_effect=raised)
|
||||
)
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
began = datetime.now(timezone.utc)
|
||||
|
||||
app._evaluate_now("run-this-app")
|
||||
|
||||
assert app._evaluation["state"] == "failed"
|
||||
assert stops == [f"cli_usage:eval_stopped:{reason}"]
|
||||
marker = evaluation_marker(after=began)
|
||||
assert marker is not None and marker["execution_id"] == "run-this-app"
|
||||
|
||||
|
||||
def test_an_evaluation_that_finishes_counts_no_stop(monkeypatch, stops) -> None:
|
||||
app = CrewRunApp()
|
||||
app._evaluation = {"state": "starting"}
|
||||
monkeypatch.setattr(
|
||||
CrewRunApp, "call_from_thread", lambda self, handler, *args: handler(*args)
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"crewai_cli.experimental.eval_crew.evaluate_run",
|
||||
Mock(return_value={"status": "done", "verdict": {"gate": "passed"}}),
|
||||
)
|
||||
|
||||
app._evaluate_now("run-this-app")
|
||||
|
||||
assert stops == []
|
||||
|
||||
@@ -30,6 +30,7 @@ dependencies = [
|
||||
# Authentication and Security
|
||||
"python-dotenv>=1.2.2,<2",
|
||||
"pyjwt>=2.15.0,<3",
|
||||
"oauthlib>=4.0.0,<5",
|
||||
# Configuration and Utils
|
||||
"click>=8.1.7,<9",
|
||||
"appdirs~=1.4.4",
|
||||
|
||||
@@ -8,6 +8,7 @@ import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
from time import monotonic, sleep
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from crewai_core.lock_store import lock as store_lock
|
||||
@@ -22,6 +23,10 @@ if TYPE_CHECKING:
|
||||
from crewai.flow.async_feedback.types import PendingFeedbackContext
|
||||
|
||||
|
||||
_SQLITE_BUSY_TIMEOUT_SECONDS = 30.0
|
||||
_SCHEMA_RETRY_DELAY_SECONDS = 0.05
|
||||
|
||||
|
||||
def _json_default(obj: Any) -> Any:
|
||||
"""Fallback serializer for non-primitive types in JSON dumps."""
|
||||
if isinstance(obj, BaseModel):
|
||||
@@ -85,9 +90,21 @@ class SQLiteFlowPersistence(FlowPersistence):
|
||||
|
||||
def init_db(self) -> None:
|
||||
"""Create the necessary tables if they don't exist."""
|
||||
deadline = monotonic() + _SQLITE_BUSY_TIMEOUT_SECONDS
|
||||
while True:
|
||||
try:
|
||||
self._create_schema()
|
||||
return
|
||||
except sqlite3.OperationalError as exc: # noqa: PERF203
|
||||
if "locked" not in str(exc).lower() or monotonic() >= deadline:
|
||||
raise
|
||||
sleep(_SCHEMA_RETRY_DELAY_SECONDS)
|
||||
|
||||
def _create_schema(self) -> None:
|
||||
with (
|
||||
store_lock(self._lock_name),
|
||||
closing(sqlite3.connect(self.db_path, timeout=30)) as conn,
|
||||
closing(
|
||||
sqlite3.connect(self.db_path, timeout=_SQLITE_BUSY_TIMEOUT_SECONDS)
|
||||
) as conn,
|
||||
conn,
|
||||
):
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
@@ -190,7 +207,9 @@ class SQLiteFlowPersistence(FlowPersistence):
|
||||
|
||||
with (
|
||||
store_lock(self._lock_name),
|
||||
closing(sqlite3.connect(self.db_path, timeout=30)) as conn,
|
||||
closing(
|
||||
sqlite3.connect(self.db_path, timeout=_SQLITE_BUSY_TIMEOUT_SECONDS)
|
||||
) as conn,
|
||||
conn,
|
||||
):
|
||||
self._save_state_sql(conn, flow_uuid, method_name, state_dict)
|
||||
@@ -204,7 +223,12 @@ class SQLiteFlowPersistence(FlowPersistence):
|
||||
Returns:
|
||||
The most recent state as a dictionary, or None if no state exists
|
||||
"""
|
||||
with closing(sqlite3.connect(self.db_path, timeout=30)) as conn, conn:
|
||||
with (
|
||||
closing(
|
||||
sqlite3.connect(self.db_path, timeout=_SQLITE_BUSY_TIMEOUT_SECONDS)
|
||||
) as conn,
|
||||
conn,
|
||||
):
|
||||
cursor = conn.execute(
|
||||
"""
|
||||
SELECT state_json
|
||||
@@ -242,7 +266,9 @@ class SQLiteFlowPersistence(FlowPersistence):
|
||||
|
||||
with (
|
||||
store_lock(self._lock_name),
|
||||
closing(sqlite3.connect(self.db_path, timeout=30)) as conn,
|
||||
closing(
|
||||
sqlite3.connect(self.db_path, timeout=_SQLITE_BUSY_TIMEOUT_SECONDS)
|
||||
) as conn,
|
||||
conn,
|
||||
):
|
||||
self._save_state_sql(conn, flow_uuid, context.method_name, state_dict)
|
||||
@@ -280,7 +306,12 @@ class SQLiteFlowPersistence(FlowPersistence):
|
||||
# Import here to avoid circular imports
|
||||
from crewai.flow.async_feedback.types import PendingFeedbackContext
|
||||
|
||||
with closing(sqlite3.connect(self.db_path, timeout=30)) as conn, conn:
|
||||
with (
|
||||
closing(
|
||||
sqlite3.connect(self.db_path, timeout=_SQLITE_BUSY_TIMEOUT_SECONDS)
|
||||
) as conn,
|
||||
conn,
|
||||
):
|
||||
cursor = conn.execute(
|
||||
"""
|
||||
SELECT state_json, context_json
|
||||
@@ -306,7 +337,9 @@ class SQLiteFlowPersistence(FlowPersistence):
|
||||
"""
|
||||
with (
|
||||
store_lock(self._lock_name),
|
||||
closing(sqlite3.connect(self.db_path, timeout=30)) as conn,
|
||||
closing(
|
||||
sqlite3.connect(self.db_path, timeout=_SQLITE_BUSY_TIMEOUT_SECONDS)
|
||||
) as conn,
|
||||
conn,
|
||||
):
|
||||
conn.execute(
|
||||
|
||||
@@ -197,6 +197,7 @@ SUPPORTED_NATIVE_PROVIDERS: Final[list[str]] = [
|
||||
"cerebras",
|
||||
"dashscope",
|
||||
"snowflake",
|
||||
"deepinfra",
|
||||
]
|
||||
|
||||
|
||||
@@ -221,6 +222,7 @@ PROVIDER_ALIASES: Final[dict[str, str]] = {
|
||||
"cerebras": "cerebras",
|
||||
"dashscope": "dashscope",
|
||||
"snowflake": "snowflake",
|
||||
"deepinfra": "deepinfra",
|
||||
}
|
||||
|
||||
|
||||
@@ -444,6 +446,12 @@ class LLM(BaseLLM):
|
||||
# OpenRouter uses org/model format but accepts anything
|
||||
return True
|
||||
|
||||
if provider == "deepinfra":
|
||||
# DeepInfra ids are exactly org/model, so a full reference is
|
||||
# deepinfra/<org>/<model>: one slash with both parts non-empty.
|
||||
parts = model_lower.split("/")
|
||||
return len(parts) == 2 and all(parts)
|
||||
|
||||
if provider == "snowflake":
|
||||
return True
|
||||
|
||||
@@ -458,7 +466,8 @@ class LLM(BaseLLM):
|
||||
a ``<prefix>/<model>`` string goes native when the prefix is a native
|
||||
provider and the model is one it knows (or, for ``openai/``, when a
|
||||
custom endpoint is configured); a bare model name infers its provider.
|
||||
``native_class`` is ``None`` when the call falls back to LiteLLM.
|
||||
``native_class`` is ``None`` when the call falls back to LiteLLM; a
|
||||
DeepInfra id that is not ``org/model`` raises instead of falling through.
|
||||
``kwargs`` is read, never mutated.
|
||||
"""
|
||||
custom_openai = bool(kwargs.get("custom_openai", False))
|
||||
@@ -481,10 +490,16 @@ class LLM(BaseLLM):
|
||||
provider = explicit_provider
|
||||
use_native = True
|
||||
model_string = model
|
||||
if provider == "deepinfra":
|
||||
cls._require_deepinfra_model(model)
|
||||
elif "/" in model:
|
||||
prefix, _, model_part = model.partition("/")
|
||||
|
||||
canonical_provider = PROVIDER_ALIASES.get(prefix.lower())
|
||||
if canonical_provider == "deepinfra":
|
||||
# Raise here rather than fall through to LiteLLM, where the same
|
||||
# malformed id fails later and differently per environment.
|
||||
cls._require_deepinfra_model(model_part, prefix=f"{prefix}/")
|
||||
|
||||
valid_native_model = bool(
|
||||
canonical_provider
|
||||
@@ -514,6 +529,20 @@ class LLM(BaseLLM):
|
||||
native_class = None
|
||||
return _Route(provider, model_string, native_class, custom_openai_route)
|
||||
|
||||
@classmethod
|
||||
def _require_deepinfra_model(cls, model: str, prefix: str = "") -> None:
|
||||
"""Raise unless ``model`` is the ``org/model`` form DeepInfra serves.
|
||||
|
||||
``prefix`` is the ``deepinfra/`` the caller wrote before ``model``, if
|
||||
any, so the example and the echoed value match what they passed.
|
||||
"""
|
||||
if cls._matches_provider_pattern(model, "deepinfra"):
|
||||
return
|
||||
raise ValueError(
|
||||
"DeepInfra model ids are org/model, for example "
|
||||
f"'{prefix}deepseek-ai/DeepSeek-V4-Flash-0731'; got '{prefix}{model}'"
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def _validate_model_in_constants(cls, model: str, provider: str) -> bool:
|
||||
"""Validate if a model name exists in the provider's constants or matches provider patterns.
|
||||
@@ -650,6 +679,7 @@ class LLM(BaseLLM):
|
||||
"hosted_vllm",
|
||||
"cerebras",
|
||||
"dashscope",
|
||||
"deepinfra",
|
||||
}
|
||||
if provider in openai_compatible_providers:
|
||||
from crewai.llms.providers.openai_compatible.completion import (
|
||||
|
||||
@@ -90,6 +90,12 @@ OPENAI_COMPATIBLE_PROVIDERS: dict[str, ProviderConfig] = {
|
||||
base_url_env="DASHSCOPE_BASE_URL",
|
||||
api_key_required=True,
|
||||
),
|
||||
"deepinfra": ProviderConfig(
|
||||
base_url="https://api.deepinfra.com/v1/openai",
|
||||
api_key_env="DEEPINFRA_API_KEY",
|
||||
base_url_env="DEEPINFRA_BASE_URL",
|
||||
api_key_required=True,
|
||||
),
|
||||
}
|
||||
|
||||
_OLLAMA_DEFAULT_PORT = 11434
|
||||
@@ -142,6 +148,7 @@ class OpenAICompatibleCompletion(OpenAICompletion):
|
||||
- hosted_vllm: vLLM server (https://github.com/vllm-project/vllm)
|
||||
- cerebras: Cerebras (https://cerebras.ai)
|
||||
- dashscope: Alibaba Dashscope/Qwen (https://dashscope.aliyun.com)
|
||||
- deepinfra: DeepInfra (https://deepinfra.com)
|
||||
|
||||
Example:
|
||||
# Using provider prefix
|
||||
|
||||
@@ -95,6 +95,14 @@ class TestProviderRegistry:
|
||||
assert config.api_key_env == "DASHSCOPE_API_KEY"
|
||||
assert config.api_key_required is True
|
||||
|
||||
def test_deepinfra_config(self):
|
||||
"""Test DeepInfra provider configuration."""
|
||||
config = OPENAI_COMPATIBLE_PROVIDERS["deepinfra"]
|
||||
assert config.base_url == "https://api.deepinfra.com/v1/openai"
|
||||
assert config.api_key_env == "DEEPINFRA_API_KEY"
|
||||
assert config.base_url_env == "DEEPINFRA_BASE_URL"
|
||||
assert config.api_key_required is True
|
||||
|
||||
|
||||
class TestNormalizeOllamaBaseUrl:
|
||||
"""Tests for _normalize_ollama_base_url helper."""
|
||||
@@ -162,6 +170,15 @@ class TestOpenAICompatibleCompletion:
|
||||
if original is not None:
|
||||
os.environ[env_key] = original
|
||||
|
||||
def test_deepinfra_missing_api_key_names_env_var(self):
|
||||
"""DeepInfra requires a key; the error tells the user which env var to set."""
|
||||
with patch.dict(os.environ, {}, clear=True), pytest.raises(
|
||||
ValueError, match="DEEPINFRA_API_KEY"
|
||||
):
|
||||
OpenAICompatibleCompletion(
|
||||
model="deepseek-ai/DeepSeek-V4-Flash-0731", provider="deepinfra"
|
||||
)
|
||||
|
||||
def test_api_key_from_env(self):
|
||||
"""Test API key is read from environment variable."""
|
||||
with patch.dict(os.environ, {"DEEPSEEK_API_KEY": "test-key-from-env"}):
|
||||
@@ -312,6 +329,87 @@ class TestLLMIntegration:
|
||||
assert llm.provider == "dashscope"
|
||||
assert llm.base_url == "https://my-dashscope.example.com/v1"
|
||||
|
||||
def test_llm_creates_openai_compatible_for_deepinfra(self):
|
||||
"""Test LLM factory creates OpenAICompatibleCompletion for DeepInfra."""
|
||||
with patch.dict(os.environ, {"DEEPINFRA_API_KEY": "test-key"}):
|
||||
llm = LLM(model="deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731")
|
||||
assert isinstance(llm, OpenAICompatibleCompletion)
|
||||
assert llm.provider == "deepinfra"
|
||||
# DeepInfra ids are org/model, so the part after the provider
|
||||
# prefix is itself two segments and must reach the API intact.
|
||||
assert llm.model == "deepseek-ai/DeepSeek-V4-Flash-0731"
|
||||
assert llm.base_url == "https://api.deepinfra.com/v1/openai"
|
||||
|
||||
def test_deepinfra_base_url_env_override(self):
|
||||
"""DEEPINFRA_BASE_URL redirects DeepInfra to a proxy or private endpoint."""
|
||||
with patch.dict(
|
||||
os.environ,
|
||||
{
|
||||
"DEEPINFRA_API_KEY": "test-key",
|
||||
"DEEPINFRA_BASE_URL": "https://proxy.example.com/v1/openai",
|
||||
},
|
||||
):
|
||||
llm = LLM(model="deepinfra/meta-llama/Llama-4-Scout-17B-16E-Instruct")
|
||||
assert isinstance(llm, OpenAICompatibleCompletion)
|
||||
assert llm.base_url == "https://proxy.example.com/v1/openai"
|
||||
|
||||
def test_llm_creates_openai_compatible_for_deepinfra_explicit_provider(self):
|
||||
"""Explicit provider="deepinfra" routes natively without the model prefix."""
|
||||
with patch.dict(os.environ, {"DEEPINFRA_API_KEY": "test-key"}):
|
||||
llm = LLM(model="deepseek-ai/DeepSeek-V4-Flash-0731", provider="deepinfra")
|
||||
assert isinstance(llm, OpenAICompatibleCompletion)
|
||||
assert llm.provider == "deepinfra"
|
||||
assert llm.model == "deepseek-ai/DeepSeek-V4-Flash-0731"
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"model",
|
||||
[
|
||||
"deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731",
|
||||
"DeepSeek-V4-Flash-0731",
|
||||
"deepseek-ai/DeepSeek-V4-Flash-0731/",
|
||||
],
|
||||
)
|
||||
def test_llm_explicit_deepinfra_provider_rejects_malformed_model(self, model):
|
||||
"""provider="deepinfra" applies the org/model check before building a client."""
|
||||
with patch.dict(os.environ, {"DEEPINFRA_API_KEY": "test-key"}), pytest.raises(
|
||||
ValueError, match="org/model"
|
||||
):
|
||||
LLM(model=model, provider="deepinfra")
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"model",
|
||||
[
|
||||
"deepinfra/DeepSeek-V4-Flash-0731",
|
||||
"deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731/",
|
||||
"deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731/extra",
|
||||
],
|
||||
)
|
||||
def test_llm_rejects_malformed_deepinfra_prefixed_model(self, model):
|
||||
"""A malformed deepinfra/... id raises the org/model error, never reaching LiteLLM."""
|
||||
with patch.dict(os.environ, {"DEEPINFRA_API_KEY": "test-key"}), pytest.raises(
|
||||
ValueError, match="org/model"
|
||||
) as exc_info:
|
||||
LLM(model=model)
|
||||
assert model in str(exc_info.value)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("model", "expected"),
|
||||
[
|
||||
("deepseek-ai/DeepSeek-V4-Flash-0731", True),
|
||||
("Qwen/Qwen3.5-27B", True),
|
||||
("meta-llama/Llama-4-Scout-17B-16E-Instruct", True),
|
||||
("", False),
|
||||
("qwen", False),
|
||||
("Qwen/", False),
|
||||
("/Qwen3.5-27B", False),
|
||||
("deepseek-ai/DeepSeek-V4-Flash-0731/extra", False),
|
||||
("deepseek-ai/DeepSeek-V4-Flash-0731/", False),
|
||||
],
|
||||
)
|
||||
def test_deepinfra_requires_an_org_and_a_model(self, model, expected):
|
||||
"""DeepInfra references are exactly org/model: one slash, both parts present."""
|
||||
assert LLM._matches_provider_pattern(model, "deepinfra") is expected
|
||||
|
||||
def test_llm_with_explicit_provider(self):
|
||||
"""Test LLM with explicit provider parameter."""
|
||||
with patch.dict(os.environ, {"DEEPSEEK_API_KEY": "test-key"}):
|
||||
|
||||
@@ -2,17 +2,27 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from contextlib import closing
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import sys
|
||||
from threading import Barrier
|
||||
|
||||
import pytest
|
||||
from crewai.flow.async_feedback.types import PendingFeedbackContext
|
||||
from crewai.flow.persistence import sqlite as sqlite_module
|
||||
from crewai.flow.persistence.sqlite import SQLiteFlowPersistence
|
||||
from crewai.state.provider.sqlite_provider import SqliteProvider
|
||||
|
||||
|
||||
_CONCURRENT_START_TIMEOUT_SECONDS = 10
|
||||
_CHILD_PROCESS_TIMEOUT_SECONDS = 60
|
||||
|
||||
|
||||
def _track_connections(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> list[sqlite3.Connection]:
|
||||
@@ -111,6 +121,79 @@ def test_flow_persistence_failed_write_rolls_back_and_closes(
|
||||
assert persistence.load_state("flow-1") is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("existing_db", [False, True])
|
||||
def test_parallel_flow_initialization_does_not_use_store_lock(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, existing_db: bool
|
||||
) -> None:
|
||||
db_path = tmp_path / "flows.db"
|
||||
if existing_db:
|
||||
persistence = SQLiteFlowPersistence(str(db_path))
|
||||
persistence.save_state("prior-flow", "start", {"step": 1})
|
||||
persistence.save_pending_feedback("prior-flow", _make_context(), {"step": 2})
|
||||
|
||||
def reject_store_lock(*args: object, **kwargs: object) -> None:
|
||||
raise AssertionError("initialization must not acquire a shared store lock")
|
||||
|
||||
monkeypatch.setattr(sqlite_module, "store_lock", reject_store_lock)
|
||||
barrier = Barrier(16)
|
||||
|
||||
def initialize(_: int) -> SQLiteFlowPersistence:
|
||||
barrier.wait(timeout=_CONCURRENT_START_TIMEOUT_SECONDS)
|
||||
return SQLiteFlowPersistence(str(db_path))
|
||||
|
||||
with ThreadPoolExecutor(max_workers=16) as pool:
|
||||
persistence_instances = list(pool.map(initialize, range(16)))
|
||||
|
||||
assert len(persistence_instances) == 16
|
||||
with closing(sqlite3.connect(db_path)) as conn:
|
||||
assert conn.execute("PRAGMA journal_mode").fetchone() == ("wal",)
|
||||
assert {
|
||||
row[0]
|
||||
for row in conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type = 'table'"
|
||||
)
|
||||
} >= {"flow_states", "pending_feedback"}
|
||||
if existing_db:
|
||||
assert persistence_instances[0].load_state("prior-flow") == {"step": 2}
|
||||
pending = persistence_instances[0].load_pending_feedback("prior-flow")
|
||||
assert pending is not None and pending[0] == {"step": 2}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("existing_db", [False, True])
|
||||
def test_parallel_processes_initialize_flow_db(
|
||||
tmp_path: Path, existing_db: bool
|
||||
) -> None:
|
||||
db_path = tmp_path / "flows.db"
|
||||
if existing_db:
|
||||
persistence = SQLiteFlowPersistence(str(db_path))
|
||||
persistence.save_state("prior-flow", "start", {"step": 1})
|
||||
|
||||
barrier = Barrier(2)
|
||||
script = (
|
||||
"import sys; from crewai.flow.persistence.sqlite import SQLiteFlowPersistence; "
|
||||
"SQLiteFlowPersistence(sys.argv[1])"
|
||||
)
|
||||
|
||||
def initialize(_: int) -> None:
|
||||
barrier.wait(timeout=_CONCURRENT_START_TIMEOUT_SECONDS)
|
||||
subprocess.run(
|
||||
[sys.executable, "-c", script, str(db_path)],
|
||||
check=True,
|
||||
timeout=_CHILD_PROCESS_TIMEOUT_SECONDS,
|
||||
)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=2) as pool:
|
||||
list(pool.map(initialize, range(2)))
|
||||
|
||||
with closing(sqlite3.connect(db_path)) as conn:
|
||||
assert conn.execute("PRAGMA journal_mode").fetchone() == ("wal",)
|
||||
persistence = SQLiteFlowPersistence(str(db_path))
|
||||
if existing_db:
|
||||
assert persistence.load_state("prior-flow") == {"step": 1}
|
||||
persistence.save_state("new-flow", "start", {"step": 2})
|
||||
assert persistence.load_state("new-flow") == {"step": 2}
|
||||
|
||||
|
||||
def test_sqlite_provider_closes_every_connection(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
|
||||
@@ -172,7 +172,7 @@ info = "Commits must follow Conventional Commits 1.0.0."
|
||||
[tool.uv]
|
||||
exclude-newer = "3 days"
|
||||
# These security fixes are newer than the global supply-chain cutoff.
|
||||
exclude-newer-package = { msgpack = "2026-06-20T00:00:00Z", pydantic-settings = "2026-06-20T00:00:00Z", langsmith = "2026-06-20T00:00:00Z" }
|
||||
exclude-newer-package = { msgpack = "2026-06-20T00:00:00Z", pydantic-settings = "2026-06-20T00:00:00Z", langsmith = "2026-06-20T00:00:00Z", oauthlib = "2026-09-29T00:00:00Z" }
|
||||
|
||||
# composio-core pins rich<14 but textual requires rich>=14.
|
||||
# onnxruntime 1.24+ dropped Python 3.10 wheels; cap it so qdrant[fastembed] resolves on 3.10.
|
||||
|
||||
3
uv.lock
generated
3
uv.lock
generated
@@ -23,6 +23,7 @@ exclude-newer-span = "P3D"
|
||||
[options.exclude-newer-package]
|
||||
msgpack = "2026-06-20T00:00:00Z"
|
||||
langsmith = "2026-06-20T00:00:00Z"
|
||||
oauthlib = "2026-09-29T00:00:00Z"
|
||||
pydantic-settings = "2026-06-20T00:00:00Z"
|
||||
|
||||
[manifest]
|
||||
@@ -1448,6 +1449,7 @@ dependencies = [
|
||||
{ name = "jsonref" },
|
||||
{ name = "lancedb" },
|
||||
{ name = "mcp" },
|
||||
{ name = "oauthlib" },
|
||||
{ name = "openai" },
|
||||
{ name = "openpyxl" },
|
||||
{ name = "opentelemetry-api" },
|
||||
@@ -1565,6 +1567,7 @@ requires-dist = [
|
||||
{ name = "litellm", marker = "extra == 'litellm'", specifier = ">=1.88.6,<2" },
|
||||
{ name = "mcp", specifier = "~=1.28.1" },
|
||||
{ name = "mem0ai", marker = "extra == 'mem0'", specifier = ">=2.0.0,<3" },
|
||||
{ name = "oauthlib", specifier = ">=4.0.0,<5" },
|
||||
{ name = "openai", specifier = ">=2.30.0,<3" },
|
||||
{ name = "openpyxl", specifier = "~=3.1.5" },
|
||||
{ name = "openpyxl", marker = "extra == 'openpyxl'", specifier = "~=3.1.5" },
|
||||
|
||||
Reference in New Issue
Block a user