mirror of
https://github.com/crewAIInc/crewAI.git
synced 2026-10-22 18:03:10 +02:00
Joao MouraandClaude Opus 5.5
9aed68d495
Most `crewai eval` attempts stopped before an evaluation began because the
project had no traced run. Telemetry since Sept 25: 117 projects stopped as
"untraced", 97 of them with no terminal (coding agents), and 67 of those never
got further. With no terminal the command could only print steps, and without
a login those steps could not work: an unattended anonymous run never uploads
its trace.
- With no traced run, or when the project's files changed since the last one,
`crewai eval` runs the crew and evaluates that run. No question, with or
without a terminal: running the command is the request to have the run
graded, and the grade is read off its trace. Tracing is on for that run only;
nothing is written to the project's `.env`.
- crewAI's tracing approves the upload of that run through the existing
handshake: a token in CREWAI_EVAL_AWAITING_RUN that counts only while its file
exists in the user's crewAI data folder, so a `.env` cannot claim it. Shared
as `crewai_core.eval_handshake`; the run app keeps its own copy because it
also runs against a project's older crewai-core, and a test pins the two.
- The run app's Evaluate on an untraced run closes the app and runs the crew
again, traced, through `crewai eval`'s own path, instead of a notice.
- Counts: `eval_ran_crew:{no_traced_run,project_changed,run_again}`,
`eval_finished:{passed,failed,inconclusive,unknown}`, and
`eval_stopped:untraced_after_run` replacing the catch-all `untraced`
(`declined` is gone with the question). The CLI tests no longer send usage
counts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
14 lines
520 B
Python
14 lines
520 B
Python
"""Shared setup for the CLI's tests."""
|
|
|
|
import pytest
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _usage_counts_stay_on_this_machine(monkeypatch):
|
|
"""The CLI counts what its commands do (`cli_usage:*`), and a suite that
|
|
exercises them would send those counts as if users had run them: every
|
|
`crewai eval` stop reason within the same second, from a laptop or from CI.
|
|
A test that checks the counting replaces `Telemetry` or sets the flag
|
|
itself."""
|
|
monkeypatch.setenv("CREWAI_DISABLE_TELEMETRY", "true")
|