Files
crewAI/lib/cli/tests/test_run_declarative_flow.py
T
João MouraandClaude Opus 5.5 4dcd19aa2e fix(tracing): turning tracing on is the answer, and an Evaluate button in the TUI (#7765)
* fix(tracing): turning tracing on is the answer; stop asking again

Two things a user who ran `crewai eval` sees today.

`crewai eval` offers to turn tracing on and run the crew. The run finishes —
and the ephemeral buffer asks "Share this execution trace with CrewAI?", which
is the same question again. Turning tracing on IS consent: `tracing_asked_for()`
is true when `CREWAI_TRACING_ENABLED` says so or when `tracing=True` was passed
in this context, and the buffer takes it as the answer. First-time
auto-collection still asks, because nobody asked for that one.

And the offer named the mechanism rather than the effect — "Turn tracing on
(CREWAI_TRACING_ENABLED=true stays in .env) and run the crew now?", then
"Tracing is on for this project (CREWAI_TRACING_ENABLED=true in .env)." The
variable is how it works, not what the user is agreeing to; both lines now say
what happens and leave the plumbing to the .env file it is written in.

491 telemetry tests and 62 eval-CLI tests green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(tracing): the TUI's modal is a prompt too, and say where traces go

Two follow-ups from running it.

The terminal UI installs its own consent callback, so turning tracing on still
bought a modal at the end of the run — the callback is consulted before the
default prompt, and the check only guarded the prompt. A host's callback stays
authoritative, because an embedder may answer no for reasons of its own; what
changed is that the TUI's callback, which exists to ASK, now answers for itself
when tracing was asked for.

And having taken the variable out of the offer, the offer should say what saying
yes means: "Turn tracing on and run the crew now? The run's trace is sent to
CrewAI AMP", and afterwards "Tracing is on for this project — its runs are
traced to CrewAI AMP."

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(tui): an Evaluate button beside View Traces and Deploy

A run finishes in the terminal UI and the next question is usually "was that
any good?" — which `crewai eval` answers, from the command line, about the run
that just happened. The button is there now, with `e` bound to it.

It behaves like Deploy rather than like View Traces: it leaves the UI and then
runs the command, because `crewai eval` talks — it prints the link to follow,
waits for the verdict and prints that — and none of it belongs inside a
full-screen app. It calls `eval_crew()` itself rather than reimplementing any
of it, so the button and the command can never say different things; a clean
SystemExit is the command deciding, and a failure is reported without failing
the run that already succeeded.

Declarative flows get it too — same button, same tail.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(telemetry): count `crewai eval`, not only the button that runs it

The TUI's buttons each record a `cli_usage:` feature span, so the new Evaluate
button was counted — but the command it runs was not, and most people will type
`crewai eval` rather than press a button. Every evaluation now records
`cli_usage:eval`; the button keeps its own `cli_usage:evaluate`, so the
difference between the two is how many were started from the TUI.

It counts the command and nothing about the run: no id, no project, no verdict.
And it is wrapped like every other telemetry call here — a missing tracer or a
refused span never stops an evaluation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(telemetry): say which run was evaluated, and by whom

`project_id` already rides every span. What only this command knows is which
run is being graded, whether the caller was logged in, and — when they are —
which organization they are logged in to. A feature span can carry attributes
now, and `cli_usage:eval` carries those three.

What it does not carry is anything about the work: no inputs, no output, no
verdict, no criteria. The execution id is the join key; what was found lives in
AMP, where the evaluation itself is.

An anonymous caller reports `authenticated=false` and an empty organization, and
is not asked for one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Address the review: a terminal, a thread, and the right run

Three findings worth acting on, one already fixed, and ruff.

**The security review is right.** `prompt_user_for_trace_viewing` fails closed
where no one can answer, so an anonymous run in CI has never uploaded. Treating
the switch as consent everywhere would have changed that: a project `.env`
copied into CI carries the variable, not the person. `tracing_asked_for()` now
answers only where a prompt could have been shown — the double-ask this PR is
for happens in a terminal, and nothing else moves.

**The ContextVar never crosses the worker.** `set_tracing_enabled` runs in the
crew's validator, on the thread the crew was BUILT on, and a flow never sets it
at all — so `tracing=True` without the env var still opened the modal. The TUI
reads the declaration off the object as well: the same yes, wherever written.

**The button now names the run it watched.** `_chain_eval()` called `crewai eval`
with no id, which looks up the last recorded run — a PREVIOUS run when this one
was not traced, and an offer to run the crew again when there is no record at
all. The TUI remembers what was recorded before the run, hands the new id to the
command, and says "this run was not traced" rather than grading someone else's.

**Already fixed**: telemetry is recorded after `get_or_create_project_id()`, not
before — that moved when the span gained its attributes.

492 telemetry tests, 187 CLI tests. Ruff format and check clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Close all three doors the prompt closes, not just the terminal

The security review named three places `prompt_user_for_trace_viewing` fails
closed — no terminal, a suite under test, and a host that suppressed tracing
messages — and the first pass only honoured the terminal. A run under test, or
inside a host that asked for silence, could still have uploaded on the strength
of a variable.

The three conditions are one function now, `_prompt_can_be_shown()`, used by the
prompt itself and by `tracing_asked_for`, so the rule and the prompt it answers
cannot drift apart. Where the question could not have been put to anybody, there
is nothing to answer and nothing leaves the machine.

The terminal UI decides for itself, and says why: it silences crewAI's console
to keep it out of its own layout, which is a reason a PROMPT cannot be shown,
not a reason to ask a second time — it has a screen and somebody in front of it.

494 telemetry tests, 88 TUI tests. Ruff clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Tests: a real session for the consent rule, and a laid-out screen for a click

The rule that a declared CREWAI_TRACING_ENABLED needs no modal only holds
where a prompt could have been shown, and a test suite is one of the places
it could not — so the three cases about a real session say a person is here.
The consent click waited for the screen, which is not the same as waiting for
its buttons to have a region; it now waits for the layout it clicks on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* A run started for an evaluation closes itself, even in a child process

A project's crew and flow are run through `uv run …`, so the app lives in
another process and the ContextVar saying "an evaluation is waiting" never
reaches it: the screen stayed open and `crewai eval` waited behind it, which
is the trap the handshake was meant to remove. The environment is the one
thing that crosses into that process, so the command sets a variable there and
clears it after; the id travels back the way it already did, through the
project's last-run record. A run that FAILED leaves too — the command behind
the screen has its own way of saying there is nothing to grade, and it cannot
say it while somebody has to quit a screen first.

And the spans a run exports reach AMP a moment after the run ends, sometimes
later than that, so a command that just watched the run would ask for a trace
still in flight and be told, correctly and uselessly, that there is nothing
there. A run we know is fresh now waits up to two minutes and says it is
waiting; an id somebody typed fails at once, as it always has.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* The evaluation happens in the app, on the screen the reader is already on

Closing the app the moment the run ended and leaving somebody in a bare
terminal, waiting some seconds for a link to appear and a browser to open, is
worse than the screen it replaced. So the evaluation runs where the run does:
`Evaluating this run` under the run's own line, the report link beside it, the
browser opened from there, and `Goal gate PASSED · goal 5/5 · …` when it comes
back. Nothing closes; the reader leaves when they are done reading, and the
link is printed once more on the way out so it outlives the screen.

The Evaluate button does the same thing now instead of quitting the app and
chaining into the command, so there is one implementation of one experience —
`_chain_eval` and `_want_eval` are gone. Pressing it again opens the report it
already made rather than paying for a second evaluation.

Underneath, `crewai eval`'s two steps became callable as values: they raise
`EvaluationStoppedError` with the sentence to show instead of printing and
exiting, and `evaluate_run()` is the app's one door — the url as soon as AMP
gives it, every unfinished answer while it waits. The command keeps the
terminal wording it had; when it ran the crew itself, it now has nothing left
to say and steps aside for the app it opened.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* The button says which report it opens

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* The button works while it waits: it spins, and it stays pressable

A dimmed label that never moves reads as a screen that has stopped. The
Evaluate button now carries the same spinner the run's own line does, the
clock goes back up to eight frames a second while the evaluation runs, and the
button is no longer disabled — the report page is where the progress is, so
pressing it while it waits is the useful thing to do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* View Traces opens the traces

The button never had a link. crewAI grants one for every exported run and
prints it after a run — but never under a TUI, where its console is kept out
of the layout — so the app had nothing to open and said so, which read as a
feature that had stopped working.

The link is now recorded beside the rest of what crewAI writes when a run's
spans reach AMP, and the button reads it off that record, matched to THIS
app's execution rather than to whatever ran last in the project. When there is
no link the notice says which of the two reasons it is: a run nobody traced,
or a traced run AMP granted no viewer for — only one of those is the reader's
to fix.

Verified against a real anonymous run: pressing `t` opens
app.crewai.com/crewai_plus/ephemeral_traces/<id> and shows it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* The review's five: a trust boundary, a fallback, a retry, an exit and a count

**A project's `.env` cannot lend itself the login.** `evaluate_run()` built its
AMP client after the project's `.env` had been loaded with override, so a
`CREWAI_PLUS_URL` a project put there landed in the set of origins trusted with
the saved `crewai login` bearer. The command reads that variable BEFORE the
`.env` — a shell export is this machine speaking — but inside the app there is
no such "before", so the variable is left out of the set entirely. The request
still follows it; the credential does not.

**A run no app evaluated is still evaluated.** The command returned as soon as
a trace existed, assuming the app had graded it — but a conversational session
never ends the way a crew does and a flow with human feedback takes the
terminal instead, so `crewai eval` could exit 0 having graded nothing. The app
now leaves a marker naming the run it started evaluating, and the command
grades anything that marker does not name.

**An evaluation that stopped can be tried again.** The button re-enabled itself
after a failure but the action still treated any existing evaluation as
finished, so a 502 could only be escaped by re-running the crew.

**A login that exits is shown on screen.** `saved_login()` fails by exiting,
which is not an `Exception` and so slipped past the worker's hands: the button
could sit on "Evaluating…" forever with the reason gone with the exit.

**An evaluation the app runs is counted.** `cli_usage:eval` is the count of
evaluations that begin, and the ones that begin in the app were missing from
it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* A stop is a sentence, and the app's own word settles what it graded

**"1" is not a reason.** A credential that exists but cannot be read stopped
the evaluation by printing its sentence and exiting — under the app the print
lands beneath the layout and the exit carries only a code, so the screen said
`1`. `saved_login()` now raises the stop like everything else that stops an
evaluation, the command catches it where it builds the client and prints it as
before, and the app shows it. The test that claimed to cover this raised a
`SystemExit` carrying a sentence, which is not what `_fail` raises; it now
breaks the credential store and reads what reaches the screen. The worker
still catches an exit, and says that the reason did not survive rather than
echoing its code.

**The record is the project's last finish, not this run.** Another run
finishing in the same project replaces it while the app is open, so "is the
marker about the id in the record" could send the command off to evaluate a
different execution. The marker is now the app's own word about the run it
watched — the id it evaluated, or null when it looked and found nothing to
grade — and the command trusts it over the record, matching on time rather
than on an id that may already have been overwritten. No marker still means no
app got here, which is the conversational session and the flow that takes the
terminal, and the command grades those itself.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* A project can say what good means, and `crewai eval` carries it

An evaluation grades a run against the crew's own `expected_output`s — which
is the prompt that produced the output, so it catches a crew that ignored its
own contract and nothing else. What discriminates lives in criteria somebody
writes, and there was nowhere to write them.

Now there is. After the first evaluation the criteria it used are written to
`eval.jsonc` in the project — once, never over a file that exists, because the
one thing worse than no criteria is criteria that vanish whenever somebody
evaluates — and the line that says so points at what to do with it. From then
on the file travels with every evaluation, as it was written, comments and
all: what a criterion means is the grader's to read, not this command's.

Both doors do it, since most evaluations now happen inside the run app: the
app writes the file and names it in the line that outlives its screen, and
`crewai eval` in a terminal says it there.

The file is bounded at the same 64KB AMP holds the field to, so one that would
be refused is never uploaded and the run is graded anyway.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* The review's asks: anonymous eval stats, a handshake a project cannot forge

Usage stats are anonymous, so the `crewai eval` count says whether the caller
was logged in and nothing else — no execution id, no organization (the product
decision on this PR). Which runs were evaluated, and by whom, is AMP's record.
A feature span now keeps only the keys its own feature lists, in both
telemetry classes; everything else — the dimensions every span carries
included — is dropped.

The run app's "an evaluation is waiting" handshake was an environment variable
any project's `.env` could set, which would start an evaluation with the
machine's login on a plain `crewai run`. Its value is now a random token that
counts only while the command's file of that name exists in the user's crewAI
data directory; a `.env` cannot create a file.

And three smaller ones: when no app evaluated the run, the command grades the
project's record only if it was written after this run began; the "Wrote
eval.jsonc" line after the app closes now appears (the key it read was never
set); an oversized eval.jsonc is said once, through the caller's note, rather
than printed under the TUI on every retry.

uv.lock is back to main's: the change was re-resolution drift, nothing this PR
needs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* A report link is a web address, and an anonymous request names no organization

Two from the security review of 52fa5f3ae.

The report url comes from whichever AMP answered — which a project's `.env`
may choose — and is printed as a link and opened without a click. It is now
checked where it arrives, before anything shows or opens it: HTTPS, or plain
HTTP to this machine, no credentials in it, no control characters. Anything
else costs the link and nothing more. The host is not pinned: the report is
served by the evaluation service, not by AMP.

A request sent without the login — nobody logged in, or an AMP this machine is
not logged in to — no longer carries the saved organization id. AMP reads it
only beside a credential, so it said nothing to AMP and told an AMP the login
does not go to which organization was asking. Dropped, as anonymous trace
grants already drop it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-28 11:28:52 -07:00

827 lines
26 KiB
Python

from __future__ import annotations
import os
from pathlib import Path
from typing import Any
from types import SimpleNamespace
import pytest
import crewai_cli.input_prompt as input_prompt_module
import crewai_cli.run_declarative_flow as run_declarative_flow_module
@pytest.fixture(autouse=True)
def _headless_by_default(monkeypatch: pytest.MonkeyPatch) -> None:
"""Default these tests to the headless/terminal path.
``run_declarative_flow`` now launches the TUI when interactive, which can't
run under pytest; tests here assert the terminal/headless contract. Tests
that exercise TUI routing override ``is_dmn_mode_enabled`` explicitly.
"""
monkeypatch.setenv("CREWAI_DMN", "true")
FLOW_YAML = """\
schema: crewai.flow/v1
name: TestFlow
config:
suppress_flow_events: true
methods:
begin:
start: true
do:
call: expression
expr: state.topic
"""
def test_run_declarative_flow_reads_definition_file(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
definition_path = tmp_path / "flow.yaml"
definition_path.write_text(FLOW_YAML, encoding="utf-8")
run_declarative_flow_module.run_declarative_flow(
str(definition_path), '{"topic":"AI"}'
)
assert capsys.readouterr().out == "AI\n"
def test_run_declarative_flow_rejects_non_object_inputs(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
definition_path = tmp_path / "flow.yaml"
definition_path.write_text(FLOW_YAML, encoding="utf-8")
with pytest.raises(SystemExit):
run_declarative_flow_module.run_declarative_flow(
str(definition_path), '["not", "an", "object"]'
)
assert "Invalid --inputs JSON: expected an object." in capsys.readouterr().err
def test_run_declarative_flow_reports_missing_file(
capsys: pytest.CaptureFixture[str],
) -> None:
with pytest.raises(SystemExit):
run_declarative_flow_module.run_declarative_flow("missing-flow.yaml")
assert (
"Invalid --definition path: missing-flow.yaml does not exist."
in capsys.readouterr().err
)
def test_run_declarative_flow_reports_empty_file(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
definition_path = tmp_path / "flow.yaml"
definition_path.write_text(" \n", encoding="utf-8")
with pytest.raises(SystemExit):
run_declarative_flow_module.run_declarative_flow(str(definition_path))
assert "Flow declaration file is empty" in capsys.readouterr().err
@pytest.mark.parametrize(
"contents, expected_error",
[
("[]\n", "Flow declaration must contain a mapping"),
("schema: crewai.flow/v1\nmethods: {}\n", "Field required"),
],
)
def test_load_declarative_flow_reports_invalid_declarations(
tmp_path: Path,
capsys: pytest.CaptureFixture[str],
contents: str,
expected_error: str,
) -> None:
definition_path = tmp_path / "flow.yaml"
definition_path.write_text(contents, encoding="utf-8")
with pytest.raises(SystemExit) as exc_info:
run_declarative_flow_module.load_declarative_flow(str(definition_path))
assert exc_info.value.code == 1
stderr = capsys.readouterr().err
assert f"Unable to read --definition path {definition_path}:" in stderr
assert expected_error in stderr
def test_run_declarative_flow_in_project_env_uses_uv(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
subprocess_calls = []
monkeypatch.chdir(tmp_path)
monkeypatch.delenv("UV_RUN_RECURSION_DEPTH", raising=False)
(tmp_path / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
monkeypatch.setattr(
run_declarative_flow_module,
"build_env_with_all_tool_credentials",
lambda: {"EXISTING": "value"},
)
monkeypatch.setattr(
run_declarative_flow_module.subprocess,
"run",
lambda command, **kwargs: subprocess_calls.append((command, kwargs)),
)
run_declarative_flow_module.run_declarative_flow_in_project_env("flow.yaml")
assert subprocess_calls == [
(
["uv", "run", "crewai", "run"],
{
"capture_output": False,
"text": True,
"check": True,
"env": {"EXISTING": "value"},
},
)
]
def test_run_declarative_flow_in_process_inside_uv(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
capsys: pytest.CaptureFixture[str],
) -> None:
monkeypatch.chdir(tmp_path)
monkeypatch.setenv("UV_RUN_RECURSION_DEPTH", "1")
(tmp_path / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
(tmp_path / "flow.yaml").write_text(FLOW_YAML, encoding="utf-8")
run_declarative_flow_module.run_declarative_flow_in_project_env(
"flow.yaml", '{"topic":"AI"}'
)
assert capsys.readouterr().out == "AI\n"
def test_run_declarative_flow_in_project_env_forwards_inputs(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
subprocess_calls = []
monkeypatch.chdir(tmp_path)
monkeypatch.delenv("UV_RUN_RECURSION_DEPTH", raising=False)
(tmp_path / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
monkeypatch.setattr(
run_declarative_flow_module,
"build_env_with_all_tool_credentials",
lambda: {},
)
monkeypatch.setattr(
run_declarative_flow_module.subprocess,
"run",
lambda command, **kwargs: subprocess_calls.append(command),
)
run_declarative_flow_module.run_declarative_flow_in_project_env(
"flow.yaml", '{"topic":"AI"}'
)
# --inputs is forwarded to the in-env run instead of being rejected.
assert subprocess_calls == [
["uv", "run", "crewai", "run", "--inputs", '{"topic":"AI"}']
]
# ── Schema-driven inputs: prompt, validate, override ────────────────
REQUIRED_FLOW_YAML = """\
schema: crewai.flow/v1
name: RequiredInputFlow
config:
suppress_flow_events: true
state:
type: json_schema
json_schema:
type: object
properties:
prospect_email:
type: string
description: Email address of the prospect to research
required: [prospect_email]
methods:
begin:
start: true
do:
call: expression
expr: state.prospect_email
"""
DEFAULTS_FLOW_YAML = """\
schema: crewai.flow/v1
name: DefaultsFlow
config:
suppress_flow_events: true
state:
type: json_schema
json_schema:
type: object
properties:
topic: {type: string}
audience: {type: string}
required: [topic, audience]
default:
topic: AI
methods:
begin:
start: true
do:
call: expression
expr: state.audience
"""
TYPED_FLOW_YAML = """\
schema: crewai.flow/v1
name: TypedFlow
config:
suppress_flow_events: true
state:
type: json_schema
json_schema:
type: object
properties:
count: {type: integer}
required: [count]
methods:
begin:
start: true
do:
call: expression
expr: state.count
"""
def _write(tmp_path: Path, contents: str) -> Path:
path = tmp_path / "flow.yaml"
path.write_text(contents, encoding="utf-8")
return path
def test_inputs_flag_satisfies_required_field(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
path = _write(tmp_path, REQUIRED_FLOW_YAML)
run_declarative_flow_module.run_declarative_flow(
str(path), '{"prospect_email":"a@b.com"}'
)
assert capsys.readouterr().out == "a@b.com\n"
def test_missing_required_reports_pointed_error(
tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(run_declarative_flow_module, "_is_interactive", lambda: False)
path = _write(tmp_path, REQUIRED_FLOW_YAML)
with pytest.raises(SystemExit):
run_declarative_flow_module.run_declarative_flow(str(path))
assert (
"Missing required input 'prospect_email' — "
"Email address of the prospect to research" in capsys.readouterr().err
)
def test_prompts_for_missing_required_when_interactive(
tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
) -> None:
path = _write(tmp_path, REQUIRED_FLOW_YAML)
monkeypatch.setattr(run_declarative_flow_module, "_is_interactive", lambda: True)
prompted: list[str] = []
def fake_prompt(text: str, **kwargs: object) -> str:
prompted.append(text)
return "typed@example.com"
monkeypatch.setattr(input_prompt_module.click, "prompt", fake_prompt)
run_declarative_flow_module.run_declarative_flow(str(path))
assert capsys.readouterr().out == "typed@example.com\n"
assert any("prospect_email" in text for text in prompted)
def test_defaults_satisfy_required_and_are_not_prompted(
tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(run_declarative_flow_module, "_is_interactive", lambda: False)
path = _write(tmp_path, DEFAULTS_FLOW_YAML)
with pytest.raises(SystemExit):
run_declarative_flow_module.run_declarative_flow(str(path))
err = capsys.readouterr().err
# topic has a state default -> satisfied; only audience is missing.
assert "Missing required input 'audience'" in err
assert "'topic'" not in err
def test_warns_on_unknown_input_with_suggestion(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
path = _write(tmp_path, REQUIRED_FLOW_YAML)
run_declarative_flow_module.run_declarative_flow(
str(path), '{"prospect_email":"a@b.com","prospect_emai":"typo"}'
)
captured = capsys.readouterr()
assert captured.out == "a@b.com\n"
assert "Ignoring unknown input 'prospect_emai'" in captured.err
assert "Did you mean 'prospect_email'?" in captured.err
def test_validates_input_types_before_kickoff(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
path = _write(tmp_path, TYPED_FLOW_YAML)
with pytest.raises(SystemExit):
run_declarative_flow_module.run_declarative_flow(str(path), '{"count":"nope"}')
assert "Invalid input 'count'" in capsys.readouterr().err
def test_reserved_id_input_is_forwarded_not_dropped(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
# `id` is a reserved kickoff key (persistence restore); it must pass through
# instead of being flagged as an unknown key and dropped.
path = _write(tmp_path, REQUIRED_FLOW_YAML)
run_declarative_flow_module.run_declarative_flow(
str(path), '{"id":"run-123","prospect_email":"a@b.com"}'
)
captured = capsys.readouterr()
assert captured.out == "a@b.com\n"
assert "Ignoring unknown input 'id'" not in captured.err
def test_run_declarative_flow_loads_project_env(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
# Flow projects must pick up the project's .env, like crew projects do,
# overriding any pre-existing value.
monkeypatch.chdir(tmp_path)
monkeypatch.setenv("DECL_FLOW_ENV_PROBE", "old")
(tmp_path / ".env").write_text("DECL_FLOW_ENV_PROBE=from_dotenv\n", encoding="utf-8")
path = _write(tmp_path, REQUIRED_FLOW_YAML)
run_declarative_flow_module.run_declarative_flow(
str(path), '{"prospect_email":"a@b.com"}'
)
assert os.environ["DECL_FLOW_ENV_PROBE"] == "from_dotenv"
def test_id_only_input_skips_required_validation(tmp_path: Path) -> None:
# Resume via `crewai run --inputs '{"id":"..."}'` must not be blocked by the
# required-field check: kickoff hydrates required state from persistence.
path = _write(tmp_path, REQUIRED_FLOW_YAML)
flow = run_declarative_flow_module.load_declarative_flow(str(path))
resolved = run_declarative_flow_module._resolve_flow_inputs(flow, {"id": "run-123"})
assert resolved == {"id": "run-123"}
def test_id_restore_still_drops_unknown_keys(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
# A persistence restore (`id` present) still filters typo keys so they don't
# reach kickoff and trip strict (extra="forbid") state models — it only
# skips the required-field prompt/validation, not the unknown-key warning.
path = _write(tmp_path, REQUIRED_FLOW_YAML)
flow = run_declarative_flow_module.load_declarative_flow(str(path))
resolved = run_declarative_flow_module._resolve_flow_inputs(
flow, {"id": "run-123", "prospect_emai": "typo"}
)
captured = capsys.readouterr()
assert resolved == {"id": "run-123"} # id kept, typo dropped
assert "Ignoring unknown input 'prospect_emai'" in captured.err
assert "Ignoring unknown input 'id'" not in captured.err
# ── TUI vs terminal (headless/deploy) routing ──────────────────────
def _install_fake_flow_app(monkeypatch, *, status, want_deploy=False, evaluation=None):
"""Replace CrewRunApp/EventListener/summary so _run_declarative_flow_tui is
driven by a controllable fake app."""
class FakeEventListener:
pass
class FakeApp:
def __init__(self, crew_name=""):
self._crew_name = crew_name
self._status = status
self._want_deploy = want_deploy
self._evaluation = evaluation
self._crew_result = "result"
def run(self):
pass
monkeypatch.setattr(
"crewai.events.event_listener.EventListener", FakeEventListener
)
monkeypatch.setattr("crewai_cli.crew_run_tui.CrewRunApp", FakeApp)
monkeypatch.setattr(
run_declarative_flow_module, "_print_flow_post_tui_summary", lambda app: None
)
def test_run_declarative_flow_dmn_uses_terminal(
tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setenv("CREWAI_DMN", "true")
monkeypatch.setattr(
run_declarative_flow_module,
"_run_declarative_flow_tui",
lambda *a, **k: pytest.fail("DMN/headless mode must not launch the TUI"),
)
path = _write(tmp_path, REQUIRED_FLOW_YAML)
run_declarative_flow_module.run_declarative_flow(
str(path), '{"prospect_email":"a@b.com"}'
)
assert capsys.readouterr().out == "a@b.com\n"
def test_run_declarative_flow_interactive_uses_tui(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(run_declarative_flow_module, "is_interactive", lambda: True)
captured: dict[str, object] = {}
monkeypatch.setattr(
run_declarative_flow_module,
"_run_declarative_flow_tui",
lambda flow, resolved: captured.update(flow=flow, inputs=resolved),
)
path = _write(tmp_path, REQUIRED_FLOW_YAML)
run_declarative_flow_module.run_declarative_flow(
str(path), '{"prospect_email":"a@b.com"}'
)
assert captured["inputs"] == {"prospect_email": "a@b.com"}
assert captured["flow"] is not None
def test_run_declarative_flow_tui_failed_exits_nonzero(
monkeypatch: pytest.MonkeyPatch,
) -> None:
_install_fake_flow_app(monkeypatch, status="failed")
with pytest.raises(SystemExit) as exc_info:
run_declarative_flow_module._run_declarative_flow_tui(
SimpleNamespace(name="Flow"), None
)
assert exc_info.value.code == 1
def test_run_declarative_flow_tui_user_quit_exits_130(
monkeypatch: pytest.MonkeyPatch,
) -> None:
_install_fake_flow_app(monkeypatch, status="chatting")
exit_calls: list[int] = []
monkeypatch.setattr(os, "_exit", lambda code: exit_calls.append(code))
run_declarative_flow_module._run_declarative_flow_tui(
SimpleNamespace(name="Flow"), None
)
assert exit_calls == [130]
def test_run_declarative_flow_tui_chains_deploy(
monkeypatch: pytest.MonkeyPatch,
) -> None:
_install_fake_flow_app(monkeypatch, status="completed", want_deploy=True)
deploy_calls: list[bool] = []
monkeypatch.setattr(
"crewai_cli.run_crew._chain_deploy", lambda: deploy_calls.append(True)
)
run_declarative_flow_module._run_declarative_flow_tui(
SimpleNamespace(name="Flow"), None
)
assert deploy_calls == [True]
def test_run_declarative_flow_tui_no_deploy_when_not_requested(
monkeypatch: pytest.MonkeyPatch,
) -> None:
_install_fake_flow_app(monkeypatch, status="completed", want_deploy=False)
deploy_calls: list[bool] = []
monkeypatch.setattr(
"crewai_cli.run_crew._chain_deploy", lambda: deploy_calls.append(True)
)
run_declarative_flow_module._run_declarative_flow_tui(
SimpleNamespace(name="Flow"), None
)
assert deploy_calls == []
def test_run_declarative_flow_tui_leaves_the_report_link_behind(capsys) -> None:
"""A declarative flow evaluates inside the app like a crew does. What the
terminal is for is afterwards: the link, once the screen that held it is
gone."""
app = SimpleNamespace(
_crew_name="Flow",
_status="completed",
_elapsed_frozen=1.0,
_start_time=0.0,
_input_tokens=0,
_output_tokens=0,
_live_out_tokens=0,
_final_output=None,
_error=None,
_evaluation={
"state": "done",
"url": "https://optimize.example/e/1",
"verdict": {"gate": "passed"},
},
)
run_declarative_flow_module._print_flow_post_tui_summary(app)
printed = capsys.readouterr().out
assert "goal gate PASSED" in printed
assert "https://optimize.example/e/1" in printed
def test_run_declarative_flow_tui_enables_flow_events(
monkeypatch: pytest.MonkeyPatch,
) -> None:
# The STEPS panel depends on flow method events; a flow that declared
# suppress_flow_events must have it forced off for the interactive TUI run.
_install_fake_flow_app(monkeypatch, status="completed")
flow = SimpleNamespace(name="Flow", suppress_flow_events=True)
run_declarative_flow_module._run_declarative_flow_tui(flow, None)
assert flow.suppress_flow_events is False
def test_flow_uses_human_feedback_detection() -> None:
hf_flow = SimpleNamespace(
_definition=SimpleNamespace(
methods={
"ask": SimpleNamespace(human_feedback=SimpleNamespace(emit=None)),
"plain": SimpleNamespace(human_feedback=None),
}
)
)
assert run_declarative_flow_module._flow_uses_human_feedback(hf_flow) is True
no_hf = SimpleNamespace(
_definition=SimpleNamespace(
methods={"a": SimpleNamespace(human_feedback=None)}
)
)
assert run_declarative_flow_module._flow_uses_human_feedback(no_hf) is False
# No definition → False, no error.
assert run_declarative_flow_module._flow_uses_human_feedback(SimpleNamespace()) is False
def test_human_feedback_flow_uses_terminal_even_when_interactive(
tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
) -> None:
# A human-feedback flow must run on the terminal (blocking input / Rich
# prompts) even in an interactive session, never on the TUI.
monkeypatch.setattr(run_declarative_flow_module, "is_interactive", lambda: True)
monkeypatch.setattr(
run_declarative_flow_module, "_flow_uses_human_feedback", lambda flow: True
)
monkeypatch.setattr(
run_declarative_flow_module,
"_run_declarative_flow_tui",
lambda *a, **k: pytest.fail("human-feedback flow must run on the terminal"),
)
path = _write(tmp_path, FLOW_YAML)
run_declarative_flow_module.run_declarative_flow(str(path), '{"topic":"AI"}')
assert capsys.readouterr().out == "AI\n"
def test_flow_method_types_from_definition() -> None:
flow = SimpleNamespace(
_definition=SimpleNamespace(
methods={
"fetch": SimpleNamespace(do=SimpleNamespace(call="expression")),
"research": SimpleNamespace(do=SimpleNamespace(call="crew")),
}
)
)
assert run_declarative_flow_module._flow_method_types(flow) == {
"fetch": "expression",
"research": "crew",
}
# No definition → empty map, no error.
assert run_declarative_flow_module._flow_method_types(SimpleNamespace()) == {}
CONVERSATIONAL_FLOW_YAML = """schema: crewai.flow/v1
name: SupportFlow
conversational:
llm: gpt-4o-mini
methods:
handle_order:
description: Order status questions.
listen: order
do:
call: expression
expr: "'shipped'"
"""
def test_run_declarative_flow_opens_the_conversational_tui(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A declarative chat flow gets the same TUI a Python one does."""
definition_path = tmp_path / "flow.yaml"
definition_path.write_text(CONVERSATIONAL_FLOW_YAML, encoding="utf-8")
launched: list[Any] = []
monkeypatch.setattr(run_declarative_flow_module, "is_interactive", lambda: True)
monkeypatch.setattr(
"crewai_cli.kickoff_flow._run_conversational_flow_tui", launched.append
)
run_declarative_flow_module.run_declarative_flow(str(definition_path))
assert len(launched) == 1
assert launched[0]._is_conversational_enabled() is True
def test_conversational_flow_does_not_take_the_steps_tui(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The STEPS TUI renders method progress and cannot drive a chat turn."""
definition_path = tmp_path / "flow.yaml"
definition_path.write_text(CONVERSATIONAL_FLOW_YAML, encoding="utf-8")
monkeypatch.setattr(run_declarative_flow_module, "is_interactive", lambda: True)
monkeypatch.setattr(
"crewai_cli.kickoff_flow._run_conversational_flow_tui", lambda flow: None
)
monkeypatch.setattr(
run_declarative_flow_module,
"_run_declarative_flow_tui",
lambda *a, **k: pytest.fail("conversational flow reached the STEPS TUI"),
)
run_declarative_flow_module.run_declarative_flow(str(definition_path))
def test_conversational_flow_rejects_inputs(
tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
) -> None:
"""`--inputs` has nowhere to go: handle_turn owns the kickoff inputs."""
definition_path = tmp_path / "flow.yaml"
definition_path.write_text(CONVERSATIONAL_FLOW_YAML, encoding="utf-8")
monkeypatch.setattr(run_declarative_flow_module, "is_interactive", lambda: True)
monkeypatch.setattr(
"crewai_cli.kickoff_flow._run_conversational_flow_tui",
lambda flow: pytest.fail("should have rejected --inputs before the TUI"),
)
with pytest.raises(SystemExit) as exc_info:
run_declarative_flow_module.run_declarative_flow(
str(definition_path), '{"topic": "AI"}'
)
assert exc_info.value.code == 1
err = capsys.readouterr().err
assert "`--inputs` is not supported for a conversational flow" in err
assert "session_id" in err
def test_conversational_flow_rejects_an_empty_inputs_object(
tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
) -> None:
"""`--inputs '{}'` asked for something unsupported; answer it, don't ignore it."""
definition_path = tmp_path / "flow.yaml"
definition_path.write_text(CONVERSATIONAL_FLOW_YAML, encoding="utf-8")
monkeypatch.setattr(run_declarative_flow_module, "is_interactive", lambda: True)
monkeypatch.setattr(
"crewai_cli.kickoff_flow._run_conversational_flow_tui",
lambda flow: pytest.fail("an empty --inputs object still passed --inputs"),
)
with pytest.raises(SystemExit) as exc_info:
run_declarative_flow_module.run_declarative_flow(str(definition_path), "{}")
assert exc_info.value.code == 1
assert (
"`--inputs` is not supported for a conversational flow"
in capsys.readouterr().err
)
def test_conversational_flow_without_inputs_still_opens_the_tui(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
definition_path = tmp_path / "flow.yaml"
definition_path.write_text(CONVERSATIONAL_FLOW_YAML, encoding="utf-8")
launched: list[Any] = []
monkeypatch.setattr(run_declarative_flow_module, "is_interactive", lambda: True)
monkeypatch.setattr(
"crewai_cli.kickoff_flow._run_conversational_flow_tui", launched.append
)
run_declarative_flow_module.run_declarative_flow(str(definition_path), None)
assert len(launched) == 1
def test_conversational_human_feedback_flow_avoids_the_tui(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Textual cannot service the runtime's blocking feedback prompt.
The STEPS TUI already declines these for that reason; a conversational one
must too, or the prompt is never shown and the run hangs.
"""
definition_path = tmp_path / "flow.yaml"
definition_path.write_text(
CONVERSATIONAL_FLOW_YAML.replace(
" listen: order\n",
" listen: order\n human_feedback:\n message: Approve?\n",
),
encoding="utf-8",
)
chatted: list[str] = []
monkeypatch.setattr(run_declarative_flow_module, "is_interactive", lambda: True)
monkeypatch.setattr(
"crewai_cli.kickoff_flow._run_conversational_flow_tui",
lambda flow: pytest.fail("human-feedback flow reached the Textual TUI"),
)
monkeypatch.setattr(
"crewai.flow.flow.Flow.chat", lambda self, **kw: chatted.append("repl")
)
run_declarative_flow_module.run_declarative_flow(str(definition_path))
assert chatted == ["repl"]
def test_conversational_flow_headless_explains_instead_of_one_turn(
tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
) -> None:
definition_path = tmp_path / "flow.yaml"
definition_path.write_text(CONVERSATIONAL_FLOW_YAML, encoding="utf-8")
monkeypatch.setattr(run_declarative_flow_module, "is_interactive", lambda: False)
with pytest.raises(SystemExit) as exc_info:
run_declarative_flow_module.run_declarative_flow(str(definition_path))
assert exc_info.value.code == 1
err = capsys.readouterr().err
assert "needs an interactive terminal" in err
assert "handle_turn" in err
def test_run_declarative_flow_still_runs_a_disabled_conversational_flow(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
definition_path = tmp_path / "flow.yaml"
definition_path.write_text(
CONVERSATIONAL_FLOW_YAML.replace(
"conversational:\n llm: gpt-4o-mini",
"conversational:\n enabled: false",
).replace(" listen: order\n", " start: true\n"),
encoding="utf-8",
)
run_declarative_flow_module.run_declarative_flow(str(definition_path))
assert capsys.readouterr().out == "shipped\n"